The storage engine of OceanBase Database uses the log-structured merge-tree (LSM tree) architecture. In this architecture, data is stored in a MemTable in memory and an SSTable on disk. As user data is written to the MemTable, the memory occupied by the MemTable exceeds the specified threshold, and data in the MemTable is flushed to the SSTable on disk to release the memory space. This process is called a minor compaction. Minor compactions generate SSTables. A large number of SSTables will be generated as minor compactions are performed. This increases the number of SSTables that need to be accessed during queries, which in turn reduces query efficiency. This is why the core concept of the LSM-tree, compaction, is introduced.
OceanBase Database uses a tiered & leveled compaction strategy, which divides SSTables into three layers: L0, L1, and L2. The L0 layer uses the tiered mode, and the L1 and L2 layers use the leveled mode.
Compaction types
Compaction is a reorganization of data. It is a multi-way merge sort that sorts several SSTables by rowkey and then outputs them as one SSTable.
OceanBase Database supports the following types of compaction:
Mini Compaction: converts a MemTable into a Mini SSTable. It can be triggered automatically or manually.
Minor Compaction: combines multiple Mini SSTables into one Mini SSTable or combines multiple Mini SSTables and one Minor SSTable into one Minor SSTable. It is triggered by a background scheduling thread and cannot be manually triggered.
Medium Compaction: combines the original baseline SSTable (Major SSTable) of a specified partition with all Mini SSTables and Minor SSTables in the partition into one Major SSTable. It can be triggered automatically or manually.
Major Compaction: combines the original baseline SSTables (Major SSTables) of all partitions in the tenant with all Mini SSTables and Minor SSTables into one Major SSTable. It can be triggered automatically or manually.
Meta Major Compaction: a special type of compaction that combines data before a specified point in time into one Meta Major SSTable. The data format of a Meta Major SSTable is the same as that of a Major SSTable, but it does not contain multi-version data or uncommitted transaction data. It is triggered by a background scheduling thread and cannot be manually triggered.
Minor compaction and major compaction
Minor compaction comprises two processes: mini compaction and minor compaction. When the memory occupied by the MemTable exceeds the specified threshold, data in the MemTable is flushed to the Mini SSTable to release the memory space. This process is called mini compaction. As user data is written, the number of Mini SSTables increases. When the number of Mini SSTables exceeds the specified threshold, the background scheduling thread automatically triggers minor compaction. Minor compactions generate new Mini SSTables. When the number of minor compactions exceeds the specified threshold, or during off-peak hours, the system combines the baseline SSTable (Major SSTable) with the incremental SSTables (Mini SSTables/Minor SSTables) generated by minor compactions into one Major SSTable. This process is called major compaction.
For more information about minor compactions, see Minor compactions.
For more information about major compactions, see Major compactions.