The storage engine of OceanBase Database is based on the LSM-tree architecture. Data is broadly divided into MemTable in memory and SSTable on disk. As user data is written to MemTable, MemTable exceeds the memory threshold and is converted into SSTable on disk. The number of SSTables keeps increasing, which increases the number of SSTable files that need to be accessed during queries and reduces query efficiency. This is why the core LSM-tree concept of Compaction is introduced.
OceanBase Database uses a Tiered & Leveled Compaction strategy, dividing SSTables into three layers: L0, L1, and L2. The L0 layer uses tiered mode, and the L1 and L2 layers use leveled mode.
Compaction types
Compaction is a reorganization of data. It is a multi-way merge sort that sorts several SSTables by rowkey and then outputs them as one SSTable.
OceanBase Database supports the following types of compaction:
Mini compaction: converts a MemTable into a Mini SSTable. It can be triggered automatically or manually.
Minor compaction: combines multiple Mini SSTables into one Mini SSTable or combines multiple Mini SSTables and one Minor SSTable into one Minor SSTable. It is triggered by a background scheduling thread and cannot be manually triggered.
Medium Compaction: combines the original baseline SSTable (Major SSTable) of a specified partition with all Mini SSTables and Minor SSTables in the partition into one Major SSTable. It can be triggered automatically or manually.
Major compaction: combines the original baseline SSTables (Major SSTables) of all partitions in the tenant with all Mini SSTables and Minor SSTables into one Major SSTable. It can be triggered automatically or manually.
Meta Major Compaction: a special type of compaction that combines data before a specified point in time into one Meta Major SSTable. The data format of a Meta Major SSTable is the same as that of a Major SSTable, but it does not contain multi-version data or uncommitted transaction data. It is triggered by a background scheduling thread and cannot be manually triggered.
Dump and merge
Dump consists of two processes: mini compaction and minor compaction. When the MemTable size exceeds a certain threshold, data in the MemTable needs to be dumped to Mini SSTable to free memory. This process is called mini compaction. As users write more data, the number of Mini SSTables increases. When the number of Mini SSTables exceeds a certain threshold, minor compaction is automatically triggered in the background. Dump (mini compaction) generates new Mini SSTables. When the number of dumps (mini compactions) exceeds a certain threshold, or during off-peak hours each day, the system merges the baseline SSTable (Major SSTable) with the incremental SSTables (Mini/Minor SSTable) produced by subsequent dumps into a new Major SSTable. This process is called merge (major compaction).
For more information about dump, see Dump (Mini and minor compactions).
For more information about merge, see Merge (Major compaction).