The storage engine of the OceanBase database is based on the LSM-Tree architecture, and the data is roughly divided into two parts: MemTable in memory and SSTable on disk. As user data is written to MemTable, when MemTable exceeds the memory threshold, it is converted into SSTable on disk, and the number of SSTables will continue to increase. This leads to more SSTable files that queries need to access, reducing query efficiency. Therefore, the most core concept of LSM-Tree is introduced: Compaction.
OceanBase database uses a Tiered & Leveled Compaction strategy, dividing SSTables into three layers: L0 uses the tiered mode, while L1 and L2 use the leveled mode.
Compaction Types
The Compaction operation of the LSM-Tree is a reintegration of data, which is essentially a multi-way merge sort. It sorts several SSTables in ascending order according to the Rowkey and finally outputs them as one SSTable.
In the OceanBase database, Compaction has the following types:
Mini Compaction: Transform the MemTable into a Mini SSTable. It can be triggered automatically or manually.
Minor Compaction: Combines multiple Mini SSTables into a new Mini SSTable or merges multiple Mini SSTables with one Minor SSTable into a new Minor SSTable. Triggered by the background scheduling thread, cannot be triggered manually.
Medium Compaction: Combines the original baseline SSTable (Major SSTable) of the specified partition with all Mini SSTables/Minor SSTables in that partition into a new Major SSTable. Can be triggered automatically or manually.
Major Compaction: All original baseline SSTables (Major SSTables) of all partitions under the tenant are merged with all Mini SSTables/Minor SSTables into a new Major SSTable. This can be triggered automatically or manually.
Meta Major Compaction: A special type of Compaction that synthesizes the data before a specified point in time into a Meta Major SSTable. Its data format is the same as that of a Major SSTable, and it does not contain multi-version data or uncommitted transaction data. It is triggered by a background scheduling thread and cannot be manually triggered.
Dump and Merge
The dump consists of two processes: Mini Compaction and Minor Compaction. When the size of MemTable exceeds a certain threshold, the data in MemTable needs to be transferred to Mini SSTable to free up memory, a process referred to as Mini Compaction. As user data is written, the number of Mini SSTables increases. When the number of Mini SSTables exceeds a certain threshold, the system automatically triggers Minor Compaction in the background. The dump (Mini Compaction) generates new Mini SSTables. When the number of dumps (Mini Compactions) exceeds a certain threshold, or during the low-traffic period of the day, the system merges the baseline SSTable (Major SSTable) with the subsequently dumped incremental SSTables (Mini/Minor SSTables) into a new Major SSTable, a process called Merge.
For more information about minor compactions, see Minor compactions.
For more information about major compactions, see Major compactions.