The storage engine of the OceanBase database is based on the LSM-Tree architecture, where data is broadly divided into two parts: MemTable in memory and SSTable on disk. As user data is written to MemTable, when the MemTable exceeds the memory threshold, it is converted into an SSTable on disk. The number of SSTables will continue to increase, leading to more SSTable files that queries need to access, which reduces query efficiency. Therefore, the most core concept of LSM-Tree is introduced: Compaction.
OceanBase database uses a Tiered & Leveled Compaction strategy, dividing SSTables into three layers: L0 uses the tiered mode, while L1 and L2 use the leveled mode.
Compaction Types
The Compaction operation of the LSM-Tree is a reintegration of data, which is essentially a multi-way merge sort. It sorts several SSTables in ascending order based on Rowkey and finally outputs them as one SSTable.
In the OceanBase database, Compaction has the following types:
Mini Compaction: Transform the MemTable into a Mini SSTable. Can be triggered automatically or manually.
Minor Compaction: Combines multiple Mini SSTables into a new Mini SSTable, or merges multiple Mini SSTables with one Minor SSTable into a new Minor SSTable. Triggered by the background scheduling thread, and cannot be manually triggered.
Medium Compaction: The original baseline SSTable (Major SSTable) of the specified partition is merged with all Mini SSTables/Minor SSTables in that partition to form a new Major SSTable. This can be triggered automatically or manually.
Major Compaction: All original baseline SSTables (Major SSTables) of all partitions under the tenant are merged with all Mini SSTables/Minor SSTables into a new Major SSTable. This can be triggered automatically or manually.
Meta Major Compaction: A special type of Compaction that synthesizes the data before a specified point in time into a Meta Major SSTable. Its data format is the same as that of a Major SSTable, and it does not contain multi-version data or uncommitted transaction data. It is triggered by a background scheduling thread and cannot be manually triggered.
Dump and Merge
The dump consists of two processes: Mini Compaction and Minor Compaction. When the size of MemTable exceeds a certain threshold, the data in MemTable needs to be transferred to Mini SSTable to free up memory, a process called Mini Compaction. As user data is written, the number of Mini SSTables increases. When the number of Mini SSTables exceeds a certain threshold, the system automatically triggers Minor Compaction in the background. The dump (Mini Compaction) generates new Mini SSTables. When the number of dumps (Mini Compactions) exceeds a certain threshold, or during the low business peak period each day, the system merges the baseline SSTable (Major SSTable) with the subsequently dumped incremental SSTables (Mini/Minor SSTables) into a new Major SSTable, a process called Merge.
For more information about minor compactions, see Minor compactions.
For more information about major compactions, see Major compactions.