The storage engine of the OceanBase database is based on the LSM-Tree architecture, where data is roughly divided into two parts: MemTable in memory and SSTable on disk. As user data is written to MemTable, when MemTable exceeds the memory threshold, it is converted into SSTable on disk, causing the number of SSTables to continuously increase. This results in queries needing to access more SSTable files, reducing query efficiency. Therefore, the most core concept of LSM-Tree is introduced: Compaction.
OceanBase database uses a Tiered & Leveled Compaction strategy, dividing SSTables into three layers: L0 uses the tiered mode, while L1 and L2 use the leveled mode.
Compaction Types
The Compaction operation of the LSM-Tree is a reintegration of data, which is essentially a multi-way merge sort. It sorts several SSTables in ascending order according to Rowkey and finally outputs them as one SSTable.
In the OceanBase database, Compaction has the following types:
Mini Compaction: Transform the MemTable into a Mini SSTable. Can be triggered automatically or manually.
Minor Compaction: Combines multiple Mini SSTables into a new Mini SSTable, or merges multiple Mini SSTables with one Minor SSTable into a new Minor SSTable. Triggered by the background scheduling thread and cannot be manually triggered.
Medium Compaction: The original baseline SSTable (Major SSTable) of the specified partition is merged with all Mini SSTables/Minor SSTables in that partition to form a new Major SSTable. This can be triggered automatically or manually.
Major Compaction: All original baseline SSTables (Major SSTables) of all partitions under the tenant are merged with all Mini SSTables/Minor SSTables into a new Major SSTable. This can be triggered automatically or manually.
Meta Major Compaction: A special type of Compaction that merges data before a specified point in time into a Meta Major SSTable. Its data format is the same as that of a Major SSTable, and it does not contain multi-version data or uncommitted transaction data. It is triggered by a background scheduling thread and cannot be manually triggered.
Dump and Merge
The dump consists of two processes: Mini Compaction and Minor Compaction. When the size of MemTable exceeds a certain threshold, the data in MemTable needs to be transferred to the Mini SSTable to free up memory, a process referred to as Mini Compaction. As user data is written, the number of Mini SSTables increases. When the number of Mini SSTables exceeds a certain threshold, the system automatically triggers Minor Compaction in the background. The dump (Mini Compaction) generates new Mini SSTables. When the number of dumps (Mini Compactions) exceeds a certain threshold, or during the off-peak business hours each day, the system merges the baseline SSTable (Major SSTable) with the subsequently dumped incremental SSTables (Mini/Minor SSTables) into a new Major SSTable, a process called Merge.
For more information about minor compactions, see Minor compactions.
For more information about major compactions, see Major compactions.