The storage engine of the OceanBase database is based on the LSM-Tree architecture, and the data is roughly divided into two parts: MemTable in memory and SSTable on disk. As user data is written to MemTable, when MemTable exceeds the memory threshold, it is converted to SSTable on disk, causing the number of SSTables to continuously increase. This results in queries needing to access more SSTable files, reducing query efficiency. Therefore, the most core concept of LSM-Tree is introduced: Compaction.
OceanBase database uses a Tiered & Leveled Compaction strategy, dividing SSTables into three layers: L0 uses the tiered mode, while L1 and L2 use the leveled mode.
Compaction Types
The Compaction operation of the LSM-Tree is a reintegration of data, which is essentially a multi-way merge sort. It sorts several SSTables in ascending order based on Rowkey and finally outputs them as one SSTable.
In the OceanBase database, Compaction has the following types:
Mini Compaction: Transform the MemTable into a Mini SSTable. Can be triggered automatically or manually.
Minor Compaction: Combines multiple Mini SSTables into a new Mini SSTable, or merges multiple Mini SSTables with one Minor SSTable into a new Minor SSTable. Triggered by the background scheduling thread and cannot be manually triggered.
Medium Compaction: Combines the original baseline SSTable (Major SSTable) of the specified partition with all Mini SSTables/Minor SSTables in that partition into a new Major SSTable. Can be triggered automatically or manually.
Major Compaction: All original baseline SSTables (Major SSTables) of all partitions under the tenant are merged with all Mini SSTables/Minor SSTables into a new Major SSTable. This can be triggered automatically or manually.
Meta Major Compaction: A special type of Compaction that synthesizes the data before a specified point in time into a Meta Major SSTable. Its data format is the same as that of a Major SSTable, and it does not contain multi-version data or uncommitted transaction data. It is triggered by the background scheduling thread and cannot be manually triggered.
Dump and Merge
Dumping involves two processes: Mini Compaction and Minor Compaction. When the size of the MemTable exceeds a certain threshold, the data in the MemTable needs to be transferred to a Mini SSTable to free up memory, a process referred to as Mini Compaction. As user data is written, the number of Mini SSTables increases. When the number of Mini SSTables exceeds a certain threshold, the system automatically triggers a Minor Compaction in the background. Dumping (Mini Compaction) generates new Mini SSTables. When the number of Mini Compactions exceeds a certain threshold, or during the low-traffic period of the day, the system merges the baseline SSTable (Major SSTable) with the subsequently dumped incremental SSTables (Mini/Minor SSTables) into a new Major SSTable, a process called merging.
For more information about minor compactions, see Minor compactions.
For more information about major compactions, see Major compactions.