The storage engine of OceanBase Database is built on the LSM-tree architecture, which divides data into baseline data stored in SSTables and incremental data stored in MemTables. SSTables are read-only and stored on disk. MemTables support read and write operations and store data in memory. During a database DML operation (insert, update, or delete), the data changes are first written into the MemTable. When the MemTable reaches a specified size, the data in the MemTable is flushed to the disk to become an SSTable. During a query, the storage engine queries both an SSTable and a MemTable and merges the query results from the two tables to return the merged query result to the SQL layer. The storage engine also maintains a block cache and a row cache in memory to avoid random reads of baseline data.
When the incremental data in memory reaches a specified size, the incremental data and baseline data are merged and the incremental data is written to the disk. In addition, every night during off-peak hours, the system automatically performs a daily major compaction.
The storage engine of OceanBase Database is a baseline-incremental storage engine that combines the advantages of the LSM-tree architecture and traditional relational database storage engines.
A traditional database divides data into many pages. OceanBase Database also adopts this approach and divides data files into macroblocks of 2 MB each. Each macroblock is divided into several variable-length microblocks. During a major compaction, data is reused based on the macroblock. Unupdated macroblocks are not reopened for reading. This approach aims to minimize write amplification and significantly reduce the cost of a major compaction compared with that in a traditional LSM-tree database.
OceanBase Database is designed to store baseline data and incremental data. Each query needs to read both the baseline data and incremental data. To optimize single-row queries, OceanBase Database caches not only data blocks but also rows in memory. A row cache can significantly speed up single-row queries. For "empty queries" that return no rows, OceanBase Database builds a bloom filter and caches it in memory. Most operations in OLTP business are small queries. By optimizing small queries, OceanBase Database avoids the overhead of parsing an entire data block in a traditional database and achieves performance comparable to that of a memory database. In addition, baseline data is read-only and stored continuously. Therefore, OceanBase Database can use more aggressive compression algorithms to achieve a high compression ratio without compromising query performance. This greatly reduces costs.
By integrating advantages from the classical database and the LSM-tree database, OceanBase Database provides a more general LSM-tree-based storage engine for relational databases and offers the following advantages:
Low costs. Leveraging the fact that data written based on the LSM-tree architecture is not updated, OceanBase Database uses self-developed hybrid row-column encoding and general compression algorithms to achieve a compression ratio that is 10 times or even higher than that in traditional databases.
Ease of use. Unlike other LSM-tree databases, OceanBase Database ensures the proper execution or rollback of large or long transactions through the disk-based storage of active transactions. It also offers a multi-level major compaction and minor compaction mechanism to help users find the optimal balance between performance and space.
High performance. To ensure low response latency for common point queries, the storage engine uses multi-level caches to accelerate data access. For range scans, the storage engine leverages data encoding features to push query filter conditions to the encoding layer and provides native vectorized support.
High reliability. In addition to end-to-end data verification, OceanBase Database also verifies the correctness of user data by comparing replicas in global major compactions and by comparing primary and index tables. It also provides a background thread to periodically scan the system to preemptively identify and correct silent data errors.
Components of the storage engine
The storage engine of OceanBase Database can be divided into the following components based on their features.
Data storage
Data organization
Like other LSM-tree databases, OceanBase Database stores data in incremental MemTables and static SSTables. MemTables are writable and stored in memory, while SSTables are read-only and stored on disk. DML operations such as INSERT, UPDATE, and DELETE are first written to MemTables. Once MemTables reach a certain size, the data in these MemTables is flushed to disk to generate SSTables.
In OceanBase Database, SSTables are further classified into mini SSTables, minor SSTables, and major SSTables. MemTables store data in mini SSTables on disk through mini compaction. When the number of mini SSTables reaches a specified threshold, these mini SSTables are merged into one or more minor SSTables through minor compaction. At the start of the daily major compaction specific to OceanBase Database, all minor SSTables and mini SSTables in a partition are merged with the baseline SSTable (major SSTable) in the partition into a new major SSTable.
Storage structure
In OceanBase Database, each partition's basic storage unit is an SSTable, and the basic storage grain is a macroblock. When the database starts, the data file is divided into macroblocks of a fixed size of 2 MB. Each SSTable is actually a collection of macroblocks.
Each macroblock is divided into multiple microblocks. The concept of microblocks is similar to that of pages or blocks in a traditional database. However, microblocks in OceanBase Database are variable-length and can be compressed. The size of microblocks can be specified by the
block_sizeparameter when the table is created.Microblocks can be stored in encoding or flat format based on the specified storage format of the user. Microblocks stored in encoding format have data that is hybridly stored in row and column modes. For microblocks stored in flat format, all data rows are stored in a flattened manner.
Compression and encoding
OceanBase Database uses compression and encoding to compress data within microblocks based on the compression and encoding mode specified for the table. When encoding is enabled for the table, data in each microblock is encoded by column. Several encoding rules, such as dictionary, run-length, constant, and delta encoding, are supported. After the compression of each column is completed, the columns are further encoded based on rules such as equal value and sub-string encoding. This helps greatly reduce the space occupied by data and accelerate subsequent queries by extracting features within columns.
After compression and encoding, OceanBase Database allows you to use a specified general lossless compression algorithm to compress the data in microblocks, further improving the compression ratio.
Minor and major compactions
Minor compaction
A minor compaction, also known as a mini compaction, is initiated when the memory usage of MemTables exceeds the specified threshold. Data in the MemTable is flushed to disk to generate a mini SSTable. As user data is written, the number of mini SSTables increases. When the number of mini SSTables exceeds the specified threshold, a minor compaction is automatically triggered to merge these mini SSTables into one or more minor SSTables.
Major compaction
A major compaction, also known as a daily major compaction in OceanBase Database, differs from those in other LSM-tree databases. As the name suggests, the major compaction was originally designed to be performed on a daily basis at 2 a.m. for the entire cluster. A major compaction is initiated by the RS of each tenant based on the write status or user settings. Each time a major compaction is performed in a tenant, a global snapshot is selected. Then a major compaction is performed on all partitions in the tenant using the data of the snapshot. This generates SSTables for all data in the tenant based on the same snapshot. This mechanism helps users regularly integrate incremental data and improve read performance. In addition, the global consistent snapshot provides a natural data verification point. OceanBase Database can perform multi-dimensional physical data verification of multi-replica data and primary/foreign key indexes based on the global consistent snapshot.
Queries
Insert
In OceanBase Database, all data tables, including heap tables without primary keys, are treated as index cluster tables. Therefore, a hidden primary key is maintained for a heap table internally. When you insert data, the system checks whether the data with the same primary key already exists in the current data table before writing the new user data into the MemTable. To speed up the repeated primary key query, a background thread asynchronously schedules the construction of a Bloom filter for each SSTable at different macroblock-level repletion rates.
Update
As an LSM-tree database, OceanBase Database inserts a new row of data for each update. The data updated in a MemTable includes only the new values of the updated columns and the primary key of the updated row. Therefore, an updated row does not necessarily contain all columns of the table. During continuous background compactions, incremental updates are merged to speed up queries.
Delete
Similar to an update, a delete operation writes a row containing the primary key of the deleted row and a deletion mark into the table instead of directly acting on the original data. The SQL layer filters data based on the filter condition pushed down from the application layer and returns only the filtered data, thus accelerating large-scale queries. In an LSM-tree database, a large number of delete operations are not friendly because they may lead to a large amount of redundant data even after a data range is completely deleted. To address this, OceanBase Database implements inherent range deletion marking logic to avoid this issue. Additionally, you can explicitly specify table modes to enable efficient minor and major compactions, accelerating the deletion process and queries.
Query
When you query for a row of data, the system traverses all MemTables and SSTables from new to old based on the version and fuses the data corresponding to the primary key. During data access, the cache is used to accelerate data retrieval. For large queries, the SQL layer pushes down filter conditions to the storage layer and uses data characteristics for quick filtering. Vectorized operations support batch calculations and results returns in scenarios with a high degree of parallelism.
Multi-level cache
To enhance performance, OceanBase Database supports multi-level cache systems. It provides a block cache for query data microblocks, a row cache for each SSTable, a fuse row cache for merged query results, and a bloomfilter cache for empty check during insertion. All caches in the same tenant share the memory space. When the write speed of MemTables is too fast, the system can flexibly reclaim memory from other caches for write.
Data verification
As a financial-grade relational database, OceanBase Database prioritizes data quality and security. Data verification is performed on every part of the data persistence layer of the full data transmission chain, and the inherent advantage of multi-replica storage, namely, inter-replica data verification, is used to further validate the overall data consistency.
Logical verification
In common deployment modes, each user table in OceanBase Database has multiple replicas. During the daily major compaction of the tenant, all replicas generate baseline data based on the same global snapshot version. After the major compaction is completed, the system compares the checksums of all replicas to ensure data consistency. In addition, the system compares the checksums of index columns based on user table indexes to ensure that the data returned to users is correct and error-free.
Physical verification
For data storage, OceanBase Database records the corresponding checksums at the level of microblocks, macroblocks, SSTables, and partitions for each data file. The data is verified each time it is read; to prevent errors caused by underlying storage hardware, the system re-verifies the data after the data is written into a macroblock during a minor compaction. In addition, a data scan and verification thread runs in the background on each server to scan and verify the overall data periodically, thus detecting silent disk errors early.