OceanBase Database freezes incremental data of an active MemTable and compacts the frozen data with the baseline data for data persistence when the data size of this MemTable reaches the specified threshold. The data structure after the minor compaction is called SSTable.
Macroblock and microblock
OceanBase Database splits the disk into several 2 MB fixed-length data blocks called macroblocks. A macroblock is the basic unit of write I/O operations. An SSTable comprises several macroblocks. Data within a macroblock is grouped into multiple variable-length data blocks with a size of about 16 KB. These blocks are called microblocks. A microblock contains several rows and is the smallest unit of read I/O operations.
Each macroblock has a fixed length, with a default size of 2 MB. Its length cannot be adjusted. A microblock is variable in length and its default size is 16 KB. The sizes here are data sizes before compression. Compressed microblocks are usually smaller than their uncompressed version. In addition, when OceanBase Database performs read I/O operations, it aligns the blocks by taking their sizes as 4 KB. Therefore, the data length of a read I/O operation may be different from that of the microblocks. The length of microblocks can be modified. To set different microblock lengths for different tables, run the following statement:
ALTER TABLE mytest SET block_size =131072;
Generally, a longer microblock length means a higher data compression ratio and a higher I/O read cost. Accordingly, a shorter microblock length means a lower data compression ratio and a lower I/O read cost. The default size of a microblock is 16 KB.
Due to the data deletion or update, a macroblock may not be filled with data. When all the rows in the adjacent macroblocks can be stored in one macroblock, these macroblocks are merged into one macroblock.
Encoding and compression
OceanBase Database uses the data encoding (compression) technology to achieve a high compression ratio. Data encoding provides a series of encoding methods based on the value range and type information of different fields in the database relational table. Data encoding "knows" data better than general compression algorithms. Therefore, it offers a higher compression ratio.
Compressing by column is more efficient than compressing by row in a relational table. Adjacent fields in a column share the same data type and value range, and the data similarity is much higher than that from the perspective of a row. Therefore, OceanBase Database encodes data by column. When data in a column is encoded into macroblocks, OceanBase Database uses the column storage to store it as microblocks. In addition, the read/write splitting architecture of OceanBase Database enables data to be written to disks in batches. This mode is more suitable for data compression than conventional dirty page flush. The encoding only involves the overhead of data compaction in the background. This does not damage the performance of real-time user writes, updates, and deletes. Instead, it optimizes the query performance of some queries. The encoded data can be directly queried without decoding. Even LZ4, the fastest compression algorithm, requires decompression of the entire data block before access. In addition, data encoding allows the same data block to store more data, which in turn improves the I/O performance. Therefore, data encoding is enabled by default for OceanBase data tables.
OceanBase Database provides a variety of data encoding methods, including dictionary encoding, RLE encoding, constant encoding, difference encoding, prefix encoding, and inter-column encoding. It can also select the most appropriate encoding method for each column based on data features. Based on data encoding, OceanBase Database uses a general compression algorithm to further compress the encoded data. OceanBase Database uses the encoding (compression) technology, microblocks of the same size (16 KB), and the unified compression algorithm (LZ4) to process and store data. The same copy of data stored in OceanBase requires less than half of the space required for storage elsewhere, without affecting the performance.