Merges

Unlike minor compactions, major compactions are more significant and time-consuming. Therefore, it is recommended to perform a major compaction once a day during off-peak hours. A major compaction (major compaction) is a process of merging dynamic and static data. During a major compaction, the database is down for a period of time. To minimize the impact on business, a major compaction is typically scheduled during off-peak hours.

A minor compaction (minor compaction) is a process of merging multiple mini SSTables or a mini SSTable and a minor SSTable into a new minor SSTable, which contains only incremental data. A minor compaction does not interrupt the normal operation of the database.

Although a major compaction is time-consuming, it provides an opportunity for the database to perform multiple CPU-intensive computation tasks based on the major compaction feature, thereby improving the overall resource utilization.

• Data compression

  During a major compaction, OceanBase Database compresses data in two steps. First, it applies semantic compression to the data internally. Then, it uses a general compression algorithm, such as lz4, to compress the encoded data. This two-step compression not only saves storage space but also significantly enhances query performance. OceanBase Database supports (snappy, lz4, lzo, and zstd) compression algorithms, allowing users to balance compression ratio and decompression time. To a certain extent, MySQL and Oracle databases also support data compression. However, due to the fixed-length page design in traditional databases, compression inevitably causes storage fragmentation, which reduces the compression efficiency. In addition, unlike in databases with the LSM-tree architecture, compression has negligible impact on data write efficiency.

• Data verification

  A major compaction allows OceanBase Database to easily verify the data consistency across multiple replicas based on tenant-level consistent snapshots. After the major compaction, the business data in the baseline data of multiple replicas can be directly compared to verify the consistency of the business data across the replicas. In addition, the data consistency between the primary and index tables can be verified based on the same consistent snapshot.

• Schema changes

  For schema changes such as adding or removing columns, the OceanBase database can complete data change operations together during the merge, making DDL operations smoother for businesses.

Merging methods

There are various methods of merging. The following descriptions outline the different methods.

• Full major compaction

  The full major compaction is the original merging method in OceanBase Database. It is similar to the major compaction process in HBase and RocksDB. During a full major compaction, the current static data is read, merged with the dynamic data in memory, and written to the disk as new static data. All data is rewritten during the process. Full major compactions consume a large amount of disk I/O and space. OceanBase Database generally does not perform full major compactions unless specified by the database administrator (DBA).

• Incremental major compaction

  In the storage engine of OceanBase Database, a macroblock is the basic unit of I/O write. In many scenarios, not all macroblocks are modified. When a macroblock is not modified, the data in the macroblock can be reused in the major compaction. This merging method is called an incremental major compaction in OceanBase Database. Incremental major compactions significantly reduce the workload of the major compaction and have become the default major compaction method in OceanBase Database. OceanBase Database further divides data into small microblocks within a macroblock. In many scenarios, not all microblocks are modified. Microblocks can be reused instead of being rewritten. Incremental major compactions at the microblock level further reduce the time required for a major compaction.

• Progressive major compaction

  To support rapid business growth, you must inevitably perform DDL operations such as adding columns, dropping columns, and creating indexes. These DDL operations are usually expensive for the database. For a long time, MySQL did not support online DDL operations (MySQL 5.6 was the first version to support online DDL operations). Even today, performing online DDL operations in MySQL 5.7 still carries risks for database administrators (DBAs), because a large DDL operation can cause replication lag between the primary and standby MySQL databases.

  OceanBase Database considered the need for Online DDL from the initial design phase. Currently, DDL operations such as adding or dropping columns and creating indexes in OceanBase do not block read or write operations and do not affect paxos synchronization between multiple replicas. The DDL changes for adding or dropping columns take effect in real time, deferring the modification of stored data until the daily merge process. However, certain DDL operations like adding or dropping columns require rewriting all the data. Completing this rewrite during a single daily merge would pose significant challenges in terms of storage space and merge duration. To address this issue, OceanBase Database introduces progressive merging, which spreads the data rewrite caused by DDL changes across multiple daily merges. For instance, if the progressive rounds are set to 60, each merge will only rewrite one-sixtieth of the data, and after 60 rounds of merging, the entire dataset will have been rewritten. Progressive merging reduces the burden on DBAs when performing DDL operations and also makes DDL changes smoother.

• Parallel major compaction

  OceanBase Database supports partitioned tables in version 1.0 and later. Major compactions for different data partitions are performed in parallel. However, data skew can result in large data volumes in some partitions. Even with incremental major compaction, major compactions can consume a large amount of disk I/O and time in some scenarios, especially when the business is frequently updated. To address this, OceanBase Database introduced parallel minor compaction within a partition. The data in a macroblock is divided into multiple parts and merged with multiple threads. This significantly speeds up the major compaction process.

Triggers

Major compactions can be triggered automatically, on a scheduled basis, or manually.

• When the number of minor freezes in a tenant exceeds the threshold, a major compaction is automatically triggered for the tenant.

• You can set parameters to schedule a major compaction during off-peak hours every day.

  obclient> ALTER SYSTEM SET major_freeze_duty_time = '02:00' TENANT = t1;
  

• You can manually trigger a major compaction by using the following O&M commands.

  • Initiate a major compaction for a tenant in the sys tenant

    obclient> ALTER SYSTEM MAJOR FREEZE TENANT = ALL; // Initiate a major compaction for all tenants
    
    obclient> ALTER SYSTEM MAJOR FREEZE TENANT = t1,t2; // Initiate a major compaction only for tenants t1 and t2
    

  • Initiate a major compaction for a user tenant

    obclient> ALTER SYSTEM MAJOR FREEZE;
    

References

For more information about major compactions, see Major compaction.