Major compaction

Compared to minor compactions, major compactions are more significant and time-consuming. Best practices recommend performing a major compaction only once per day, ideally during off-peak hours. This is why major compactions are sometimes referred to as daily compactions.

A major compaction merges dynamic and static data, which is a more resource-intensive process. When the incremental data generated by minor compactions accumulates to a certain level, a major freeze is initiated to perform a major compaction. The key difference between a major compaction and a minor compaction is that a major compaction involves merging all partitions of a tenant with their corresponding static data at a unified snapshot point. This results in a tenant-level snapshot.

Although major compactions are time-consuming, they provide a window for the database to perform multiple compute-intensive tasks, thereby improving overall resource utilization efficiency.

• Data compression

  During a major compaction, OceanBase Database performs two layers of compression. The first layer is internal semantic encoding compression, and the second layer is general compression based on the user-specified algorithm, such as lz4. This further reduces the size of the encoded data. Compression not only saves storage space but also significantly improves query performance. OceanBase Database supports compression algorithms such as snappy, lz4, lzo, and zstd, allowing users to balance compression ratio and decompression time. While MySQL and Oracle also support data compression, traditional databases with fixed-size pages inevitably create storage gaps during compression, which affects compression efficiency. More importantly, for an LSM-tree-based storage system like OceanBase Database, compression has almost no impact on write performance.

• Data verification

  Major compactions, which generate a tenant-level snapshot, make it easy for OceanBase Database to verify data consistency across multiple replicas. After a major compaction, replicas can directly compare baseline data to ensure business data consistency. Additionally, this snapshot baseline data can be used to verify data consistency between the main table and index tables.

• Schema changes

  For schema changes such as adding or removing columns, OceanBase Database can perform data change operations during a major compaction, making DDL operations smoother for business operations.

Merge Mode

Merge has several different ways, which are described below.

• Minor compaction

  Minor Compaction is a compaction strategy in OceanBase Database and is similar to the major compaction strategy in HBase and RocksDB. In a minor compaction, the data on the tablet will be read and merged with dynamic data stored in the memory. Then, the merged data is written to disk as the new static data. In this process, all the data will be rewritten. Minor compactions will greatly consume the I/O and space resources. Therefore, OceanBase Database does not actively perform minor compactions unless specified by a DBA.

• Incremental compaction

  In OceanBase Database, a macroblock is the basic unit of I/O in the storage engine. Often, not all macroblocks need to be modified. If no changes are made to a macroblock, it can be directly reused during a major compaction. This process is known as incremental major compaction. By only modifying the data that has changed, incremental major compaction significantly reduces the amount of work required, making it the default compaction method in OceanBase Database. Additionally, OceanBase Database further splits macroblocks into smaller microblocks. In many cases, not all microblocks will be modified. As such, microblocks can be reused without being rewritten, further reducing the time required for major compactions.

• Progressive merge

  As the business grows rapidly, users are inevitably required to perform various DDL operations such as adding, removing, and creating indexes. These DDL operations are typically expensive for the database. MySQL did not support online DDL operations for a long time (Online DDL operations were only moderately supported starting from version 5.6). Even today, performing Online DDL operations in MySQL 5.7 still carries significant risks for DBAs, as a large DDL operation can lead to replication lag between the primary and standby servers.

  OceanBase Database was designed to meet the needs of Online DDL. Today, in OceanBase Database, operations such as adding and dropping columns and creating indexes are not blocking for reads and writes, nor do they impact the Paxos synchronization across replicas. The DDL operations for adding and dropping columns take effect in real time and defer the modification of stored data until the daily compaction. However, for some DDL operations, such as adding and dropping columns, all data must be rewritten. In this case, the daily compaction for rewriting all data would consume a large amount of storage space and take up significant time. To address this issue, OceanBase Database introduces progressive major compactions. It distributes the data rewriting caused by DDL changes across multiple major compactions. For example, if you set the number of progressive rounds to 60, only one-sixtieth of the data is rewritten in one major compaction. After 60 major compactions, all data is rewritten. The progressive major compaction reduces the burden on DBAs when they perform DDL operations and makes DDL changes smoother.

• Windowed: incremental, on-demand, and executed within a specified time window.

  To address the issues of unpredictable merge duration during tenant merges and slow performance in scenarios with a large number of partitions, OceanBase Database introduced the window-based compaction feature for V4.6.x versions starting from V4.6.0. Window-based compaction is an incremental and on-demand compaction executed within a specified time window. It only processes partitions with incremental data that meet specific conditions and schedules the compaction order based on priority. Additionally, it integrates with the DBMS_SCHEDULER system package for managing periodic tasks, supporting timezone-based scheduling and precise control over the merge task duration. This feature allows for fine-grained resource allocation and adaptive adjustment of the number of merge threads, ensuring controllable resource usage during merges.

Merge triggers

There are three types of trigger methods: automatic trigger, scheduled trigger, and manual trigger.

• When the number of minor freezes for a tenant exceeds the threshold, the system automatically initiates a major compaction for the tenant.

• You can set a parameter to trigger compactions during off-peak hours on a daily basis.

  obclient> ALTER SYSTEM SET major_freeze_duty_time = '02:00' TENANT = tenant1;
  

• Merge can also be triggered manually via the following operations and maintenance commands.

  • Tenant-level compactions

    • Initiate a tenant-level major compaction in the sys tenant for itself and other tenants.

      • Initiate a tenant-level major compaction for the sys tenant.

        obclient(root@sys)[oceanbase]> ALTER SYSTEM MAJOR FREEZE TENANT = sys;
        

      • Initiate a tenant-level major compaction for all user tenants.

        obclient(root@sys)[oceanbase]> ALTER SYSTEM MAJOR FREEZE TENANT = all_user;
        

      • Initiate a tenant-level major compaction for all meta tenants.

        obclient(root@sys)[oceanbase]> ALTER SYSTEM MAJOR FREEZE TENANT = all_meta;
        

      • Initiate a tenant-level major compaction for tenants tenant1 and tenant2.

        obclient(root@sys)[oceanbase]> ALTER SYSTEM MAJOR FREEZE TENANT = tenant1,tenant2; 
        

    • Initiate a tenant-level major compaction in a user tenant for itself.

      obclient> ALTER SYSTEM MAJOR FREEZE;
      

  • Table-level compactions

    Assume the table_id of the table to be compacted is 500002:

    • Initiate a table-level major compaction in the sys tenant for tenant1.

      obclient(root@sys)[oceanbase]> ALTER SYSTEM MAJOR FREEZE TENANT = tenant1 TABLE_ID = 500002;
      

    • Initiate a table-level major compaction in a user tenant for itself.

      obclient> ALTER SYSTEM MAJOR FREEZE TABLE_ID = 500002;
      

  • Partition-level compactions

    Assume the tablet_id of the table partition to be compacted is 200007:

    • Initiate a partition-level major compaction in the sys tenant for tenant1.

      obclient(root@sys)[oceanbase]> ALTER SYSTEM MAJOR FREEZE TENANT = tenant1 TABLET_ID = 200001;
      

    • Initiate a partition-level major compaction in a user tenant for itself.

      obclient> ALTER SYSTEM MAJOR FREEZE TABLET_ID = 200007;
      

Related Topics

For more information about merge operations, see Consolidation Management.