Compared with minor compactions, major compactions are more time-consuming. Based on best practices, a major compaction is performed only once within a day and during off-peak hours. Therefore, a major compaction is also called a daily compaction.
A major compaction compacts all dynamic and static data, which is a time-consuming operation. When the incremental data generated by minor compactions reaches the specified threshold, OceanBase Database performs a major compaction on data of the same major version. The main difference between a minor compaction and a major compaction is that a major compaction compacts data in a tenant with its static data at a unified snapshot point and finally generates a tenant-level snapshot.
| Mini compaction | Minor compaction | Major compaction |
|---|---|---|
| A partition- or tenant-level operation that only persists a MemTable. | A partition-level operation. | A tenant-level operation that generates a tenant-level snapshot. |
| Each tenant of each OBServer node independently performs the minor freeze operation on the MemTable. The primary and secondary partitions can be inconsistent. | Each partition performs minor compactions based on its current number of SSTables. | All partitions in a tenant perform the minor freeze operation on MemTables together, during which the primary and secondary partitions must be consistent. Data consistency is checked during the major compaction. |
| Data rows of different versions may be involved. | Data rows of different versions may be involved. | Only data rows at the snapshot point are involved. |
| A mini compaction persists one or more MemTables into an SSTable. | A minor compaction compacts the incremental data with the minor SSTable of the same major version and generates a new minor SSTable. A minor compaction involves only the incremental data. Special tags are required to mark the rows to be deleted. | A major compaction compacts MemTables and SSTables of the current major version with the full static data of the previous major version and generates a new set of full data. |
While a major compaction is time-consuming, it provides an operation window. During this operation window, OceanBase Database can complete multiple computing-intensive tasks based on the characteristics of major compactions. This improves the overall resource utilization.
Data compression
During a major compaction, OceanBase Database compresses the data twice: semantics-based encoding within the database and general compression by using the specified compression algorithm. In general compression, the encoded data is compressed by using an algorithm such as LZ4. Compression saves storage space and greatly improves query performance. OceanBase Database supports the Snappy, LZ4, LZO, and Zstandard compression algorithms. You can choose a compression algorithm based on the desired compression ratio and decompression duration. MySQL and Oracle also support data compression to some extent. However, due to the design of fixed page lengths in traditional databases, compression leads to inevitable voids in storage. In this case, compression efficiency is compromised. In contrast, compression has a slight impact on data write performance in OceanBase Database, which is built based on the log-structured merge-tree (LSM tree) architecture.
Data verification
Major compactions based on tenant-level consistent snapshots facilitate multi-replica data consistency verification in OceanBase Database. After a major compaction is complete, the replica data can be compared with the baseline data to ensure the consistency of service data among different replicas. In addition, data verification is performed on the primary table and index table based on the snapshot baseline data to ensure data consistency between the two tables.
Schema changes
OceanBase Database can complete schema changes such as column creation and deletion during major compactions. This makes DDL operations smoother.
Major compaction modes
The following section describes different modes of major compactions.
Full compaction
The full compaction mode is the earliest compaction mode of OceanBase Database and is similar to the major compaction process of HBase and RocksDB. During a full compaction, all the current static data is read and compacted with the dynamic data in the memory. The compacted data is written as the new static data to a disk. In this process, all data is rewritten. A full compaction significantly consumes the disk I/O and space. Therefore, unless specified by a database administrator (DBA), OceanBase Database does not initiate full compactions.
Incremental compaction
In the storage engine of OceanBase Database, a macroblock is the basic I/O write unit. Not all macroblocks are modified in many cases. If a macroblock has no incremental modifications, the data of this macroblock can be directly reused during a major compaction. This mode is called incremental compaction in OceanBase Database. This mode greatly reduces the compaction workload. Therefore, it is the default major compaction mode in OceanBase Database. OceanBase Database splits each macroblock into multiple microblocks. Not all microblocks are modified in many cases. Therefore, users can reuse the data of specific microblocks without the need to rewrite data to them. Microblock-level incremental compactions further reduce the major compaction time.
Progressive compaction
Users may need to make DDL modifications such as creating columns, deleting columns, and creating indexes to support rapid business development. These DDL modifications are usually operations that come with high costs for a database. MySQL did not support online DDL modifications until the release of version 5.6. Even now, online DDL modifications in MySQL 5.7 are highly risky for a DBA because a major DDL modification may cause replication lag between the primary and standby MySQL clusters.
Requirements for online DDL modifications are considered in the design of OceanBase Database. DDL modifications such as creating columns, deleting columns, and creating indexes do not block reads or writes, or affect Paxos synchronization among multiple replicas. DDL modifications on columns take effect in real time, but DDL modifications on the stored data are performed during major compactions. For some DDL modifications, such as column creation or deletion, all data needs to be rewritten. Rewriting all data during a single daily compaction is a significant challenge in terms of storage space and compaction time. To resolve this issue, OceanBase Database introduces the progressive compaction mode to distribute data rewrites caused by DDL modifications to multiple daily compactions. For example, if the number of progressive compaction rounds is set to 60, one progressive compaction rewrites only 1/60 of the data. The data rewrite process is complete after 60 rounds. The progressive compaction mode alleviates the load of a DBA in executing DDL statements and makes DDL modifications smoother.
Parallel compaction
OceanBase Database V1.0 and later support partitioned tables. Major compactions are performed in parallel for different data partitions. However, some partitions may contain a large amount of data. Although incremental compactions remarkably reduce the amount of data to be compacted, some frequently updated services may still involve a large amount of data to be compacted. To resolve this issue, OceanBase Database introduces the parallel compaction mode for different partitions. In this mode, data is distributed to different threads for parallel compactions. The compaction efficiency is significantly improved.
Trigger methods of major compactions
Three trigger methods are supported: automatic trigger, scheduled trigger, and manual trigger.
If the number of minor freezes in a tenant exceeds the limit, OceanBase Database automatically triggers a major compaction for the tenant.
You can set parameters to regularly trigger major compactions during off-peak hours every day.
obclient> ALTER SYSTEM SET major_freeze_duty_time = '02:00' TENANT = t1;You can also run the following commands to manually trigger major compactions.
Initiate a major compaction in the
systenant for itself and other tenants.Initiate a major compaction for the
systenant.obclient> ALTER SYSTEM MAJOR FREEZE TENANT = sys;Initiate a major compaction for all user tenants.
obclient> ALTER SYSTEM MAJOR FREEZE TENANT = all_user;Initiate a major compaction for all meta tenants.
obclient> ALTER SYSTEM MAJOR FREEZE TENANT = all_meta;Initiate a major compaction for tenants t1 and t2.
obclient> ALTER SYSTEM MAJOR FREEZE TENANT = t1,t2;
Initiate a major compaction in a user tenant for itself.
obclient> ALTER SYSTEM MAJOR FREEZE;
References
For more information about major compactions, see Overview.