Compared with minor compactions, major compactions are more time-consuming. Based on best practices, a major compaction is performed only once within a day and during off-peak hours. Therefore, a major compaction is also called a daily compaction.
A major compaction compacts all dynamic and static data, which is a time-consuming operation. When the incremental data generated by minor compactions reaches the specified threshold, OceanBase Database performs a major compaction on data of the same major version. The main difference between a minor compaction and a major compaction is that a major compaction compacts data in all partitions in the tenant with the global static data at a unified snapshot point. A major compaction is a global operation and generates a global snapshot.
| Minor compaction | Major compaction |
|---|---|
| A partition- or tenant-level operation that only persists a MemTable. | A global operation that generates a global snapshot. |
| Each tenant of each OBServer node independently performs the minor freeze operation on the MemTable. The primary and secondary partitions can be inconsistent. | The primary and secondary partitions must be consistent when their MemTables are frozen. Data consistency is checked during the major compaction. |
| Involves data rows of different versions. | Involves data rows at the snapshot point. |
| A minor compaction compacts the incremental data with the minor SSTable of the same major version and generates a new minor SSTable. A minor compaction involves only the incremental data. Special tags are required to mark the rows to be deleted. | A major compaction compacts MemTables and SSTables of the current major version with the full static data of an earlier version and generates a new set of full data. |
While a major compaction is time-consuming, it provides an operation window. During this operation window, OceanBase Database can complete multiple computing-intensive tasks based on the characteristics of major compactions. This improves the overall resource utilization.
Data compression
During major compaction, OceanBase Database compresses data twice: syntax-based encoding within the database and general compression by using the specified compression algorithm. In general compression, the encoded data is compressed by using an algorithm such as LZ4. Compression saves storage space and greatly improves query performance. OceanBase Database supports the Snappy, LZ4, LZO, and Zstandard compression algorithms. Users can choose a compression algorithm based on the desired compression rate and decompression duration. MySQL and Oracle also support data compression to some extent. However, due to the design of fixed page lengths in traditional databases, compression leads to inevitable voids in storage. In this case, compression efficiency is compromised. In contrast, compression has a slight impact on data write performance in OceanBase Database, which is built based on the log-structured merge-tree (LSM tree) architecture.
Data verification
Major compactions based on globally consistent snapshots facilitate multi-replica data consistency verification in OceanBase Database. After a major compaction is complete, the replica data can be compared with the baseline data to ensure the consistency of service data among different replicas. In addition, data verification is performed on the primary table and index table based on the snapshot baseline data to ensure data consistency between the two tables.
Schema changes
OceanBase Database can complete schema changes such as column creation and deletion during major compactions. This makes DDL operations smoother.
Major compaction modes
The following section describes different modes of major compactions.
Full major compaction
The full major compaction mode in OceanBase Database is similar to the major compaction process of HBase and RocksDB. During a full major compaction, all the current static data is read and compacted with the dynamic data in the memory. The compacted data is written as the new static data to a disk. In this process, all data is rewritten. A full major compaction significantly consumes the disk I/O and space. Therefore, unless specified by a database administrator (DBA), OceanBase Database does not initiate full major compactions.
Incremental major compaction
On the storage engine of OceanBase Database, a macroblock is the basic I/O write unit. Not all macroblocks are modified in many cases. If a macroblock has no incremental modifications, the data of this macroblock can be directly reused during a major compaction. This mode is called incremental compaction in OceanBase Database. This mode greatly reduces the compaction workload. Therefore, it is the default compaction mode in OceanBase Database. OceanBase Database splits each macroblock into multiple microblocks. Not all microblocks are modified in many cases. Therefore, users can reuse the data of specific microblocks without the need to rewrite data to them. Microblock-level incremental compactions further reduce the compaction time.
Progressive major compaction
Users may need to make DDL modifications such as creating columns, deleting columns, and creating indexes to support rapid business development. These DDL modifications are often expensive operations for a database. MySQL did not support online DDL modifications until the release of version 5.6. Even now, online DDL modifications in MySQL 5.7 are highly risky for a DBA because a major DDL modification may cause replication lag between the primary and standby MySQL clusters.
Requirements for online DDL modifications are considered in the design of OceanBase Database. DDL modifications such as creating columns, deleting columns, and creating indexes do not block reads or writes, or affect Paxos synchronization among multiple replicas. DDL modifications on columns take effect in real time, but DDL modifications on the stored data are performed during major compactions. For some DDL modifications, such as column creation or deletion, all data needs to be rewritten. Rewriting all data during a single daily compaction is a significant challenge in terms of storage space and compaction time. To resolve this issue, OceanBase Database introduces the progressive major compaction mode to distribute data rewrites caused by DDL modifications to multiple daily compactions. For example, if the number of progressive major compaction rounds is set to 60, one progressive major compaction rewrites only 1/60 of the data. The data rewrite process is complete after 60 rounds. The progressive major compaction mode alleviates the load of a DBA in executing DDL statements and makes DDL modifications smoother.
Parallel major compaction
OceanBase Database V1.0 and later support partitioned tables. Major compactions are performed in parallel for different data partitions. However, some partitions may contain a large amount of data. Although incremental compaction remarkably reduces the amount of data to be compacted, some frequently updated services may still involve a large amount of data to be compacted. To resolve this issue, OceanBase Database introduces the parallel compaction mode for different partitions. In this mode, data is distributed to different threads for parallel major compaction. The compaction efficiency is significantly improved.
Trigger methods of major compactions
Three trigger methods are supported: automatic trigger, scheduled trigger, and manual trigger.
If the number of minor freezes of a tenant exceeds the limit, OceanBase Database automatically triggers a major compaction of the tenant.
You can set parameters to regularly trigger major compactions during off-peak hours every day.
obclient> ALTER SYSTEM SET major_freeze_duty_time = '02:00' TENANT = t1;You can also manually trigger major compactions by running the following O&M command:
The sys tenant initiates a major compaction for other tenants.
obclient> ALTER SYSTEM MAJOR FREEZE TENANT = ALL; // Major compaction for all tenants obclient> ALTER SYSTEM MAJOR FREEZE TENANT = t1,t2; // Major compaction for tenants t1 and t2A user tenant initiates a major compaction of itself.
obclient> ALTER SYSTEM MAJOR FREEZE;
References
For more information about major compactions, see Major compactions.