Blocked tenant minor compaction

OceanBase Database is designed based on the LSM-tree architecture. If the memory usage reaches a specified threshold, the system triggers a minor freeze and a minor compaction during which data is dumped to the disk. After the minor compaction, frozen memory is released for processing subsequent requests. Therefore, if the minor compaction of a tenant is blocked, the memory release is affected. This leads to the full usage of memory. This topic describes common scenarios in which minor compactions are blocked and provides solutions.

Check whether a minor compaction is blocked

You can use one of the following two methods to check whether a minor compaction is blocked:

1. Log on to the sys tenant and execute the following statement to check whether frozen MemTables exist. If yes, the minor compaction is not terminated. If a minor compaction is not terminated for a long period of time, the minor compaction is blocked.

   SELECT count(*) FROM __all_virtual_table_mgr WHERE table_type = 0 AND is_active=0;
   

2. If you use OceanBase Database V3.2 or later, query the internal tables of the sys tenant to check the progress of the minor compaction. If the progress remains unchanged for a long period of time, the minor compaction is blocked.

   1. Check the compaction progress of the partition.

      SELECT * FROM __all_virtual_partition_compaction_progress;
      

   2. Check whether exceptions occur in partitions being compacted.

      SELECT * FROM __all_virtual_compaction_diagnose_info;
      

Emergency procedure

The minor compaction of a tenant is likely blocked due to the following causes:

• High I/O of the node

  The high I/O of the node affects the minor compaction of a tenant. In this case, you must resolve the issue that causes high I/O. For more information, see High disk I/O of the node.

• Hardware faults

  Hardware faults of the OBServer node, such as disk and memory failures, can also affect minor compactions. For more information, see Other exceptions related to hardware and network. For specific hardware failures, replace the faulty hardware.

• Long-running transactions

  If a minor compaction is blocked, you can check whether long-running transactions exist in the database and terminate the long-running transactions as needed. For more information about how to troubleshoot long-running transactions, see Troubleshoot transaction errors.

• Full usage of the clog disk

  If the blockage is not caused by long-running transactions, check whether the clog disk is full. For more information, see Full usage of OBServer clog disk.

• Creation of an index

  If the blockage is not caused by any of the preceding reasons, it is possible that an index is being created. In this case, you can cancel index creation and delete the index.