Troubleshoot merge exceptions

Merge exceptions are one of the common issues in test environments. Merge exceptions may cause slow queries, abnormal disk space usage, DDL not taking effect, Checksum errors, and other problems.

To help you quickly locate the root cause of the problem and efficiently resolve it, this topic summarizes a clear and practical merge exception troubleshooting procedure. This procedure provides clear operation steps, aiming to improve problem handling efficiency, minimize the impact on business, and provide strong support for daily operations.

Procedure description

1. View the system tables and the RS layer merge initiation status.

   1. View the tenant merge status. In the sys tenant, view the CDB_OB_MAJOR_COMPACTION view to confirm the merge status of each tenant.

      obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_MAJOR_COMPACTION;
      

      Based on the query result, if LAST_SCN = GLOBAL_BROADCAST_SCN, it means that the version number of the last completed merge is the same as the version number of the current merge, and this round of merge has ended. Otherwise, further confirmation is needed based on other fields:

      • If the IS_SUSPENDED column is True, it means that the merge has been manually suspended. You need to execute the SQL statement ALTER SYSTEM RESUME MERGE TENANT tenant_name; to resume it.

      • If STATUS remains COMPACTING, it means the merge is stuck and further investigation is needed.

      • If IS_ERROR is non-null, further investigation is needed. If the value of IS_ERROR is CHECKSUM_ERROR, it means that a Checksum verification inconsistency has occurred in the tenant.

   2. Query the merge progress of all tablets in the current tenant and filter the records that have not been merged to the specified version.

      The logic for determining the end of a merge is to check whether all tablets in the tenant (filtering out some tablets that do not meet the requirements) have been merged to the specified version number. As long as one tablet has not been merged to the specified version, the merge of the tenant will not end. Next, you need to check which tablets in the tenant have not been merged to the specified version.

      Query the CDB_OB_TABLET_REPLICAS and CDB_OB_MAJOR_COMPACTION views, taking tenant_id = 1004 as an example.

      obclient(root@sys)[oceanbase]> SELECT 
          t.svr_ip,
          t.svr_port,
          t.tenant_id,
          t.ls_id,
          t.tablet_id,
          t.compaction_scn AS tablet_version,
          m.global_broadcast_scn AS target_version
      FROM 
          oceanbase.CDB_OB_TABLET_REPLICAS t,
          oceanbase.CDB_OB_MAJOR_COMPACTION m
      WHERE 
          t.tenant_id = m.tenant_id
          AND t.tenant_id = 1004
          AND m.global_broadcast_scn > 0
          AND (t.compaction_scn < m.global_broadcast_scn)
      ORDER BY t.compaction_scn;
      

      Based on the query result:

      • If the query result is Null, it means the merge is in the RS verification phase and the merge is normal.

      • If the query result is not Null, the queried tablets are those that have not been merged to the specified version.

        The compaction_scn field comes from the CDB_OB_TABLET_REPLICAS view and represents the maximum SCN at which the current tablet has completed the merge. The global_broadcast_scn field comes from the CDB_OB_MAJOR_COMPACTION view and represents the target version of this round of merge. If compaction_scn < global_broadcast_scn, it means there are tablets that have not completed the merge and further investigation is needed.

   3. If there are tablets that have not completed the merge, observe whether there are tablets currently executing merge tasks.

      Query the GV$OB_TABLET_COMPACTION_PROGRESS view, taking tenant_id = 1004 as an example.

      obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.GV$OB_TABLET_COMPACTION_PROGRESS WHERE tenant_id = 1004 AND type = "MAJOR_MERGE" AND status LIKE "%RUNNING%";
      

      Based on the query result, if there are tablets currently executing merges, you can wait for the merge to complete. After waiting for a period of time, if no tablets are in the execution process and the number of uncompleted merge tablets found through the CDB_OB_TABLET_REPLICAS and CDB_OB_MAJOR_COMPACTION views has not decreased, you can query the merge diagnostic view to further locate the problem.

2. Query the merge diagnostic view to obtain diagnostic information.

   The merge diagnostic view contains various abnormal error information during the merge process, which can be used to assist in troubleshooting. Taking tenant_id = 1004 and the Tablet_ID = 200001 of the uncompleted merge tablet as an example.

   obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.GV$OB_COMPACTION_DIAGNOSE_INFO WHERE TENANT_ID = 1004 AND TABLET_ID = 200001\G
   

   Based on the information displayed in the diagnose_info field, proceed with the following handling:

   • weak read ts is not ready or slave_read_version is less than freeze_ts, can not merge: Indicates that the standby read has not passed the major point, causing a merge exception. Contact technical support for assistance.

   • memtable can not minor merge: Indicates that the MemTable has not reached the dump condition, causing slow Mini Compaction, slow clog reclamation, and memory explosion. Contact technical support for assistance.

   • sstable count is not safe: Indicates that the number of SSTables is too large, causing Mini compaction to fail, slow clog reclamation, and memory explosion. Contact technical support for assistance.

     Meanwhile, diagnose_info will also display the reasons for the excessive number of SSTables:

     • SNAPSHOT_FOR_MAJOR: Major Compaction

     • SNAPSHOT_FOR_DDL: Table DDL, such as index creation.

     • SNAPSHOT_FOR_MULTI_VERSION: Too much multi-version data, the value of undo_retention is too large.

   • dag may hang: The DAG has not been updated for a period of time and may be stuck. If this state persists for more than 10 minutes, contact technical support personnel for assistance.

   • freeze_info is invalid: The freeze_info has not been flushed, causing a merge exception. Contact technical support personnel for assistance.

   • memtable rec_log_ts not stable: During the log stream freeze process, the continuous maximum callback (replay) point has not pushed past the rec_log_ts of the corresponding tablet's MemTable for a long time. The deeper reason is slow callback (replay). Contact technical support personnel for assistance.

   • traverse_trans_to_submit_redo_log failed: Failed to submit redo log. Contact technical support personnel for assistance.

   • trans table has not merged, can not schedule minor merge: In V3.2.x versions, the data table can initiate a dump only after the transaction status table dump is successful, which may cause the data table not to dump. Contact technical support personnel for assistance.

   • memtable not ready for flush: During the log stream freeze process, some MemTables have not reached the dump condition for a long time, which may cause dump timeout and memory explosion. Contact technical support personnel for assistance.

   • memtable no destroy after release: The MemTable has not been destroyed for a long time after being released. The main reason is a MemTable reference count leak. Contact technical support personnel for assistance.

   • memtable can not create dag successfully: After the MemTable reaches the dump condition, the dump task has not been successfully initiated for a long time. Contact technical support personnel for assistance.

   • compaction has finished in storage: The storage layer merge has ended but the subsequent RS processing has not been completed. Contact technical support personnel for assistance.

   • there is too many tablets. tablet count xxx: Too many partitions need to be traversed during diagnosis, and the diagnosis of this LS has been skipped.

3. Based on the error information in the merge diagnostic view, the problems can be classified. Specific problems can be referred to the relevant documents or typical cases for troubleshooting.

References

• View merge information

• View dump information

• GV$OB_COMPACTION_DIAGNOSE_INFO