Root cause analysis scenario: card merging major_hold

Applicable scenarios

This article is applicable to card merging scenarios, and locates the problem based on OceanBase database internal log and internal table data analysis.

Currently, OceanBase database V4.0.0.0 and later versions are supported. When using obdiag, you need to configure the cluster information in the ~/.obdiag/config.yml file according to your choice, or configure the cluster information through the --config option in the command.

Usage example

The default execution command is as follows, which only analyzes the reasons for card merging in the current situation.

obdiag rca run --scene=major_hold 

At this stage, the card merging scenario is mainly used for preliminary analysis and positioning and effective information collection. After completion, the collected effective information needs to be packaged and uploaded to the community [Q&A Area] (https://ask.oceanbase.com/) or OceanBase operation and maintenance for further analysis.

Analyze content

Since obdiag V4.0.0, this scenario has enhanced the following analysis capabilities:

1. Duplicate index name detection: Check whether there are duplicate index names that cause the merge to get stuck.
2. DDL task status check: Analyze the execution status of the current DDL task to determine whether the merge is stuck due to DDL task blocking.
3. Log analysis enhancement: Analyze more merge-related error information in OceanBase cluster logs.

Since obdiag V4.1.0, this scenario further enhances the following analysis capabilities:

1. Memory throttling check: Check the tenant's memory usage and memory throttling status to identify merge failures caused by insufficient memory. Includes:
   • Tenant memory usage check
   • MemStore usage and write throttling trigger threshold check
   • TxShare throttling limit check
   • Check for transaction timeout issues related to throttling
2. Dump/Merge/Freeze information query: When the diagnostic information displays medium wait for freeze or major wait for freeze, the following information will be queried and saved:
   • DAG scheduler freeze information (dag_scheduler_freeze.json)
   • Minor Freeze information (minor_freeze_info.json, obtained from __all_virtual_minor_freeze_info view)
   • Provide possible reasons and suggestions for merge pending freeze

Since obdiag V4.2.0, this scenario has the following new analysis capabilities:

1. Column group Schema problem check: Detect the old __co_all record in __all_virtual_column_group_history in the heterogeneous row-column conversion scenario, query the SSTABLE_MAJOR_MERGE task in __all_virtual_sys_task_status, and provide processing steps.