ob_cluster_merge_timeout

Description

This alert is triggered when a zone of an OceanBase cluster fails to be compacted in three hours.

Principle

The following table describes the key parameters that are involved in the monitoring and alerting logic.

For a major compaction task that is initiated automatically or manually, after the task run time exceeds 10,800 seconds (three hours), the flag bit is set and the major compaction status is updated in the __all_zone table.

The value of the metric indicates the cluster compaction timeout status. Valid values: 0 and 1.

Alert rule

Alert information

Alert templates

• Overview: /${alarm_target} /${alarm_name}

• Details: /${alarm_target} /${alarm_name}

• Overview example: ob_cluster=C1-1000. OceanBase cluster compaction timed out.

• Details example: ob_cluster=C1-1000. OceanBase cluster compaction timed out.

Impact on the system

When a major compaction times out, the storage pressure of the disk increases. If users go on writing data to it, the disk will be full, blocking services of the server. When you use OceanBase Database of an earlier version, the memory will be used up and the cluster will stop writing.

Possible causes

This problem is commonly found in the following scenarios:

• The OceanBase cluster has a huge amount of data to compact.

• An error occurred with the disk.

• The major compaction efficiency of the disk medium is low.

  When the CPU and memory specifications are the same, the major compaction efficiency depends on the disk medium. For example, a Serial Advanced Technology Attachment (SATA) disk has a lower major compaction efficiency than that of a Solid State Disk (SSD).

Suggested solutions

If the problem is caused by a huge amount of data, modify the timeout value or thread parameters for major compaction. If the problem is caused by a disk failure, replace the corresponding OBServer. For more information, see Exception handling for OceanBase cluster compaction.

1. Check whether the major compaction is stuck on the Compaction Management page of the OCP console.

   1. Go to the Compaction Management page.

      

   2. View the major compaction details on the Details of Major Freeze tab. The following figure shows the estimated end time, total partition replicas, and replicas of partitions after major compaction on this tab. Check the replicas of partitions after major compaction every minute. If the number increases, the major compaction is not stuck.

      

   3. View the major compaction statistics. If the elapsed time of major compaction significantly increases compared with the last three major compactions with no business traffic surge in the recent days, an error may have occurred.

      

   • Based on the preceding analysis, if the major compaction is stuck, check for the faulty OBServer. For more information, see Exception handling for OceanBase cluster compaction.If the faulty OBServer must be replaced, replace it. For more information, see Replace an OBServer.

   • Based on the preceding analysis, if the major compaction is not stuck, go to the next step.

2. Check whether a large amount of business data needs to be compacted or the major compaction efficiency of the disk medium is low.

   Adjust the timeout value or thread number for major compaction. Perform the following steps:

   1. Choose Cluster Overview > Compaction Management > Configuration for Major Freeze > Major Freeze Strategy .

   2. Click Modify .

      Modify the Worker Threads for Major Freeze and Major Freeze Timeout Period parameters.

      

   3. Click Save .