ob_cluster_merge_timeout

Description

This alert is triggered when the major compaction of a zone of an OceanBase cluster fails to be completed in three hours.

Principle

The following table describes the key parameters that are involved in the monitoring and alerting logic.

For a major compaction task that is initiated automatically or manually, after the task run time exceeds 10,800 seconds (three hours), the flag bit is set and the major compaction status is updated in the __all_zone table.

The value of the metric indicates the cluster compaction timeout status. Valid values: 0 and 1.

Alert rule

Alert information

Alert templates

• Alert overview: ${alarm_target} ${alarm_name}

• Alert details: ${alarm_target} ${alarm_name}

• Overview example: ob_cluster=C1-1000. OceanBase cluster compaction timed out.

• Details example: ob_cluster=C1-1000. OceanBase cluster compaction timed out.

Impact on the system

When a major compaction times out, the storage pressure of the disk increases. If users go on writing data to it, the disk will be full, blocking services of the server. When you use OceanBase Database of an earlier version, the memory will be used up and the cluster will stop writing.

Possible causes

This problem is commonly found in the following scenarios:

• The OceanBase cluster has a huge amount of data to compact.

• An error occurred with the disk.

• The major compaction efficiency of the disk medium is low.

  When the CPU and memory specifications are the same, the major compaction efficiency depends on the disk medium. For example, a Serial Advanced Technology Attachment (SATA) disk has a lower major compaction efficiency than that of a Solid State Disk (SSD).

Solutions

If the problem is caused by a huge amount of data, modify the timeout value or the number of threads for major compaction. If the problem is caused by a disk failure, replace the faulty node. For more information, see Exception handling for OceanBase cluster compaction.

1. Check whether the major compaction is stuck on the Compaction Management page of the OceanBase Cloud Platform (OCP) console.

   1. Go to the Compaction Management page.

      

   2. View the major compaction details on the Details of Major Compaction tab. The tab displays the estimated progress, total number of partition replicas, and the number of replicas for which major compaction has been completed. Check the number of replicas for which major compaction has been completed every minute. If the number increases, the major compaction is not stuck.

      

   3. View the major compaction statistics. If the elapsed time of major compaction significantly increases compared with the last three major compactions with no business traffic surge in the recent days, an error may have occurred.

      

   • Based on the preceding analysis, if the major compaction is stuck, check for the faulty OBServer. For more information, see Exception handling for OceanBase cluster compaction. If the faulty OBServer must be replaced, replace it by referring to Replace an OBServer.

   • Based on the preceding analysis, if the major compaction is not stuck, go to the next step.

2. Check whether a large amount of business data needs to be compacted or the major compaction efficiency of the disk medium is low. Adjust the timeout value or the number of threads for major compaction. Perform the following steps:

   1. Choose Cluster Overview > Compaction Management > Configuration for Major Compaction > Major Compaction Strategy .

   2. Click Edit .

      Modify the Worker Threads for Major Compaction and Major Compaction Timeout Period parameters.

      

   3. Click Save .