arbitration_auto_replace_operation_info

Description

When an OceanBase cluster cannot connect to its originally associated arbitration service and the automatic switching strategy of the arbitration service group is triggered, the OceanBase cluster automatically switches to another arbitration service in the group. In this case, OCP triggers this alert to remind you that the OceanBase cluster is automatically switching to another arbitration service.

Principle

This alert does not rely on monitoring information. OCP triggers this alert when it initiates a task to automatically switch the arbitration service for an OceanBase cluster.

Alert rule

Alert information

Alert templates

• Overview

  • Template: ${alarm_name}
  • Example: An OceanBase cluster is automatically switching to another arbitration service.

• Details

  • Template: An arbitration service switching task is initiated because the automatic arbitration service switching strategy is triggered for the ${ob_cluster_names} cluster.
  • Example: An arbitration service switching task is initiated because the automatic arbitration service switching strategy is triggered for the clusterA and clusterB clusters.

Impact on the system

An OceanBase cluster automatically switches the arbitration service when it cannot connect to the originally associated arbitration service and the switching strategy of the arbitration service group is triggered. The delay of the automatic arbitration service switching downgrades the high availability of tenants with two or four full-featured replicas in the cluster. This may cause a catastrophic impact on the system.

Possible causes

1. The OceanBase cluster cannot connect to the port of the originally associated arbitration service due to network issues.
2. The server hosting the original arbitration service is down.
3. The original arbitration service process stops running.

Suggested solutions

1. Make sure that the original arbitration service process is running and its port is available for external service.
2. Make sure that the network connection between the OceanBase cluster and the arbitration service is normal.