Achieve high availability by using arbitration service groups

Scenarios

OceanBase Database V4.1 provides arbitration service capabilities to achieve high availability with two replicas across two IDCs (2F1A) at low costs. However, in this deployment mode, the high availability of the A replica cannot be guaranteed. If the A replica becomes unavailable due to process exceptions or server failures, OceanBase Database cannot maintain high availability. To address this issue, OCP V4.3.5 introduces the high availability capability of arbitration service groups.

What is an arbitration service group?

An arbitration service group contains multiple arbitration services. When an arbitration service becomes unavailable, OCP automatically switches the associated cluster to another arbitration service in the group.

An arbitration service group consists of two parts: arbitration services and switching strategies.

• Arbitration services in the group have priority levels. When the associated OceanBase cluster triggers a switching strategy, it will prioritize switching to a higher-priority arbitration service. If the highest-priority resource is insufficient, it will transfer some OceanBase clusters to the next-priority arbitration service.

• Switching strategies are divided into two parts: unit disconnection strategy and duration.

  • The unit disconnection strategy includes the following three options:

    1. Default strategy: When half of the replicas of tenants in the OceanBase cluster are disconnected from the arbitration service.
    2. Number of units: When a specified number of Unit Servers in the OceanBase cluster are disconnected from the arbitration service.
    3. Percentage of units: When a specified percentage of Unit Servers in the OceanBase cluster are disconnected from the arbitration service.

  • Duration: The time interval after which the OceanBase cluster triggers the unit disconnection strategy before implementing the switch.

Prerequisites

1. At least one arbitration service exists in OCP.
2. The specified arbitration services must be in the normal state when creating an arbitration service group.
3. The versions of the arbitration services in the group must be consistent.

Technical principles

The architecture for checking and switching arbitration services is as follows:

OCP Agent reporting rules: The monagent process on the cluster RS node reports the connection status between the cluster and the arbitration service to MonitorDB every 5 seconds.

OCP Server checking rules:

1. Shard the arbitration service groups and have each OCP-Server check the arbitration service groups in its shard.
2. Every 10 seconds, query MonitorDB to obtain the list of all servers disconnected from the arbitration service. Then, based on the user-defined switching strategy, determine whether the number of disconnected units meets the switching conditions.

Considerations

• When the cluster is in an abnormal or maintenance state, OCP will not initiate automatic arbitration service switching.

• At least one additional arbitration service must be running normally, with sufficient resources and a stable network connection to the OceanBase cluster. Otherwise, OCP will not initiate automatic switching. The criteria for sufficient resources are as follows:

  1. The memory usage rate of the target arbitration service server must be ≤ 90%.
  2. The disk space of the target arbitration service server must meet the requirements for 2F or 4F tenants: 12 MiB for 2F tenants and 24 MiB for 4F tenants. Check whether the resources are sufficient when migrating to this arbitration service server.

Procedure

1. Log in to OCP.

2. In the left-side navigation pane, click Cluster and go to the Arbitration Service tab.

3. Click the Service Group tab to go to the Service Group list page.

4. Click Create Service Group in the upper-right corner.

5. In the Create Service Group panel, set the basic information of the arbitration service group.