This topic describes how to use the arbitration service of OceanBase Database to achieve high availability (HA) in a scenario with two IDCs.
Scenarios
Usually, only two IDCs are available for external users of OceanBase Database in the same region. To achieve IDC-level disaster recovery, you can deploy OceanBase Database in three IDCs across two regions.
Two full-featured replicas are deployed in each of two IDCs in Zone 1 and Zone 2 of Region 1, and one log replica is deployed in the IDC in Zone 3 of Region 2. Therefore, if an IDC in either Zone 1 or Zone 2 fails, three out of the five Paxos replicas can still operate to ensure service availability.
In the preceding deployment mode, you may encounter the following problems in actual disaster recovery:
Increased service response time (RT) following an IDC failure
When the IDCs in Zone 1 and Zone 2 operate properly, the majority of Paxos replicas need to reach a consensus only in the same region, and the delay of a single log synchronization task is less than 1 ms.
When an IDC in either Zone 1 or Zone 2 fails, the majority of Paxos replicas need to reach a consensus across regions, and the delay of a single log synchronization task is 5 ms to 10 ms.
Higher delay in synchronization greatly compromises user experience.
To resolve this problem, OceanBase Database allows you to reduce the number of replicas from 5 to 3. For more information, see Reduce the high availability of OceanBase clusters and tenants.
Limited cross-region network bandwidth resources
To ensure proper operation of OceanBase Database in three IDCs across two regions, Region 1 and Region 2 must have sufficient network bandwidth resources. Otherwise, logs in Region 2 will lag behind during peak hours. If an IDC in Region 1 fails before log synchronization in Region 2 is completed, the entire cluster becomes unavailable. Cross-region network bandwidth resources are scarce and costly. Most external users must consider the required bandwidth when evaluating whether to use OceanBase Database. However, no effective or proper solution was provided for this problem.
OceanBase Database is typically deployed in 2F+1L mode (two full-featured replicas and one log replica). The three replicas provide HA disaster recovery based on Paxos. The one log replica reduces the costs of data storage. However, in this deployment mode, log storage costs and CPU resource overheads of the OBServer node where the log replica resides degrade the competitiveness of OceanBase Database.
Therefore, OceanBase Database must replace the 2F+1L mode with the 2F+1A mode (two full-featured replicas and one arbitration replica) to further reduce total costs. Compared with the synchronization models of MySQL-based cloud databases provided by peer vendors, the arbitration service of OceanBase Database supports lossless active-active disaster recovery in the event of any OBServer node or IDC failure. This significantly enhances the competitiveness of OceanBase Database.
The following figure shows the 2F+1A deployment mode in a scenario with two IDCs.
Prerequisites
- The version of the OceanBase cluster is later than V4.1.0.0.
- The version of the arbitration replica is the same as or slightly later than that of the full-featured replica, to ensure compatibility in the RPC format.
- The OceanBase Database tenant is deployed with two or four full-featured replicas and one arbitration replica. That is, only the 2F+1A and 4F+1A modes are supported.
Procedure
Step 1: Create an arbitration service
Log on to the OceanBase Cloud Platform (OCP) console.
In the left-side navigation pane, click Clusters . The Clusters tab automatically appears.
Click the Arbitration Service tab to go to the Arbitration Service List page.
Click Create Arbitration Service in the upper-right corner.
In the Create Arbitration Service panel, configure the basic information of the arbitration service.
Click OK. In the window that appears, click the task ID to view the task execution status.
The system verifies the parameter settings. If the verification fails, you can perform troubleshooting or modify the parameters as prompted.
Step 2: Associate the arbitration service with an OceanBase cluster
Log on to the OCP console.
In the left-side navigation pane, click Clusters. The Clusters tab automatically appears.
Find the target cluster and click its name.
On the Overview page of the cluster, go to the Add Arbitration Service panel in either of the following ways:
- Click the ... icon in the upper-right corner and select Add Arbitration Service from the menu.
- Click Add Arbitration Service in the Arbitration Service List section.
In the Add Arbitration Service panel, select an arbitration service endpoint for the cluster.
Click Submit.
Step 3: Enable the arbitration service for a tenant
Log on to the OCP console.
In the left-side navigation pane, click Tenants. The Tenants tab automatically appears.
Find the target tenant and click its name.
In the Basic Information section on the Overview page of the tenant, verify that the tenant has two or four full-featured replicas, and enable Arbitration Service.
FAQ
The system returned an error message when I upgraded the OceanBase cluster, indicating that the cluster version xx cannot be upgraded to the arbitration service version xx because version compatibility cannot be ensured. What should I do?
In the following scenarios, versions of the OceanBase cluster and the arbitration service must be compatible. Therefore, when you upgrade an OceanBase cluster associated with an arbitration service, you must check the version compatibility between them.
- Scenario 1: An OceanBase cluster of Version B is associated with an arbitration service of Version A.
- Scenario 2: An arbitration service of Version A is selected when an OceanBase cluster of Version B is created.
- Scenario 3: The arbitration service for an OceanBase cluster of Version B is replaced with a target arbitration service of Version A.
- Scenario 4: An OceanBase cluster associated with an arbitration service of Version A is upgraded to Version B.
- Scenario 5: An arbitration service associated with an OceanBase cluster of Version B is upgraded to Version A.
Compatibility rules for versions A and B are as follows:
- When both level-1 and level-2 version numbers in A and B are the same, A and B are compatible. For example, both A and B are V4.2.x.x.
- When the level-1 and level-2 version numbers in A are earlier than those in B, A and B are incompatible. For example, A is V4.1.x.x, and B is V4.2.x.x.
- When the level-1 and level-2 version numbers in A are later than those in B, if B can be directly upgraded to A without any intermediate versions, A and B are compatible; otherwise, A and B are incompatible.
The system returned an error message when I enabled the arbitration service for a tenant, indicating that the arbitration service cannot be enabled for the tenant because the CPU utilization is {0}%, the memory usage is {1}%, and the available clog disk space is {2} MB. These values do not meet the required conditions: CPU utilization must be lower than {3}%, memory usage must be lower than {4}%, and available clog disk space must be greater than {5} MB. What should I do?
The arbitration service enabled for each tenant consumes CPU and memory resources on the host. To ensure HA of the arbitration service, OCP checks the available resources on the host when you enable the arbitration service for a tenant. You can modify the following system parameters to skip this check:
ocp.arbitration.min.remain.disk.size: the minimum size inMBof the clog disk on the arbitration service host required when the arbitration service is enabled for the tenant. The data format(a,b)indicates the values when the tenant has two full-featured replicas and four full-featured replicas.ocp.arbitration.max.cpu.used.percentage: the maximum CPU utilization allowed on the arbitration service host when the arbitration service is enabled for the tenant. Unit:%.ocp.arbitration.max.memory.used.percentage: the maximum memory usage allowed on the arbitration service host when the arbitration service is enabled for the tenant. Unit:%.