Overview

OceanBase Database introduces the arbitration service since this version. The arbitration service not only prevents the increase of response time (RT) due to the failure of replicas in the same city in the deployment mode of three IDCs across two regions, but also reduces the cross-city bandwidth overhead and minimizes the costs of the third IDC.

The arbitration service maintains arbitration members corresponding to log streams of a tenant. An arbitration member has the following characteristics:

• An arbitration member votes in leader election, Paxos Prepare, and member group change, but does not participate in Paxos Accept.

• An arbitration member does not store logs and has no MemTables or SSTables. Therefore, it consumes minimal bandwidth, memory, disk, and CPU resources.

• An arbitration member cannot be elected as a leader to provide services.

When log synchronization fails because half of the full-featured replicas failed and the log stream downgrade control time is reached, the arbitration service automatically performs the log stream downgrade process to remove the failed replicas from the member list. The locality of the tenant, however, does not change. This way, services can be recovered with a recovery point objective (RPO) of 0. After the full-featured replicas recover, the arbitration service performs the log stream upgrade process to add the downgraded replicas to the member list, thus ensuring high availability.

Log stream downgrade strategy

The arbitration service downgrades a log stream when log synchronization fails due to the following exceptions of half of the replicas:

• The OBServer nodes where the replicas reside are down.

• The ALTER SYSTEM STOP SERVER statement is executed on the OBServer nodes where the replicas reside, or the ALTER SYSTEM STOP ZONE statement is executed in the zones where the replicas reside.

• The kill -9, kill -15, or kill -19 operation is performed on the observer processes of the OBServer nodes where the replicas reside.

• The OBServer nodes where the replicas reside are disconnected from the network.

• Tenant log disks on the OBServer nodes where the replicas reside are full.

• Logs of the replicas lag behind those of the leader due to a long-term failure, and the replicas will be rebuilt after the failure is rectified. During the rebuild, the replicas do not support log stream upgrade because the data and logs of the replicas are incomplete.

The log stream downgrade process is as follows:

1. The arbitration service running on the log stream leader detects that some replicas return no log confirmation messages within the period specified by the arbitration_timeout parameter. Then the arbitration service checks the replicas with out-of-sync logs.

2. The arbitration service periodically checks whether the replicas with out-of-sync logs encounter the exceptions listed in the downgrade strategy. If yes, the arbitration service downgrades the log stream.

   Notice

   The arbitration service downgrades a log stream only when the total number of failed replicas, including downgraded replicas, reaches half of the total number of full-featured replicas. Assuming that four full-featured replicas and the arbitration service are deployed:

   • If one full-featured replica fails, the majority of full-featured replicas are still available for log synchronization. Therefore, the arbitration service does not perform log stream downgrade.
   • If two full-featured replicas fail, the arbitration service performs log stream downgrade. For example, two full-featured replicas are disconnected from the network, or one full-featured replica is being rebuilt and the OBServer node where the other full-featured replica resides is down.
   • If three or more full-featured replicas fail, only one full-featured replica and the arbitration service are available and are not in the majority. In this case, no log stream leader exists, and the arbitration service cannot perform log stream downgrade.

You can execute the SELECT * FROM oceanbase.GV$OB_LOG_STAT WHERE DEGRADED_LIST NOT LIKE ''; statement to check whether log streams in a cluster are downgraded. If the DEGRADED_LIST field for a log stream in the query result is not empty, the log stream is downgraded.

Log stream upgrade strategy

After a log stream is downgraded, the arbitration service periodically checks the replicas in the DEGRADED_LIST field. If log synchronization of the downgraded replicas recovers and no exception listed in the downgrade strategy occurs in the replicas, the arbitration service can upgrade the log stream.

The log stream upgrade process is as follows:

1. The arbitration service running on the log stream leader periodically checks the replicas in the DEGRADED_LIST field. If the downgraded replicas return log confirmation messages to the leader, the arbitration service further checks the downgraded replicas.

2. The arbitration service checks the downgraded replicas for any exception listed in the log stream downgrade strategy. If no exception is detected, the arbitration service upgrade the log stream.

   Notice

   When you upgrade a log stream, it is not required that all downgraded replicas be free of exceptions. For example, assume that four full-featured replicas and the arbitration service are deployed, and two full-featured replicas are downgraded due to downtime of OBServer nodes where they reside. If one downgraded full-featured replica recovers and starts to synchronize logs, the arbitration service upgrades the log stream.

References

For more information about the arbitration service, see the following topics:

• Deploy an OceanBase cluster with two replicas and the arbitration service

• Enable the arbitration service for a tenant

• Disable the arbitration service for a tenant

• Modify the log stream downgrade control time

• Replace the arbitration service for a cluster

• Remove the arbitration service for a cluster

• Clear residual cluster information