Replace the arbitration service for a cluster

When the machine where the arbitration service resides fails, you can deploy the arbitration service on a new machine and then replace the failed arbitration service for the cluster.

Prerequisites

Confirm that OceanBase Database software is installed on the new machine for replacing the arbitration service and the arbitration service has been started. For more information, see Deploy an OceanBase cluster with two replicas and the arbitration service.

Procedure

1. Log in to the sys tenant of the cluster as the root user.

   Below is a connection exampl. Use your actual environment when connecting to the database.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -A
   

   For more detailed guidance on connecting to the database, see Overview of connection methods (MySQL-compatible mode) and Overview of connection methods (Oracle-compatible mode).

2. Execute the following statement to obtain the current arbitration service address of the cluster.

   SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
   

   The ARBITRATION_SERVICE column in the query result shows the current arbitration service address of the cluster.

3. Execute the following statement to replace the arbitration service for the cluster:

   ALTER SYSTEM REPLACE ARBITRATION SERVICE 'curr_arb_server_ip:curr_arb_server_port' WITH 'new_arb_server_ip:new_arb_server_port';
   

   The parameters are described as follows:

   • curr_arb_server_ip:curr_arb_server_port: The IP address and RPC port of the current failed arbitration service. The default port is 2882.

   • new_arb_server_ip:new_arb_server_port: The IP address and RPC port of the new arbitration service. The default port is 2882.

   Example:

   ALTER SYSTEM REPLACE ARBITRATION SERVICE 'xx.xx.xx.64:2882' WITH 'xx.xx.xx.192:2882';
   

   If the execution times out after you execute the replace command, you can query the DBA_OB_ARBITRATION_SERVICE view to confirm the replacement status of the arbitration service.

   SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
   

   The query result is as follows:

   +---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
   | CREATE_TIME         | MODIFY_TIME         | ARBITRATION_SERVICE_KEY | ARBITRATION_SERVICE | PREVIOUS_ARBITRATION_SERVICE | TYPE |
   +---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
   | 2023-06-26 19:30:19 | 2023-06-30 09:42:40 | default                 | xx.xx.xx.192:2882   |                              | ADDR |
   +---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
   1 row in set
   

   According to the query result, if the value of the PREVIOUS_ARBITRATION_SERVICE column is not empty, the background change process is not yet complete and you need to wait for it to finish. If the value of the PREVIOUS_ARBITRATION_SERVICE column is empty, the background change process is complete and the arbitration service has been successfully replaced.

   For more information about the DBA_OB_ARBITRATION_SERVICE view, see DBA_OB_ARBITRATION_SERVICE.

   After the replace command is executed successfully, if the original arbitration service was in normal state before replacement, the system automatically clears residual cluster information on the original arbitration server. If the original arbitration service was in faulty state before replacement, the system displays a warning message, indicating that after the arbitration service is successfully replaced, the original arbitration server still contains residual cluster information and you need to choose whether to manually clear the residual information by using the ob_admin tool. For more information about clearing residual cluster information, see Clear residual cluster information.

   Execute the following statement to view the warning message:

   SHOW WARNINGS;
   

   The warning message is as follows:

   +---------+------+---------------------------------------------------------------------------------------------------------------------------+
   | Level   | Code | Message                                                                                                                   |
   +---------+------+---------------------------------------------------------------------------------------------------------------------------+
   | Warning | 4757 | Cluster info may remain on arbitration server 'xx.xx.xx.64:2882', please make sure whether to use ob_admin to clean it.   |
   +---------+------+---------------------------------------------------------------------------------------------------------------------------+
   1 row in set
   

4. (Optional) After the arbitration service is successfully replaced, you can query the replacement history of the arbitration service by specifying the module in the DBA_OB_ROOTSERVICE_EVENT_HISTORY view.

   The syntax is as follows:

   SELECT * FROM oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE module LIKE "%arb_service%";
   

   The query result is as follows:

   +----------------------------+-------------+-----------------------------------+-------+--------+-------+--------------------------------------------------------------------------------------------------+-------+--------+-------+--------+-------+--------+-------+--------+------------+----------------+-------------+
   | TIMESTAMP                  | MODULE      | EVENT                             | NAME1 | VALUE1 | NAME2 | VALUE2                                                                                           | NAME3 | VALUE3 | NAME4 | VALUE4 | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO | RS_SVR_IP      | RS_SVR_PORT |
   +----------------------------+-------------+-----------------------------------+-------+--------+-------+--------------------------------------------------------------------------------------------------+-------+--------+-------+--------+-------+--------+-------+--------+------------+----------------+-------------+
   | 2023-06-26 19:30:19.245664 | arb_service | admin_add_arbitration_service     | ret   | 0      | arg   | {arbitration_service_:"xx.xx.xx.192:2882"}                                                       |       |        |       |        |       |        |       |        |            | xx.xx.xx.194   |        2882 |
   | 2023-06-30 09:42:35.623655 | arb_service | admin_replace_arbitration_service | ret   | 0      | arg   | {arbitration_service_:"xx.xx.xx.64:2882", previous_arbitration_service_:"xx.xx.xx.192:2882"}     |       |        |       |        |       |        |       |        |            | xx.xx.xx.194   |        2882 |
   +----------------------------+-------------+-----------------------------------+-------+--------+-------+--------------------------------------------------------------------------------------------------+-------+--------+-------+--------+-------+--------+-------+--------+------------+----------------+-------------+
   2 rows in set
   

References

For more information about the arbitration service, see the following:

• Deploy an OceanBase cluster with two replicas and the arbitration service

• Overview of the arbitration service

• Enable the arbitration service for a tenant

• Disable the arbitration service for a tenant

• Modify the log stream downgrade control time

• Remove the arbitration service for a cluster

• Clear residual cluster information