Replace the arbitration service for a cluster

If the OBServer node where the arbitration service resides fails, you can deploy the arbitration service on another OBServer node.

Prerequisites

The OceanBase Database software is installed on the new OBServer node and the arbitration service is enabled. For more information, see Deploy an OceanBase cluster with two replicas and the arbitration service.

Procedure

1. Log in to the sys tenant of the cluster as the root user.

   Note that you must specify the corresponding parameters in the following sample code based on your actual database configurations.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -A
   

   For more information about how to connect to a database, see Overview (MySQL mode) or Overview (Oracle mode).

2. Execute the following statement to obtain the current arbitration service address of the cluster:

   SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
   

   The ARBITRATION_SERVICE column in the query result shows the current arbitration service address of the cluster.

3. Execute the following statement to replace the arbitration service for the cluster:

   ALTER SYSTEM REPLACE ARBITRATION SERVICE 'curr_arb_server_ip:curr_arb_server_port' WITH 'new_arb_server_ip:new_arb_server_port';
   

   The parameters are described as follows:

   • curr_arb_server_ip:curr_arb_server_port: the IP address and remote procedure call (RPC) port of the faulty OBServer node on which the arbitration service is deployed. By default, Port 2882 is used.

   • new_arb_server_ip:new_arb_server_port: the IP address and RPC port of the new OBServer node on which the arbitration service is deployed. By default, Port 2882 is used.

   Here is an example:

   ALTER SYSTEM REPLACE ARBITRATION SERVICE 'xx.xx.xx.64:2882' WITH 'xx.xx.xx.192:2882';
   

   If the execution times out, you can query the DBA_OB_ARBITRATION_SERVICE view to check the replacement status of the arbitration service.

   SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
   

   A sample query result is as follows:

   +---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
   | CREATE_TIME         | MODIFY_TIME         | ARBITRATION_SERVICE_KEY | ARBITRATION_SERVICE | PREVIOUS_ARBITRATION_SERVICE | TYPE |
   +---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
   | 2023-06-26 19:30:19 | 2023-06-30 09:42:40 | default                 | xx.xx.xx.192:2882   |                              | ADDR |
   +---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
   1 row in set
   

   If the value of the PREVIOUS_ARBITRATION_SERVICE field in the query result is not empty, the replacement is in progress in the background. Wait for the replacement process to complete. If the value of the PREVIOUS_ARBITRATION_SERVICE field is empty, the arbitration service is replaced.

   For more information about the DBA_OB_ARBITRATION_SERVICE view, see DBA_OB_ARBITRATION_SERVICE.

   If the arbitration service works properly before the replacement, the system automatically clears residual cluster information on the original OBServer node where the arbitration service resides. If the arbitration service fails before the replacement, the system displays a warning, which indicates that the original OBServer node where the arbitration service resides contains residual cluster information. You can manually clear the residual cluster information by using the ob_admin tool. For more information, see Clear residual cluster information.

   Execute the following statement to view the warning:

   SHOW WARNINGS;
   

   The query result is as follows:

   +---------+------+---------------------------------------------------------------------------------------------------------------------------+
   | Level   | Code | Message                                                                                                                   |
   +---------+------+---------------------------------------------------------------------------------------------------------------------------+
   | Warning | 4757 | Cluster info may remain on arbitration server 'xx.xx.xx.64:2882', please make sure whether to use ob_admin to clean it.   |
   +---------+------+---------------------------------------------------------------------------------------------------------------------------+
   1 row in set
   

4. (Optional) Query the replacement history of the arbitration service from the specified module in the DBA_OB_ROOTSERVICE_EVENT_HISTORY view.

   Here is a sample statement:

   SELECT * FROM oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE module LIKE "%arb_service%";
   

   A sample query result is as follows:

   +----------------------------+-------------+-----------------------------------+-------+--------+-------+--------------------------------------------------------------------------------------------------+-------+--------+-------+--------+-------+--------+-------+--------+------------+----------------+-------------+
   | TIMESTAMP                  | MODULE      | EVENT                             | NAME1 | VALUE1 | NAME2 | VALUE2                                                                                           | NAME3 | VALUE3 | NAME4 | VALUE4 | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO | RS_SVR_IP      | RS_SVR_PORT |
   +----------------------------+-------------+-----------------------------------+-------+--------+-------+--------------------------------------------------------------------------------------------------+-------+--------+-------+--------+-------+--------+-------+--------+------------+----------------+-------------+
   | 2023-06-26 19:30:19.245664 | arb_service | admin_add_arbitration_service     | ret   | 0      | arg   | {arbitration_service_:"xx.xx.xx.192:2882"}                                                       |       |        |       |        |       |        |       |        |            | xx.xx.xx.194   |        2882 |
   | 2023-06-30 09:42:35.623655 | arb_service | admin_replace_arbitration_service | ret   | 0      | arg   | {arbitration_service_:"xx.xx.xx.64:2882", previous_arbitration_service_:"xx.xx.xx.192:2882"}     |       |        |       |        |       |        |       |        |            | xx.xx.xx.194   |        2882 |
   +----------------------------+-------------+-----------------------------------+-------+--------+-------+--------------------------------------------------------------------------------------------------+-------+--------+-------+--------+-------+--------+-------+--------+------------+----------------+-------------+
   2 rows in set
   

References

For more information about the arbitration service, see the following topics:

• Deploy an OceanBase cluster with two replicas and the arbitration service

• Overview

• Enable the arbitration service for a tenant

• Disable the arbitration service for a tenant

• Modify the log stream downgrade control time

• Remove the arbitration service for a cluster

• Clear residual cluster information