Replace a node

You can replace a node in the following scenarios: hardware failure or capacity expansion (replacement of a node of a different specification). The number of nodes in the cluster remains the same after the replacement. Replacing a node involves a series of O&M operations: adding a new node, migrating replicas, and then deleting the old node. The entire process takes a long time.

Procedure

1. Log in to the sys tenant of the cluster as the root user.

   The following example shows how to connect to the database. Please replace the actual environment variables with the correct values.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -A
   

   For more information about how to connect to the database, see Overview of connecting to a database (MySQL mode) and Overview of connecting to a database (Oracle mode).

2. Add a new node in the zone where the node to be replaced is located.

   For more information, see Add a node.

3. Migrate the units on the old node to the new node.

   1. Query the DBA_OB_SERVERS view to obtain the information of the node to be replaced.

      Here is an example of the query:

      obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_SERVERS;
      

      The query result is as follows:

      +-------------+----------+----+--------------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
      | SVR_IP      | SVR_PORT | ID | ZONE         | SQL_PORT | WITH_ROOTSERVER | STATUS | START_SERVICE_TIME         | STOP_TIME | BLOCK_MIGRATE_IN_TIME | CREATE_TIME                | MODIFY_TIME                | BUILD_VERSION                                                                             | LAST_OFFLINE_TIME |
      +-------------+----------+----+--------------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
      | 172.xx.xx.xx|     2882 |  6 | sa128_obv4_2 |     2881 | NO              | ACTIVE | 2022-12-30 16:17:03.173519 | NULL      | NULL                  | 2022-12-30 16:08:04.749100 | 2023-01-04 11:48:36.589270 | 4.0.0.0_101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322(Dec  7 2022 16:22:15) | NULL              |
      | 172.xx.xx.xx|     2882 |  4 | sa128_obv4_3 |     2881 | NO              | ACTIVE | 2022-12-30 16:36:35.567437 | NULL      | NULL                  | 2022-12-30 16:08:02.755200 | 2023-01-04 14:13:36.976548 | 4.0.0.0_101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322(Dec  7 2022 16:22:15) | NULL              |
      | 172.xx.xx.xx|     2882 |  3 | sa128_obv4_3 |     2881 | NO              | ACTIVE | 2022-12-12 12:42:00.054759 | NULL      | NULL                  | 2022-11-03 15:37:09.530894 | 2022-12-22 14:43:26.717736 | 4.0.0.0_101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322(Dec  7 2022 16:22:15) | NULL              |
      | 172.xx.xx.xx|     2882 |  1 | sa128_obv4_1 |     2881 | NO              | ACTIVE | 2022-12-12 12:25:17.555651 | NULL      | NULL                  | 2022-11-03 15:37:08.990683 | 2022-12-12 12:25:18.553763 | 4.0.0.0_101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322(Dec  7 2022 16:22:15) | NULL              |
      | 172.xx.xx.xx|     2882 |  2 | sa128_obv4_2 |     2881 | YES             | ACTIVE | 2022-12-12 11:46:37.222980 | NULL      | NULL                  | 2022-11-03 15:37:09.490511 | 2022-12-12 11:47:31.075335 | 4.0.0.0_101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322(Dec  7 2022 16:22:15) | NULL              |
      | 172.xx.xx.xx|     2882 |  5 | sa128_obv4_1 |     2881 | NO              | ACTIVE | 2022-12-30 16:25:45.420996 | NULL      | NULL                  | 2022-12-30 16:08:03.928478 | 2023-01-04 11:48:36.578231 | 4.0.0.0_101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322(Dec  7 2022 16:22:15) | NULL              |
      +-------------+----------+----+--------------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
      6 rows in set
      

      For more information about the DBA_OB_SERVERS view, see DBA_OB_SERVERS.

   2. Query the list of units on the node to be replaced based on its IP address.

      Here is the statement:

      obclient [(none)]> SELECT  unit_id FROM  oceanbase.DBA_OB_UNITS WHERE SVR_IP = 'svr_ip';
      

      In this statement, svr_ip is the IP address of the node to be replaced.

      Here is an example of the query:

      obclient [(none)]> SELECT unit_id FROM  oceanbase.DBA_OB_UNITS WHERE SVR_IP = '172.xx.xx.xx';
      +---------+
      | unit_id |
      +---------+
      |       1 |
      |    1002 |
      |    1016 |
      |    1024 |
      +---------+
      4 rows in set
      

   3. Submit a unit migration task to migrate the units on the old node to other nodes in the same zone.

      Here is the statement:

      obclient [(none)]> ALTER SYSTEM MIGRATE UNIT unit_id DESTINATION 'svr_ip:svr_port';
      

      The parameters are described as follows:

      • unit_id: the unit ID of the unit to be migrated.

      • svr_ip:svr_port: the IP address and RPC port number of the new node. The default RPC port number is 2882.

      Each statement can migrate only one unit. To migrate multiple units, you need to execute this statement multiple times.

      Here is an example of the migration:

      obclient [(none)]> ALTER SYSTEM MIGRATE UNIT 1016 DESTINATION '172.xx.xx.xx:2882';
      

   4. Query the oceanbase.DBA_OB_UNIT_JOBS view to confirm the migration progress of the unit.

      obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_UNIT_JOBS WHERE JOB_TYPE = 'MIGRATE_UNIT';
      +--------+--------------+------------+-------------+----------+----------------------------+----------------------------+-----------+---------+----------+------------+------------+-------------+
      | JOB_ID | JOB_TYPE     | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | UNIT_ID | SQL_TEXT | EXTRA_INFO | RS_SVR_IP  | RS_SVR_PORT |
      +--------+--------------+------------+-------------+----------+----------------------------+----------------------------+-----------+---------+----------+------------+------------+-------------+
      |      4 | MIGRATE_UNIT | INPROGRESS |        NULL |        0 | 2023-01-04 17:22:02.208219 | 2023-01-04 17:22:02.208219 |      1004 |    1006 | NULL     | NULL       |xx.xx.xx.106|        2882 |
      +--------+--------------+------------+-------------+----------+----------------------------+----------------------------+-----------+---------+----------+------------+------------+-------------+
      

      If the query result is empty, the unit migration is completed.

   For more information about unit migration, see Unit migration.

4. After all the units on the old node are migrated, delete the old node.

   The specific steps are as follows:

   1. Execute the following statement to delete the old node.

      Here is the statement:

      obclient [(none)]> ALTER SYSTEM DELETE SERVER 'svr_ip:svr_port' [,'svr_ip:svr_port'...] [ZONE [=] 'zone_name']
      

      The parameters are described as follows:

      • svr_ip: the IP address of the old node to be deleted.

      • port: the RPC port number of the old node to be deleted. The default value is 2882.

      • zone_name: the zone to which the old node to be deleted belongs.

      Here is an example:

      obclient [(none)]> ALTER SYSTEM DELETE SERVER "172.xx.xx.xx:2882" zone='zone1'
      

   2. After the operation is completed, query the oceanbase.DBA_OB_SERVERS view to confirm whether the old node is deleted.

      obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_SERVERS;
      

      If the information of the old node is no longer displayed in the query result, the old node is deleted. If the node is still displayed in the list and its status is DELETING, the node is still being deleted.

References

For more information about node management, see the following topics:

• View a node

• Restart a node

• Add a node

• Delete a node

• Isolate a node