Delete a node

You can remove a node from an OceanBase cluster for elastic scaling or deployment adjustment.

• Elastic scaling: When resources in a cluster are in surplus, you can perform a series of operations to reduce resource usage on nodes and remove unnecessary nodes. For example, you can adjust the distribution of units or decrease the value of UNIT_NUM for a tenant.

• Deployment adjustment: If you want to modify the locality of a tenant to delete its units from a zone, you can perform the following operations to remove this zone: Stop Zone -> Delete Server -> Delete Zone.

Considerations

The Delete Server operation involves load balancing. The resource units on a deleted node will be migrated to other nodes in the same zone. Resource unit migration implements automatic balancing of resource units, which is controlled by the Root Service. After the resource units are migrated, the Delete Server operation will succeed. If you want to specify the destination server for the resource units on the node to be deleted in the migration, you can manually migrate the resource units.

During automatic unit migration, the migration task will consume the network and I/O resources on the source and destination nodes. As a result, the migration traffic overlaps with the business traffic, and the business traffic is affected by the migration traffic. To prevent the business systems from being affected, you can:

• Perform the Stop Zone operation first and then the Delete Server operation to avoid overlapping of the two types of traffic.

  For more information about the Stop Zone operation, see Isolate a node.

• Adjust the parameters of the migration task to control the migration speed.

  The following table describes the parameters of a migration task.

  Parameter	Description
  balancer_idle_time	The interval for initiating the migration task. It is a cluster-level parameter. Default value: 5m, indicating 5 minutes. Value range: [10s, +∞).
  sys_bkgd_net_percentage	The percentage of NIC bandwidth occupied by the migration task. It is a cluster-level parameter. Default value: 60. Value range: [0, 100].

  For more information about how to view and modify the parameters, see Modify cluster parameters.

Procedure

1. Log on to the sys tenant of the cluster as the root user.

   Run the following command to log on. You must replace the related information in the command based on the actual database environment.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys -p***** -A
   

   For more information about how to connect to OceanBase Database, see Overview of the Connect to OceanBase Database chapter in Develop Applications in MySQL Mode and Overview of the Connect to OceanBase Database chapter in Develop Applications in Oracle Mode.

2. (Optional) Manually migrate the resource units on the node to be deleted.

   Notice

   Before you remove a node, we recommend that you manually migrate the resource units on the node to be deleted and then perform the Delete Server operation.

   1. View the distribution of resource units and obtain the IDs of the resource units to be migrated.

      obclient [(none)]> SELECT UNIT_ID,TENANT_ID,STATUS,ZONE,SVR_IP FROM oceanbase.DBA_OB_UNITS;
      +---------+-----------+--------+-------+----------------+
      | UNIT_ID | TENANT_ID | STATUS | ZONE  | SVR_IP         |
      +---------+-----------+--------+-------+----------------+
      |       1 |         1 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |       2 |         1 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |       3 |         1 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1001 |      1002 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1002 |      1002 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1003 |      1002 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |    1010 |      1008 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1011 |      1008 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |    1012 |      1008 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1013 |      1010 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1014 |      1010 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1015 |      1010 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |    1016 |      NULL | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1017 |      NULL | ACTIVE | zone2 |   xx.xx.xx.238 |
      +---------+-----------+--------+-------+----------------+
      14 rows in set
      

      • UNIT_ID: the IDs of resource units on the node to be deleted.

      • TENANT_ID: the ID of the tenant to which a unit belongs. If the value is NULL, the current unit does not belong to any tenant.

      • STATUS: the current status of the unit.

        • ACTIVE: The unit is normal.

        • DELETING: The unit is being deleted.

      • ZONE: the zone to which the unit belongs.

      • SVR_IP: the IP address of the node to which the unit belongs.

      For more information about the oceanbase.DBA_OB_UNITS view, see oceanbase.DBA_OB_UNITS.

   2. Execute the following statement to manually migrate the resource units.

      obclient [(none)]> ALTER SYSTEM MIGRATE UNIT = unit_id DESTINATION = 'svr_ip:svr_port';
      

      • unit_id: the ID of the resource unit to be migrated.

      • svr_ip: the IP address of the destination node of the resource unit.

      • svr_port: the RPC port of the destination node of the resource unit.

      For example, to migrate a resource unit with the ID 1012 to the server with the IP address of 172.xx.xx.xx in the same zone, execute the following statement:

      obclient [(none)]> ALTER SYSTEM MIGRATE UNIT = 1012 DESTINATION = '172.xx.xx.xx:2882';
      

      You can execute this statement repeatedly. If the manual migration fails, you can resolve the issue based on the error information and then try again.

      For more information about how to migrate a resource unit, see Unit migration.

3. Execute the following statement to delete the node.

   obclient [(none)]> ALTER SYSTEM DELETE SERVER 'svr_ip:svr_port' [,'svr_ip:svr_port'...] [ZONE [=] 'zone_name']
   

   • svr_ip: the IP address of the node to be deleted.

   • svr_port: the RPC port of the node to be deleted.

   • zone_name: the name of the zone to which the node to be deleted belongs. To delete multiple nodes, make sure that they are in the same zone.

   For example, to delete an OBServer node from zone1, execute the following statement:

   obclient [(none)]> ALTER SYSTEM DELETE SERVER "172.xx.xx.xx:2882" zone='zone1'
   

4. Query the oceanbase.DBA_OB_SERVERS view to verify whether the node is successfully deleted.

   obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_SERVERS;
   

   If this node no longer appears in the list, it is successfully deleted. If it still appears in the list and its status is DELETING, this node is still being deleted.

   If you directly perform the Delete Server operation without manually migrating the resource units from the node to be deleted, the operation may fail. If the node fails to be deleted, see the [What to do next](# What to do next) section in this topic.

What to do next

If you directly perform the Delete Server operation without manually migrating the resource units from the node to be deleted, resources may become insufficient during unit load balancing. That is, the remaining resources of other nodes in the zone are insufficient to accommodate the resource units on the node to be deleted. As a result, the unit migration will fail and this node will remain in the deleting state. You can check whether the unit migration fails based on the rootservice.log file in the /home/admin/oceanbase/log directory of the Root Service server. If the unit migration fails, you must perform the Cancel Delete Server operation, add a node to the zone, and delete the target node. For more information about how to add a node to a zone, see Add a node.

To perform the Cancel Delete Server operation, complete the following steps:

1. Log on to the sys tenant of the cluster as the root user.

   Run the following command to log on. You must replace the related information in the command based on the actual database environment.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys -p***** -A
   

2. Execute the following statement to cancel the node deletion.

   obclient [(none)]> ALTER SYSTEM CANCEL DELETE SERVER 'svr_ip:svr_port' [,'svr_ip:svr_port'...] [ZONE [=] 'zone_name']
   

   • svr_ip: the IP address of the target node whose deletion is to be canceled.

   • svr_port: the RPC port of the target node. Default value: 2882.

   • zone_name: the name of the zone to which the target node belongs.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM CANCEL DELETE SERVER '172.xx.xx.xx:2882' zone='zone1';
   

3. Query the oceanbase.DBA_OB_SERVERS view to verify whether the deletion of the node is successfully canceled.

   obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_SERVERS;
   

   If the status of the target OBServer node changes from DELETING to ACTIVE, the deletion of the node is successfully canceled.

More information

For more node-related O&M operations, see the following topics:

• View a node

• Restart a node

• Add a node

• Isolate a node

• Replace a node

• Delete a zone

• Unit migration