Delete a node

You can delete a node from an OceanBase cluster for elastic scaling or deployment adjustment.

• Elastic scaling: When resources in a cluster are in surplus, you can perform a series of operations to reduce resource usage on nodes and delete unnecessary nodes. For example, you can adjust the distribution of units or decrease the value of UNIT_NUM for a tenant.

• Deployment adjustment: If you want to modify the locality of a tenant to delete its units from a zone, you can perform the following operations to remove this zone: Stop Zone -> Delete Server -> Delete Zone.

Considerations

The Delete Server operation involves load balancing. The resource units on a deleted node will be migrated to other nodes in the same zone. Resource unit migration implements automatic balancing of resource units, which is controlled by the RootService. After the resource units are migrated, the Delete Server operation will succeed. If you want to specify the destination server to which the resource units on the node to be deleted are to be migrated, you can manually migrate the resource units.

During automatic unit migration, the migration task will consume the network and I/O resources on the source and destination nodes. As a result, the migration traffic overlaps with the business traffic, and the business traffic is affected by the migration traffic. To prevent the business systems from being affected, you can perform the following operations:

• Perform the Stop Zone operation first and then the Delete Server operation to avoid overlapping of the two types of traffic.

  For more information about the Stop Zone operation, see Isolate a node.

• Adjust the parameters of the migration task to control the migration speed.

  The following table describes the parameters of a migration task.

  Parameter	Description
  balancer_idle_time	The interval for initiating the migration task. It is a cluster-level parameter. Default value: 10s. Value range: [10s, +∞).
  sys_bkgd_net_percentage	The percentage of NIC bandwidth occupied by the migration task. It is a cluster-level parameter. Default value: 60. Value range: [0, 100].

  For more information about how to view and modify the parameters, see Modify cluster parameters.

Procedure

1. Log on to the sys tenant of an OceanBase cluster as the root user.

   Note that you must specify the corresponding fields in the following sample code based on your actual database configurations.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -A
   

   For more information about how to connect to a database, see Connection methods (MySQL mode) or Connection methods (Oracle mode).

2. (Optional) Manually migrate the resource units on the node to be deleted.

   Notice

   Before you delete a node, we recommend that you manually migrate the resource units on the node to be deleted and then perform the Delete Server operation.

   1. View the distribution of resource units and obtain the IDs of the resource units to be migrated.

      obclient [(none)]> SELECT UNIT_ID,TENANT_ID,STATUS,ZONE,SVR_IP FROM oceanbase.DBA_OB_UNITS;
      +---------+-----------+--------+-------+----------------+
      | UNIT_ID | TENANT_ID | STATUS | ZONE  | SVR_IP         |
      +---------+-----------+--------+-------+----------------+
      |       1 |         1 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |       2 |         1 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |       3 |         1 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1001 |      1002 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1002 |      1002 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1003 |      1002 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |    1010 |      1008 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1011 |      1008 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |    1012 |      1008 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1013 |      1010 | ACTIVE | zone3 |   xx.xx.xx.218 |
      |    1014 |      1010 | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1015 |      1010 | ACTIVE | zone2 |   xx.xx.xx.238 |
      |    1016 |      NULL | ACTIVE | zone1 |   xx.xx.xx.237 |
      |    1017 |      NULL | ACTIVE | zone2 |   xx.xx.xx.238 |
      +---------+-----------+--------+-------+----------------+
      14 rows in set
      

      The columns in the query result are described as follows:

      • UNIT_ID: the ID of the resource unit on the node to be deleted.

      • TENANT_ID: the ID of the tenant to which a unit belongs. If the value is NULL, the current unit does not belong to any tenant.

      • STATUS: the current status of the unit.

        • ACTIVE: The unit is normal.

        • DELETING: The unit is being deleted.

      • ZONE: the zone to which the unit belongs.

      • SVR_IP: the IP address of the node to which the unit belongs.

      For more information about the DBA_OB_UNITS view, see DBA_OB_UNITS.

   2. Execute the following statement to manually migrate the resource units.

      The syntax is as follows:

      obclient [(none)]> ALTER SYSTEM MIGRATE UNIT = unit_id DESTINATION = 'svr_ip:svr_port';
      

      where

      • unit_id specifies the ID of the resource unit to be migrated.

      • svr_ip specifies the IP address of the destination node of the resource unit.

      • svr_port specifies the RPC port of the destination node of the resource unit.

      To migrate a resource unit with the ID 1012 to the server with the IP address of 172.xx.xx.xx in the same zone, execute the following statement:

      obclient [(none)]> ALTER SYSTEM MIGRATE UNIT = 1012 DESTINATION = '172.xx.xx.xx:2882';
      

      You can execute this statement repeatedly. If the manual migration fails, you can resolve the issue based on the error information and then try again.

      For more information about how to migrate a resource unit, see Migrate units.

3. Execute the following statement to delete the node:

   obclient [(none)]> ALTER SYSTEM DELETE SERVER 'svr_ip:svr_port' [,'svr_ip:svr_port'...] [ZONE [=] 'zone_name']
   

   where

   • svr_ip specifies the IP address of the node to be deleted.

   • svr_port specifies the RPC port of the node to be deleted.

   • zone_name specifies the name of the zone to which the node to be deleted belongs. To delete multiple nodes, make sure that they are in the same zone.

   To delete an OBServer node from zone1, execute the following statement:

   obclient [(none)]> ALTER SYSTEM DELETE SERVER "172.xx.xx.xx:2882" zone='zone1'
   

4. Query the oceanbase.DBA_OB_SERVERS view to verify whether the node is successfully deleted.

   obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_SERVERS;
   

   If this node no longer appears in the list, it is successfully deleted. If it still appears in the list and its status is DELETING, this node is still being deleted.

   If you directly perform the Delete Server operation without manually migrating the resource units from the node to be deleted, the operation may fail. If the node fails to be deleted, see the What to do next section in this topic.

What to do next

If you directly perform the Delete Server operation without manually migrating the resource units from the node to be deleted, resources may become insufficient during unit load balancing. That is, the remaining resources of other nodes in the zone are insufficient to accommodate the resource units on the node to be deleted. As a result, the unit migration will fail and this node will remain in the deleting state. You can check whether the unit migration fails based on the rootservice.log file in the /home/admin/oceanbase/log directory of the RootService server. If the unit migration fails, you must perform the Cancel Delete Server operation, add a node to the zone, and delete the target node. For more information about how to add a node to a zone, see Add a node.

To perform the Cancel Delete Server operation, follow these steps:

1. Log on to the sys tenant of the cluster as the root user.

   Note that you must specify the corresponding fields in the following sample code based on your actual database configurations.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -A
   

2. Execute the following statement to cancel the deletion of the node:

   obclient [(none)]> ALTER SYSTEM CANCEL DELETE SERVER 'svr_ip:svr_port' [,'svr_ip:svr_port'...] [ZONE [=] 'zone_name']
   

   where

   • svr_ip specifies the IP address of the target node whose deletion is to be canceled.

   • svr_port specifies the RPC port of the target node. Default value: 2882.

   • zone_name specifies the name of the zone to which the target node belongs.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM CANCEL DELETE SERVER '172.xx.xx.xx:2882' zone='zone1';
   

3. Query the oceanbase.DBA_OB_SERVERS view to verify whether the deletion of the node is successfully canceled.

   obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_SERVERS;
   

   If the status of the target OBServer node changes from DELETING to ACTIVE, the deletion of the node is successfully canceled.

References

For more node-related O&M operations, see the following topics:

• View a node

• Restart a node

• Add a node

• Isolate a node

• Replace a node

• Delete a zone

• Migrate units