Scale in an OceanBase cluster and scale down an OceanBase Database tenant

This topic describes how to scale in an OceanBase cluster and scale down an OceanBase Database tenant.

Scenarios

Scale-in and scale-down are the reverse operations of scale-out and scale-up, respectively. If a cluster has surplus resources or the business traffic declines, you can decrease the resource usage of OBServer nodes by reducing the specifications or number of resource units in tenants, and scale in the cluster by removing OBServer nodes.

Prerequisites

• Make sure that you have the following permissions:

  • To scale in a tenant, make sure that you have the following permissions:
    • Resource Permissions : Cluster maintenance or tenant maintenance permission
    • Menu Permissions : Permission on the Overview menu of Tenants
  • To scale in a cluster, make sure that you have the following permissions: Resource Permissions : Cluster Maintenance permission
    • Menu Permissions : Permission on the Overview menu of Clusters

• When you remove an OBServer node from a cluster, the cluster must be in the Running state. When you scale down a tenant, the cluster and the tenant must be in the Running state.

• Before you scale down a tenant, we recommend that you perform a major compaction to release the memory occupied by the tenant. For more information, see Perform a major compaction.

Considerations

• The removal of an OBServer node involves load balancing. The resource units on the removed OBServer node are automatically migrated to other OBServer nodes in the same zone. You can manually migrate the resource units to specific OBServer nodes before you remove the OBServer node. For more information, see View the unit distribution.

• The removal of an OBServer node results in the reduction of resources available in the system. If the available resources of remaining OBServer nodes in the same zone are insufficient to accommodate the resource units on the OBServer node to be removed, the resource unit migration fails. Before you remove an OBServer node, we recommend that you go to the Resource Management page of the cluster and check whether the remaining resources are sufficient.

• If the remaining OBServer nodes cannot accommodate the data replicas or resource units of all tenants, the removal fails. We recommend that you reduce the specifications or number of resource units before you remove an OBServer node.

Procedure

Scale down a tenant

• Reduce the unit specifications of a tenant.

  1. Go to the Overview page of the tenant that you want to scale down.

  2. In the Replica Details section, click Edit in the Actions column, and select unit specifications with smaller values.

     If the unit specifications you need are not available in the Unit drop-down list, click Add Specification to add a unit specification. Then, you can use the added unit specification.

     Note

     We recommend that you use the same specifications for each resource unit in the tenant.

• Reduce the number of resource units in a tenant.

  1. Go to the Overview page of the tenant that you want to scale down.

  2. In the Replica Details section, click Modify Unit in the upper-right corner to reduce the number of resource units.

     Note

  3. We recommend that you use the same number of resource units in each tenant.
  4. If you want to reduce the number of resource units in an OceanBase cluster of V4.0 or V4.1, you must first upgrade the cluster to V4.2 or later.

Scale in a cluster

If the OBServer nodes in your cluster have surplus resources after you scale down tenants, you can scale in the cluster by removing OBServer nodes.

1. Go to the Overview page of the cluster that you want to scale in.

2. Select the OBServer nodes that you want to remove and click Batch Delete to start the task.

   In general, you must remove the same number of OBServer nodes from each zone to ensure that resources are equally distributed among zones. For example, you can remove one OBServer node from each zone of a cluster in 2-2-2 mode to scale in the cluster to the 1-1-1 mode.

3. Wait until the task of removing OBServer nodes is completed.

   After the task is completed, you can view the deployment mode on the Overview page of the cluster and check whether the number of OBServer nodes in the cluster has reduced.

   If yes, tenant resources and data are automatically balanced among the remaining OBServer nodes. You can view the resource unit distribution on the Resource Management page of the cluster. For more information, see View the unit distribution.

FAQ

Q: What can I do if the task of scaling down tenants remains in progress? A: Tenant scale-down involves the migration of replicas, which may take a long time. Please wait until the task is completed. You can execute the following statement to query the oceanbase.DBA_OB_TENANT_JOBS view and check the task status:

-- In OceanBase Database V4.0 and earlier, if the value of JOB_STATUS is INPROGRESS, the task is in progress.
SELECT tenant_id, job_id, job_type, job_status, progress 
FROM oceanbase.__all_rootservice_job 
WHERE job_type = 'SHRINK_RESOURCE_POOL_UNIT_NUM' 
ORDER BY job_id DESC LIMIT 1;

-- In versions later than OceanBase Database V4.0, if the value of JOB_STATUS is INPROGRESS, the task is in progress.
SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS
FROM oceanbase.DBA_OB_TENANT_JOBS
WHERE JOB_TYPE = 'SHRINK_RESOURCE_POOL_UNIT_NUM'
ORDER BY JOB_ID DESC;

Q: What can I do if the task of removing OBServer nodes failed? A: It is likely that the remaining healthy OBServer nodes cannot constitute the majority of replicas. You need to check the number of tenant replicas.

Q: What can I do if the progress of removing OBServer nodes is slow? A: The removal of OBServer nodes involves the migration of resource units, which may take a long time to complete. You can modify the parameters that specifies the migration and replication concurrency. Here are some examples:

-- In OceanBase Database V4.0 and earlier, you can use the following parameters to specify the replica migration concurrency:
alter system set data_copy_concurrency = 100;
alter system set server_data_copy_out_concurrency = 40;
alter system set server_data_copy_in_concurrency = 40;

-- In versions later than OceanBase Database V4.0, you can use the following parameters to specify the replica migration concurrency:
alter system set ha_high_thread_score = 80;
alter system set ha_mid_thread_score = 80;

For more information about the parameters, see the following topics:

• OceanBase Database V4.0 and earlier: data_copy_concurrency, server_data_copy_out_concurrency, and server_data_copy_in_concurrency
• Versions later than OceanBase Database V4.0: ha_high_thread_score and ha_mid_thread_score

Q: What can I do if the task of waiting for the removal of OBServer nodes remains in progress? A: If an OBServer node has failed before you remove it, the OBServer node is removed after a permanent offline wait period, which is by default 1 hour after the removal task succeeds. The wait task may time out and fail. To solve the issue, you can specify a smaller value for the permanent offline wait period. Here is an example:

alter system set server_permanent_offline_time = '120s';

For more information about the parameter, see server_permanent_offline_time.