Smooth scaling application scenarios

For businesses that are highly sensitive to RT, during tenant scaling, it is necessary to strictly limit the load balancing operations on the nodes where the log stream leaders are located. These load balancing operations include transfer tasks, log stream replica migration and replication, and leader handover. In this case, heterogeneous zones can be used to avoid leader handover and leader replica migration during peak business hours, making the entire scaling process smoother.

Considerations

In general, only the Zone with a changed UNIT_NUM value will have log stream migration during scaling or rollback operations. Other Zones will only have local transfers. However, if the tenant is not in a balanced state before initiating a scaling operation, such as during partition balancing or if a new scaling task is started before the previous one is completed, the system cannot guarantee that log streams will not be migrated to other Zones.

Scenario 1: Scale out or in by adding or removing zones

If the PRIMARY_ZONE of a tenant is spread across multiple zones and cannot be concentrated in a single zone, you need to use the locality change method to scale out or in. This involves first adding or removing zones, then using the locality change method to add replicas to the new zones. The process of adding replicas ensures a balanced distribution.

Let's take scaling out as an example.

Scenario 1: Expand the number of units

Assume that you have a tenant named tenant1. The Locality of tenant1 is F@zone1, F@zone2, A, and the PRIMARY_ZONE is zone1, zone2. Tenant1 has a resource pool named pool that is distributed across zone1 and zone2. The UNIT_NUM is 2 (i.e., the UNIT_NUM is set to 2:2). The distribution of user log streams is shown in the following figure.

You need to expand the UNIT_NUM of the resource pool of tenant1 to 4:4.

Procedure

1. Log in to the sys tenant of the cluster as the root user.

2. Disable the enable_ls_leader_balance parameter.

   The tenant-level parameter enable_ls_leader_balance is the switch for log stream leader balancing. When the automatic load balancing switch enable_rebalance is enabled, this parameter can be used to independently control the automatic balancing of log stream leaders. The default value is True, which indicates that automatic balancing of log stream leaders is enabled by default.

   obclient(root@sys)[(none)]> ALTER SYSTEM SET enable_ls_leader_balance = False TENANT = 'tenant1';
   

3. Add two zones with four nodes each and add resources for tenant1 in the new zones.

   1. Add two zones, zone3 and zone4, to the cluster. For more information, see Add a zone.

   2. Add four nodes to each of the newly added zones, zone3 and zone4. For more information, see Add a node.

   3. Create a resource pool named pool2 in the newly added zones, zone3 and zone4. The specifications of the new resource pool must be the same as those of the original resource pool, and the UNIT_NUM must be set to 4.

      obclient(root@sys)[(none)]> CREATE RESOURCE POOL pool2 UNIT='unit_name', UNIT_NUM = 4, ZONE_LIST=('zone3','zone4');
      

      In the preceding statement, replace unit_name with the actual unit name.

   4. Modify the resource pool of tenant1 to allocate the newly added resource pool pool2 to tenant1.

      obclient(root@sys)[(none)]> ALTER TENANT tenant1 RESOURCE_POOL_LIST =('pool','pool2');
      

      After the preceding statement is executed, the UNIT_NUM of tenant1 is 2:2:4:4 across zone1, zone2, zone3, and zone4.

4. Modify the Locality of tenant1 to add the newly added zones, zone3 and zone4, to the Locality.

   Change the Locality from F@zone1, F@zone2, A to F@zone1, F@zone2, F@zone3, F@zone4, A. The statement is as follows:

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 LOCALITY="F@zone1,F@zone2,F@zone3,F@zone4,A";
   

   After the preceding statement is executed, the system automatically fills the log stream replicas on the newly added units and evenly distributes them across all units. In this process, only Follower replicas are added, so no leader switching occurs.

   The distribution of user log streams is shown in the following figure.

5. Modify the PRIMARY_ZONE of tenant1 to the newly added zones, zone3 and zone4, and enable the enable_ls_leader_balance parameter.

   Change the PRIMARY_ZONE from zone1, zone2 to zone3, zone4. The statement is as follows:

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 PRIMARY_ZONE='zone3,zone4'; 
   

   Enable the enable_ls_leader_balance parameter. The statement is as follows:

   obclient(root@sys)[(none)]> ALTER SYSTEM SET enable_ls_leader_balance = True TENANT = 'tenant1';
   

   After the preceding statements are executed, the leader replicas of all log streams are automatically switched to zone3 and zone4. If you only change the PRIMARY_ZONE of log streams, the process is fast and the impact is controllable.

6. Modify the Locality of tenant1 to remove zone1 and zone2 from the Locality.

   Change the Locality from F@zone1, F@zone2, F@zone3, F@zone4, A to F@zone3, F@zone4, A. The statement is as follows:

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 locality="F@zone3,F@zone4,A";
   

   After the preceding statement is executed, the replicas of log streams in zone1 and zone2 are deleted, but those in zone3 and zone4 are not affected.

7. Modify the resource pool of tenant1 to remove the original resource pool pool and retain only the newly added resource pool pool2.

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 RESOURCE_POOL_LIST =('pool2');
   

   After the original resource pool pool is removed from tenant1, the two-node zone1 and zone2 can also be deleted.

   After the preceding operations are completed, only zone3 and zone4 with four nodes (UNIT_NUM of 4) remain in the current tenant. The expansion is completed.

Scenario 2: In-place expansion or contraction

For a tenant with a concentrated PRIMARY_ZONE, you can perform in-place expansion or contraction. In-place expansion or contraction ensures that only the zones that host Follower replicas are modified.

The following example describes how to perform in-place expansion.

Scenario information

Assume that you have a tenant named tenant1. The Locality of tenant1 is F@zone1, F@zone2, A, and the PRIMARY_ZONE is zone1. Tenant1 has a resource pool named pool that is distributed across zone1 and zone2. The UNIT_NUM is 2 (i.e., the UNIT_NUM is set to 2:2). The distribution of user log streams is shown in the following figure.

You need to expand the UNIT_NUM of the resource pool of tenant1 to 4:4.

Procedure

1. Log in to the sys tenant of the cluster as the root user.

2. Split the resource pool of the tenant into pool1 in zone1 and pool2 in zone2.

   obclient(root@sys)[(none)]> ALTER RESOURCE POOL pool SPLIT INTO ('pool1','pool2') ON ('zone1','zone2');
   

   For more information about how to split a resource pool, see Merge and split resource pools.

3. Change the UNIT_NUM of the split resource pool pool2 to 4.

   obclient(root@sys)[(none)]> ALTER RESOURCE POOL pool2 UNIT_NUM = 4;
   

   After the statement is executed, the log stream is automatically split and distributed to 4 units in zone1. The distribution of the user log stream is as follows.

4. Change the PRIMARY_ZONE of the tenant to zone2.

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 PRIMARY_ZONE='zone2'; 
   

5. Change the UNIT_NUM of the split resource pool pool1 to 4.

   obclient(root@sys)[(none)]> ALTER RESOURCE POOL pool1 UNIT_NUM = 4;
   

   After the statement is executed, the system starts to balance the resources. After the balancing is completed, the expansion is completed. The distribution of the user log stream is as follows.

6. (Optional) If you want the leader replicas to remain concentrated in zone1, change the PRIMARY_ZONE of the tenant back to zone1.

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 PRIMARY_ZONE='zone1';