Example of smooth scaling in and out

For businesses highly sensitive to latency, it's crucial to limit load balancing operations on nodes hosting log stream leaders during scaling in or out. These operations include transfer tasks, log stream replica migration, and leader switching. By leveraging heterogeneous zones, you can avoid leader switching and replica migration during peak hours, ensuring a smoother scaling process.

Considerations

Generally, only zones where the UNIT_NUM changes will experience replica migration during scaling or rollback operations. Other zones will not undergo log stream migration, but may have local transfers. However, if the tenant is not balanced before scaling—for example, if partition balancing is ongoing or a new scaling task is started before the previous one completes—the system cannot guarantee that log streams in other zones will not be migrated.

Scenario 1: Scaling by adding or removing zones

If a tenant's PRIMARY_ZONE is distributed across multiple zones and cannot be concentrated in a single zone, you need to use the locality change method to scale in or out. This involves first adding or removing zones, then using locality changes to add replicas to the new zones, ensuring a balanced distribution.

Let's take scaling out as an example.

Scenario

Assume the current tenant is tenant1, with locality F@zone1, F@zone2, A and PRIMARY_ZONE set to zone1, zone2. The tenant has a resource pool named pool distributed across zone1 and zone2, with UNIT_NUM set to 2 (configured as 2:2). The distribution of user log streams is shown below:

Now, the tenant's resource pool UNIT_NUM needs to be expanded to 4:4.

Procedure

1. Log in to the cluster's sys tenant as the root user.

2. Disable the enable_ls_leader_balance parameter.

   The tenant-level parameter enable_ls_leader_balance controls log stream leader balancing. When the global load balancing switch enable_rebalance is enabled, this parameter can independently control automatic balancing of log stream leaders. The default value is True, meaning automatic leader balancing is enabled by default.

   obclient(root@sys)[(none)]> ALTER SYSTEM SET enable_ls_leader_balance = False TENANT = 'tenant1';
   

3. Add two zones with four nodes each and allocate resources to the tenant in these new zones.

   1. Add zone3 and zone4 to the cluster. See Add a zone for details.

   2. Add four nodes to each of the new zones. See Add a node.

   3. Create a resource pool named pool2 in zone3 and zone4 with the same unit specifications as the original, and set UNIT_NUM to 4.

      obclient(root@sys)[(none)]> CREATE RESOURCE POOL pool2 UNIT='unit_name', UNIT_NUM = 4, ZONE_LIST=('zone3','zone4');
      

      Replace unit_name with the actual unit name.

   4. Update the tenant's resource pool to include the new resource pool pool2.

      obclient(root@sys)[(none)]> ALTER TENANT tenant1 RESOURCE_POOL_LIST =('pool','pool2');
      

      After this operation, the tenant will have 2 units in zone1, 2 units in zone2, 4 units in zone3, and 4 units in zone4.

4. Update the tenant's locality to include zone3 and zone4.

   Change the locality from F@zone1, F@zone2, A to F@zone1, F@zone2, F@zone3, F@zone4, A:

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 LOCALITY="F@zone1, F@zone2, F@zone3, F@zone4, A";
   

   After executing this command, the system will automatically supplement log stream replicas in the new units and distribute them evenly. Only follower replicas are added, so no leader switching occurs.

   The distribution of user log streams after adding zones is shown below.

   

5. Set the tenant's PRIMARY_ZONE to zone3 and zone4, and re-enable the enable_ls_leader_balance parameter.

   Change the PRIMARY_ZONE from zone1, zone2 to zone3, zone4:

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 PRIMARY_ZONE='zone3,zone4';
   

   Re-enable enable_ls_leader_balance:

   obclient> ALTER SYSTEM SET enable_ls_leader_balance = True TENANT = 'tenant1';
   

   After these commands are executed, leader replicas will automatically switch to zone3 and zone4. Changing only the PRIMARY_ZONE is fast and the impact is limited.

6. Update the tenant's locality again to remove zone1 and zone2.

   Change the locality from F@zone1, F@zone2, F@zone3, F@zone4, A to F@zone3, F@zone4, A:

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 LOCALITY="F@zone3, F@zone4, A";
   

   After executing this command, log stream replicas in zone1 and zone2 will be deleted; zone3 and zone4 are unaffected.

7. Update the tenant's resource pool to remove the original pool pool, leaving only pool2.

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 RESOURCE_POOL_LIST =('pool2');
   

   After removing the original resource pool, the two-node zone1 and zone2 can also be deleted.

   Once completed, only zone3 and zone4 (each with 4 nodes) remain under the tenant, and the scaling out is complete.

Scenario 2: In-place scaling

For tenants with a concentrated PRIMARY_ZONE, in-place scaling can be performed. This ensures that only follower replicas are affected.

Let's use scaling out as an example.

Scenario

Assume tenant1 has locality F@zone1, F@zone2, A and PRIMARY_ZONE set to zone1. The tenant has a resource pool named pool distributed across zone1 and zone2, with UNIT_NUM set to 2 (configured as 2:2). The distribution of user log streams is shown below.

Now, the tenant's resource pool UNIT_NUM needs to be scaled out to 4:4.

Procedure

1. Log in to the cluster's sys tenant as the root user.

2. Split the tenant's resource pool into pool1 for zone1 and pool2 for zone2.

   obclient(root@sys)[(none)]> ALTER RESOURCE POOL pool SPLIT INTO ('pool1','pool2') ON ('zone1','zone2');
   

   For more information, see Merge and split resource pools.

3. Change the UNIT_NUM of pool2 to 4.

   obclient(root@sys)[(none)]> ALTER RESOURCE POOL pool2 UNIT_NUM = 4;
   

   After executing this command, log streams will automatically split and be distributed across 4 units in zone1. The distribution is shown below.

   

4. Change the tenant's PRIMARY_ZONE to zone2.

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 PRIMARY_ZONE='zone2';
   

5. Change the UNIT_NUM of pool1 to 4.

   obclient(root@sys)[(none)]> ALTER RESOURCE POOL pool1 UNIT_NUM = 4;
   

   After this command, the system will start balancing. Once balancing completes, scaling out is finished. The final distribution is shown below.

   

6. (Optional) If you want leader replicas to remain concentrated in zone1, you can change the tenant's PRIMARY_ZONE back to zone1.

   obclient(root@sys)[(none)]> ALTER TENANT tenant1 PRIMARY_ZONE='zone1';