Tenant pre-scaling

OceanBase Database implements a pre-scaling feature to achieve smooth scaling for tenants. This feature works by configuring the scaling factor (ls_scale_out_factor) for the tenant, which determines the number of log streams to be created before scaling occurs. By generating enough log streams in advance, their states and positions will align with those required after scaling, allowing for smooth expansion or contraction operations.

Background information

In the current version, OceanBase Database supports homogeneous zone mode and heterogeneous zone mode for tenants:

• In homogeneous zone mode, the UNIT_NUM of each zone in a tenant must be the same.

• In heterogeneous zone mode, the UNIT_NUM of each zone in a tenant can be the same or different, but a tenant can have at most two different UNIT_NUM values across all zones.

This topic mainly uses tenants in homogeneous zone mode as an example. For tenants in heterogeneous zone mode, we recommend that you use the O&M approach for heterogeneous zones to achieve smooth scaling. For an example of smooth scaling by using heterogeneous zones, see Application cases of smooth scaling in and out.

Introduction to scaling factor

In earlier versions, a tenant could have only one log stream on each node. If a node had enough log streams, log stream position redistribution could be triggered directly during scaling, without needing to use Transfer to split log streams.

To ensure that a tenant has enough log streams before scaling, OceanBase Database introduced the concept of the scaling factor (ls_scale_out_factor). The scaling factor defines how many log streams each log stream should be split into to meet future scaling requirements.

Formula for calculating the scaling factor

In OceanBase Database, the number of regular log streams for a tenant (excluding system and broadcast log streams) is calculated as: number of the tenant's UNIT_NUM (U) × number of the tenant's PRIMARY_ZONE with highest priority (P).

Assume that before scaling, the number of the tenant's UNIT_NUM is U1, and the number of the tenant's PRIMARY_ZONE with highest priority is P1. After scaling, the number of the tenant's UNIT_NUM is U2, and the number of PRIMARY_ZONE with highest priority is P2. The formula for calculating the scaling factor (ls_scale_out_factor) is as follows:

ls_scale_out_factor = LCM(U1 * p1 , U2 * P2 )/ (U1 * P1)

In other words, the least common multiple (LCM) of (U1 * p1) and (U2 * P2), divided by (U1 * P1), gives the scaling factor for the tenant.

Pre-scaling

Assume that the current tenant is mq_t1, with locality set to F@zone1, F@zone2, F@zone3. Each zone has one unit, and the PRIMARY_ZONE is distributed across zone1 and zone2. The goal is to increase the tenant's UNIT_NUM to 2, while keeping the number of first-priority PRIMARY_ZONEs unchanged. The steps for pre-scaling are as follows:

1. Log in to the sys tenant of the cluster as the root user.

   For example:

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Check the TENANT_ID, UNIT_NUM, and PRIMARY_ZONE information for the tenant mq_t1.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, UNIT_NUM FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME = 'mq_t1';
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+----------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    | UNIT_NUM |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+----------+
   |      1002 | mq_t1       | USER        | zone1,zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |        1 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+----------+
   1 row in set
   

3. Calculate the expansion factor ls_scale_out_factor and configure it for the tenant.

   In this example, before expansion, tenant mq_t1 has a UNIT_NUM of 1 and two first-priority PRIMARY_ZONE. After expansion, the tenant's UNIT_NUM will be 2, and the number of first-priority PRIMARY_ZONE remains 2. According to the expansion factor formula, the expansion factor is calculated as LCM(2 * 1, 2 * 2) / (2 * 1) = 2.

   Based on this calculation, set the value of the ls_scale_out_factor parameter for the tenant.

   obclient(root@sys)[oceanbase]> ALTER SYSTEM SET ls_scale_out_factor = 2 TENANT = mq_t1;
   

4. Enable the tenant-level parameters enable_rebalance and enable_transfer by setting them to True. This allows enough log streams to be created for the tenant.

   obclient(root@sys)[oceanbase]> ALTER SYSTEM SET enable_rebalance = True TENANT = mq_t1;
   

   obclient(root@sys)[oceanbase]> ALTER SYSTEM SET enable_transfer = True TENANT = mq_t1;
   

5. After waiting for the tenant to generate enough log streams, check whether the number of log streams in the NORMAL state matches expectations.

   According to the expansion factor formula, after expansion, the number of log streams in the NORMAL state (excluding system and broadcast log streams) should be: the original UNIT_NUM (U1) × the original number of first-priority PRIMARY_ZONE (P1) × the expansion factor (ls_scale_out_factor). In this case, the expected number of log streams is 1 * 2 * 2 = 4.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID, STATUS, COUNT(*) FROM oceanbase.CDB_OB_LS WHERE TENANT_ID = 1002 AND FLAG = '' AND LS_ID != 1 GROUP BY STATUS;
   

   Sample query result:

   +-----------+--------+----------+
   | TENANT_ID | STATUS | COUNT(*) |
   +-----------+--------+----------+
   |      1002 | NORMAL |        4 |
   +-----------+--------+----------+
   1 row in set
   

6. Confirm that there are no remaining load balancing tasks for the tenant being expanded.

   obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_BALANCE_JOBS WHERE TENANT_ID = 1002;
   

   If the query result is empty, the pre-scaling operation is complete.

What to do next

After the pre-scaling is completed, you can scale in or out the tenant based on the business needs and the expected scaling plan:

• Scale in or out a tenant by adjusting the unit number

• Scale in or out a tenant by adjusting the number of primary zones