Isolating GTS for high availability: a practical example

The Global Timestamp Service (GTS) for a tenant is provided by the system log stream, delivering global timestamps to all OBServer nodes under that tenant. Because GTS availability directly affects the tenant's overall availability, it is crucial to isolate GTS traffic from regular user traffic—especially in clusters with many nodes and high QPS—to ensure GTS response times are always prioritized.

This guide shows how to use UNIT_NUM expansion and enable GTS isolation, so GTS traffic is separated from user traffic, enhancing the stability of your business operations.

Background information

The current version supports GTS isolation, controlled by the tenant-level parameter enable_gts_standalone, which is disabled by default. You can enable this feature as needed. Once enabled, the system ensures that the log stream carrying GTS runs exclusively on its own group of units.

Scenario

Suppose you have a tenant called tenant1 with locality set to F@zone1, F@zone2, A and primary zones zone1 and zone2. The tenant has a UNIT_NUM of 2 in both zone1 and zone2. The log stream distribution before isolation is shown below:

Procedure

In this scenario, when the machine hosting the SYS log stream reaches its bottleneck, to ensure business stability, you can first expand the UNIT_NUM of all resource pools for the tenant from 2 to 3. Then, migrate and copy the SYS log stream (log stream 1) to the newly added group of units, making log stream 1 exclusively occupy a group of units. Finally, enable GTS standalone mode.

The detailed steps are as follows:

1. Log in as the root user to the cluster's sys tenant.

2. Disable automatic balancing for the tenant.

   Set the tenant-level parameter enable_rebalance to False to prevent automatic balancing:

   obclient(root@sys)[(none)]> ALTER SYSTEM SET enable_rebalance = False TENANT = tenant1;
   

3. (Optional) If there are not enough nodes in each zone, add one node to both zone1 and zone2 before expanding.

   For more information about how to add a node, see Add a node.

4. Increase the UNIT_NUM for all resource pools of the tenant to 3.

   obclient(root@sys)[(none)]> ALTER RESOURCE TENANT tenant1 UNIT_NUM = 3;
   

   For more information about how to expand the tenant by adjusting the UNIT_NUM, see Adjust the number of units to scale in or out a tenant.

5. Reassign the SYS log stream to the new units.

   Change the UNIT_GROUP (for homogeneous zones) or UNIT_LIST (for heterogeneous zones) of the SYS log stream (log stream 1) for tenant tenant1 to the newly added group of units, so log stream 1 exclusively occupies a group of units. The commands are as follows:

   Homogeneous zones

   Heterogeneous zones

   obclient(root@sys)[(none)]> ALTER SYSTEM MODIFY LS 1 UNIT_GROUP unit_group_id TENANT = tenant1;
   

   Replace unit_group_id with the actual ID of the newly added unit group.

   obclient(root@sys)[(none)]> ALTER SYSTEM MODIFY LS 1 UNIT_LIST (unit_id1,unit_id2) TENANT = tenant1;
   

   Replace unit_id1, unit_id2 with the actual IDs of the new units.

   For more information, see Modify a log stream.

6. Enable GTS isolation.

   Set enable_gts_standalone to True:

   obclient(root@sys)[(none)]> ALTER SYSTEM SET enable_rebalance = True TENANT = tenant1;
   

7. Re-enable automatic balancing for the tenant.

   Set the tenant-level parameter enable_rebalance to True:

   obclient(root@sys)[(none)]> ALTER SYSTEM SET enable_rebalance = True TENANT = tenant1;
   

In this scenario, after expanding the tenant's UNIT_NUM and enabling GTS isolation, the UNIT_NUM for user log streams remains unchanged. Only the SYS log stream (log stream 1) is migrated and copied; no log stream balancing or transfer occurs. This ensures business stability. The final distribution of the tenant's log streams is shown below.