SYS tenant/RS service issues

In OceanBase Database, the Root Service (RS) is a core service responsible for managing cluster resources and executing distributed strategies. It plays a crucial role in the cluster. The SYS tenant, which provides computing and storage resources for the RS, also achieves high availability through multi-replica self-election. In some large-scale or high-load environments, issues may occasionally occur due to queue backlog in the sys tenant, especially in extreme scenarios or hardware failures. This section introduces some emergency handling strategies for such situations.

Emergency handling methods

When the sys tenant queue backlog occurs, you can try the following methods to handle it.

Expand the resources of the sys tenant

1. View the resource allocation method for each tenant as follows:

   obclient> SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
   FROM
   OCEANBASE.DBA_OB_TENANTS a,
   OCEANBASE.DBA_OB_UNIT_CONFIGS b,
   OCEANBASE.DBA_OB_RESOURCE_POOLS c
   WHERE a.tenant_id=c.tenant_id
   AND b.unit_config_id = c.unit_config_id
   ORDER BY a.tenant_id desc;
   

2. Execute the following command to expand the CPU and memory resources of the sys tenant.

   ALTER RESOURCE UNIT unit_config_name min_cpu = 2,max_cpu = 2,memory_size = '2G',max_iops = 10000,min_iops = 10000;
   

Execute stop server

When the sys tenant queue backlog occurs, you can execute stop server to isolate the machine where the RS/SYS tenant is located.

obclient> ALTER SYSTEM STOP SERVER "xx.xx.xx.xx:2882";

Restart the problematic node

Restart the OBServer node where the problematic RS is located. For more information, see Restart an OBServer node.