sys tenant or RootService exceptions

RootService plays a crucial role in an OceanBase cluster. RootService manages cluster resources and executes distributed strategies. The sys tenant provides RootService with computing and storage resources and achieves high availability based on automatic leader election from multiple replicas. In a large or overloaded database system, the request queue of the sys tenant may be accumulated due to extreme scenarios or hardware exceptions. This topic describes the emergency procedure that you can follow to cope with an accumulated request queue.

Emergency procedure

You can use one of the following methods to relieve the request queue of the sys tenant.

Scale up the resources of the sys tenant

1. Execute the following statement to view the resources allocated to tenants:

   obclient> SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
   FROM
   OCEANBASE.DBA_OB_TENANTS a,
   OCEANBASE.DBA_OB_UNIT_CONFIGS b,
   OCEANBASE.DBA_OB_RESOURCE_POOLS c
   WHERE a.tenant_id=c.tenant_id
   AND b.unit_config_id = c.unit_config_id
   ORDER BY a.tenant_id desc;
   

2. Execute the following statement to scale up the CPU and memory resources for the sys tenant:

   ALTER RESOURCE UNIT unit_config_name min_cpu = 2,max_cpu = 2,memory_size = '2G',max_iops = 10000,min_iops = 10000;
   

Run the stop server command

If requests are accumulated in the request queue of the sys tenant, you can also run the stop server command to stop the OBServer node that hosts RootService and the sys tenant.

obclient> ALTER SYSTEM STOP SERVER "xx.xx.xx.xx:2882";

Restart the faulty OBServer node

Restart the OBServer node that hosts the sys tenant. For more information, see Restart an OBServer node.