sys tenant or RootService exceptions

RootService plays a crucial role in an OceanBase cluster. RootService manages cluster resources and executes distributed strategies. The sys tenant provides RootService with computing and storage resources and achieves high availability based on automatic leader election from multiple replicas. In a large or overloaded database system, the request queue of the sys tenant may be accumulated due to extreme scenarios or hardware exceptions. This topic describes the emergency procedure that you can follow to cope with an accumulated request queue.

Emergency procedure

You can use one of the following methods to relieve the request queue of the sys tenant.

Scale up the resources of the sys tenant

1. Execute the following statement to view the resources allocated to tenants:

   obclient> SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu,
   round((b.max_memory/1024/1024/1024)) max_memory,round((b.min_memory/1024/1024/1024)) min_memory
   FROM
   __all_tenant a,
   __all_unit_config b,
   __all_resource_pool c
   WHERE a.tenant_id=c.tenant_id
   AND b.unit_config_id = c.unit_config_id
   ORDER BY a.tenant_id desc;
   

2. Execute the following statement to scale up the CPU and memory resources for the sys tenant:

   ALTER RESOURCE UNIT sys_unit_config min_cpu=8,max_cpu=8, max_memory='32G',min_memory=32G', max_disk_size='500G',max_iops=10000,min_iops=10000,max_session_num=10000;
   

   Notice

   As of OceanBase Database V3.2, you can modify only the CPU and memory configurations of a resource unit. The modifications to other I/O configurations, such as the disk and input/output operations per second (IOPS), do not take effect, and the default values are used.

Perform stop server or follower-to-leader switchover

RootService manages task scheduling in an OceanBase cluster. If the request queue of the sys tenant is accumulated, you can use RootService to relieve the resource pressure by performing follower-to-leader switchover. Statement syntax:

obclient> ALTER SYSTEM SWITCH ROOTSERVICE {LEADER | FOLLOWER} {zone | server};

You can also run the stop server command to stop the OBServer node that hosts RootService and the sys tenant.

obclient> ALTER SYSTEM STOP SERVER "xx.xx.xx.xx:2882";

Restart the faulty OBServer node

Restart the OBServer node that hosts the sys tenant. For more information, see Restart an OBServer node.