Enable cgroups-based CPU isolation between tenant worker threads

Overview of cgroups

Control groups (cgroups) is a mechanism provided by the Linux kernel to consolidate (or segregate) a series of system tasks and their subtasks into different groups based on resource hierarchy on demand. This mechanism provides a unified framework for system resource management. In the current version of OceanBase Database, tenant worker threads and most background threads are tenant specific, while network threads are shared by all tenants. You can configure cgroups to control CPU utilization among the tenants.

Configure the cgroup system directory

Notice

You must configure the cgroup system directory before you install OceanBase Database.

This topic describes how to configure the cgroup system directory on one OBServer node. If the OceanBase cluster consists of multiple OBServer nodes, you must configure the cgroup system directory on each OBServer node.

1. Log on as the admin user to the server where the OBServer node resides.

2. Run the following command to mount the /sys/fs/cgroup directory.

   Note

   If the /sys/fs/cgroup directory already exists, skip this step.

   [admin@xxx /]$ sudo mount -t tmpfs cgroups /sys/fs/cgroup
   

   In this example, cgroups is a user-defined name for identification when you view the mount information.

   Mounting result:

   $df
   Filesystem      1K-blocks       Used Available Use% Mounted on
   /               293601280   28055472 265545808  10% /
   /dev/v01d      2348810240 2113955876 234854364  91% /data/1
   /dev/v02d      1300234240 1170211208 130023032  91% /data/log1
   shm              33554432          0  33554432   0% /dev/shm
   /dev/v04d       293601280   28055472 265545808  10% /home/admin/logs
   cgroups         395752136          0 395752136   0% /sys/fs/cgroup
   

3. Create a directory named /sys/fs/cgroup/cpu and change its owner. This directory is used for mounting the cpu subsystem later.

   [admin@xxx /]$ sudo mkdir /sys/fs/cgroup/cpu
   
   [admin@xxx /]$ sudo chown admin:admin -R /sys/fs/cgroup/cpu
   

   Note

   If the /sys/fs/cgroup/cpu directory already exists and is empty, skip this step.

4. Mount the cpu subsystem.

   Create a directory hierarchy named cpu, attach the cpu subsystem to this hierarchy, and mount this hierarchy to the /sys/fs/cgroup/cpu directory.

   [admin@xxx /]$ sudo mount -t cgroup -o cpu cpu /sys/fs/cgroup/cpu
   

5. Create a subdirectory named oceanbase and change its owner to admin.

   [admin@xxx /]$ sudo mkdir /sys/fs/cgroup/cpu/oceanbase
   
   [admin@xxx /]$ sudo chown admin:admin -R /sys/fs/cgroup/cpu/oceanbase
   

6. Run the following commands to set the oceanbase directory to inherit the CPU and memory configurations from the upper-level directory and set the lower-level directories of the oceanbase directory to automatically inherit its configurations.

   Notice

   At present, the cpu, cpuset, and cpuacct subsystems cannot be mounted to different directories. If they are mounted to different directories on your server, clear the mounting information and then run the sudo mount -t cgroup -o cpuset,cpu,cpuacct cpu /sys/fs/cgroup/cpu command to mount them to the same directory.

   Confirm that the cpu, cpuset, and cpuacct subsystems are mounted to the same directory and then run the following commands:

   [admin@xxx /]$ sudo sh -c "echo `cat /sys/fs/cgroup/cpu/cpuset.cpus` > /sys/fs/cgroup/cpu/oceanbase/cpuset.cpus"
   
   [admin@xxx /]$ sudo sh -c "echo `cat /sys/fs/cgroup/cpu/cpuset.mems` > /sys/fs/cgroup/cpu/oceanbase/cpuset.mems"
   
   [admin@xxx /]$ sudo sh -c "echo 1 > /sys/fs/cgroup/cpu/oceanbase/cgroup.clone_children"
   

After the cgroup system directory is configured, you can deploy OceanBase Database. For the deployment procedure, see Deploy OceanBase Database.

Establish a soft link to OceanBase Database

After OceanBase Database is installed, establish a soft link between the installation directory of OceanBase Database and the cgroup system directory.

1. Log on as the admin user to the server where the OBServer node resides.

2. Manually establish a soft link between the installation directory of OceanBase Database and the cgroup system directory.

   [admin@xxx /home/admin]$ cd /home/admin/oceanbase/
   
   [admin@xxx /home/admin]
   $ ln -sf /sys/fs/cgroup/cpu/oceanbase/ cgroup
   

   /home/admin/oceanbase/ is the installation directory of OceanBase Database.

   Execution result:

   [admin@xxx /home/admin/oceanbase]
   $ll cgroup
   lrwxrwxrwx 1 admin admin 29 Dec  8 11:09 cgroup -> /sys/fs/cgroup/cpu/oceanbase/
   

3. Restart the observer process.

   To restart the observer process, you must stop it first. For more information, see Stop or start an OBServer node.

   If the observer process detects that a soft link has been established, it will create the cgroup directory in the /sys/fs/cgroup/cpu/oceanbase/ directory.

   After the process is restarted, you can view the cgroup directory structure in the OceanBase Database installation directory /home/admin/oceanbase as follows:

   [admin@xxx /home/admin/oceanbase]
   $tree -d cgroup
   cgroup
   ├── other
   ├── tenant_0001
   │   ├── OBCG_DEFAULT
   │   ├── OBCG_ELECTION
   │   └── OBCG_ID_SERVICE
   ├── tenant_0506
   │   └── OBCG_DEFAULT
   ├── tenant_0507
   │   └── OBCG_DEFAULT
   ├── tenant_0508
   │   └── OBCG_DEFAULT
   ├── tenant_0509
   │   └── OBCG_DEFAULT
   ├── tenant_0510
   │   └── OBCG_DEFAULT
   ├── tenant_0512
   │   └── OBCG_DEFAULT
   ├── tenant_0999
   │   └── OBCG_DEFAULT
   ├── tenant_1001
   │   ├── OBCG_DEFAULT
   │   ├── OBCG_ELECTION
   │   └── OBCG_ID_SERVICE
   └── tenant_1002
       ├── OBCG_DEFAULT
       └── OBCG_ELECTION
   

Enable the cgroup feature

In OceanBase Database, the cluster-level enable_cgroup parameter specifies whether to enable the cgroup feature for an OBServer node. By default, the parameter is set to False, which indicates that the cgroup feature is disabled. You can perform the following steps to enable the cgroup feature:

1. Log on to the sys tenant as the root user.

2. Execute the following statement to enable the cgroup feature:

   obclient> ALTER SYSTEM SET enable_cgroup=true;
   

   or

   obclient> ALTER SYSTEM SET enable_cgroup=1;
   

   or

   obclient> ALTER SYSTEM SET enable_cgroup=ON;
   

Control tenant resources by using cgroups

After you configure the cgroup system directory and enable the cgroup feature, you can configure the cpu.cfs_period_us, cpu.cfs_quota_us, and cpu.shares files in the directory of a tenant to control its CPU utilization.

• Specify the upper limit of the CPU time that can be used by a tenant in a CPU period by configuring cpu.cfs_quota_us and cpu.cfs_period_us.

  The cpu.cfs_quota_us and cpu.cfs_period_us files control the absolute CPU time available to processes in a tenant.

  • cpu.cfs_period_us: the length of a single CPU period, in μs. Default value: 100000, which is equivalent to 100 ms. The total length of CPU periods in the system is Number of CPU cores × cfs_period_us.

  • cpu.cfs_quota_us: the amount of available CPU time within a period, in μs. Default value: 100000. Value range: [1000, 1000000].

  You can use cpu.cfs_quota_us together with cpu.cfs_period_us to set, for a single CPU core, the maximum percentage of CPU time available to processes in a tenant. Formula: Maximum percentage of CPU time in a single CPU core available to a tenant = cfs_quota_us/cfs_period_us.

  If the value of cpu.cfs_quota_us is twice the value of cpu.cfs_period_us, the processes in the tenant can fully use two CPU cores.

  Example: Configure to allow processes in the tenant_1001 tenant to use at most 20% CPU time of one CPU core.

  [admin@xxx /]$ cd /sys/fs/cgroup/cpu/oceanbase/tenant_1001
  
  [admin@xxx /sys/fs/cgroup/cpu/oceanbase/tenant_1001/]
  $ echo 50000 > cpu.cfs_period_us
  
  [admin@xxx /sys/fs/cgroup/cpu/oceanbase/tenant_1001/]
  $ echo 10000 > cpu.cfs_quota_us
  

  Example: Configure to allow processes of the tenant_1002 tenant to fully use four CPU cores.

  [admin@xxx /]$ cd /sys/fs/cgroup/cpu/oceanbase/tenant_1002
  
  [admin@xxx /sys/fs/cgroup/cpu/oceanbase/tenant_1002]
  $ echo 1000000 > cpu.cfs_period_us
  
  [admin@xxx /sys/fs/cgroup/cpu/oceanbase/tenant_1002]
  $ echo 4000000 > cpu.cfs_quota_us
  

• Use the cpu.shares file to set the CPU allocation ratio.

  cpu.shares specifies the ratio of CPU resources allocated to processes in a tenant to all the CPU resources in the system. The default value of the cpu.shares file is 1024.

  Assume that the system has two tenants, namely, tenant_1001 and tenant_1002. For tenant_1001, set cpu.shares to 1024. For tenant_1002, set cpu.shares to 512.

  The settings for the tenant_1001 tenant are as follows:

  [admin@xxx /]$ cd /sys/fs/cgroup/cpu/oceanbase/tenant_1001
  
  [admin@xxx /sys/fs/cgroup/cpu/oceanbase/tenant_1001]
  $ echo 1024 > cpu.shares
  

  The settings for the tenant_1002 tenant are as follows:

  [admin@xxx /]$ cd /sys/fs/cgroup/cpu/oceanbase/tenant_1002
  
  [admin@xxx /sys/fs/cgroup/cpu/oceanbase/tenant_1002]
  $ echo 512 > cpu.shares
  

  When the processes in both tenants are running at full capacity, the tenant_1001 tenant gets 1024/(1024+512)=66% CPU resources, and the tenant_1002 tenant gets 33% CPU resources. If processes in one tenant are idle, the processes in the other tenant can use all CPU resources.

References

Clear cgroup configurations