Configure cgroups

This topic describes how to configure cgroups in OceanBase Database.

Background information

In the current version of OceanBase Database, worker threads and most background threads are distinguished based on the tenant, and network threads are shared threads. Therefore, you can control the CPU usage of the tenant through cgroup configuration. Before you configure cgroups, we recommend that you learn about the concept of cgroup first. For more information, see Overview.

Limitations and considerations

• Resource isolation considerably compromises performance. Therefore, we recommend that you do not use the cgroup feature to isolate tenant resources in the following scenarios:

  • Single-tenant scenarios that have only one tenant in the cluster.

  • Scenarios where tenants are associated with each other. For example, multiple tenants serve different microservices, resulting in an upstream and downstream relationship among the tenants.

  • Small-scale tenant scenarios where each tenant has two or four CPU cores.

• If the operating system of the OBServer node is Alibaba Cloud Linux, to use the cgroup feature, the operating system version must be 4.19 or later.

• The cgroup feature compromises performance of OceanBase Database. Therefore, weigh the isolation benefits and performance loss before you enable the cgroup feature.

Configure cgroups in OceanBase Database Enterprise Edition

Step 1: Configure the cgroup system directory

This section describes how to configure the cgroup system directory on one OBServer node. If the OceanBase cluster consists of multiple OBServer nodes, you must configure the cgroup system directory on each OBServer node.

1. Log in to the OBServer server as the admin user.

2. Run the following command to mount the /sys/fs/cgroup directory.

   Note

   If the /sys/fs/cgroup directory already exists, skip this step.

   [admin@xxx /]$ sudo mount -t tmpfs cgroups /sys/fs/cgroup
   

   Here, cgroups is a user-defined name for identification when you view the mount information.

   The mounting result is as follows:

   $df
   Filesystem      1K-blocks       Used Available Use% Mounted on
   /               293601280   28055472 265545808  10% /
   /dev/v01d      2348810240 2113955876 234854364  91% /data/1
   /dev/v02d      1300234240 1170211208 130023032  91% /data/log1
   shm              33554432          0  33554432   0% /dev/shm
   /dev/v04d       293601280   28055472 265545808  10% /home/admin/logs
   cgroups         395752136          0 395752136   0% /sys/fs/cgroup
   

3. Create a directory named /sys/fs/cgroup/cpu and change its owner. This directory is used for mounting the cpu subsystem later.

   Note

   If the /sys/fs/cgroup/cpu directory already exists and is empty, skip this step.

   [admin@xxx /]$ sudo mkdir /sys/fs/cgroup/cpu
   
   [admin@xxx /]$ sudo chown admin:admin -R /sys/fs/cgroup/cpu
   

4. Create a hierarchy named cpu, attach the cpu subsystem to this hierarchy, and mount it to the /sys/fs/cgroup/cpu directory.

   [admin@xxx /]$ sudo mount -t cgroup -o cpu cpu /sys/fs/cgroup/cpu
   

5. Create a subdirectory named oceanbase and change its owner to admin.

   [admin@xxx /]$ sudo mkdir /sys/fs/cgroup/cpu/oceanbase
   
   [admin@xxx /]$ sudo chown admin:admin -R /sys/fs/cgroup/cpu/oceanbase
   

6. Allocate CPU and memory resources for the oceanbase directory.

   1. Execute the following command to check the mounting status of the cpu, cpuacct, and cpuset subsystems on your machine.

      [admin@xxx /]$ ll /sys/fs/cgroup 
      

   2. Based on the subsystems' mounting status, proceed with the appropriate action:

      • Scenario 1: The cpuset subsystem is mounted alongside cpu and cpuacct.

        In this scenario, typically all three subsystems are mounted under the same directory, as shown in the example below:

        drwxr-xr-x 3 root root 0 Jul 24 2020 blkio
        lrwxrwxrwx 1 root root 33 Jul 24 2020 cpu -> /sys/fs/cgroup/cpuset,cpu,cpuacct
        lrwxrwxrwx 1 root root 33 Jul 24 2020 cpuacct -> /sys/fs/cgroup/cpuset,cpu,cpuacct
        lrwxrwxrwx 1 root root 33 Jul 24 2020 cpuset -> /sys/fs/cgroup/cpuset,cpu,cpuacct
        drwxr-xr-x 4 root root 0  Jul 24 2020 cpuset,cpu,cpuacct   
        

        For this setup, execute the following commands to allocate CPU and memory resources for the oceanbase directory:

        [admin@xxx /]$ sudo sh -c "echo `cat /sys/fs/cgroup/cpu/cpuset.cpus` > /sys/fs/cgroup/cpu/oceanbase/cpuset.cpus"
        
        [admin@xxx /]$ sudo sh -c "echo `cat /sys/fs/cgroup/cpu/cpuset.mems` > /sys/fs/cgroup/cpu/oceanbase/cpuset.mems"
        

      • Scenario 2: The cpuset subsystem is mounted separately from cpu and cpuacct.

        In this case, where the cpuset subsystem is independently mounted from cpu and cpuacct, a configuration frequently encountered in Elastic Compute Service (ECS) environments, as shown below:

        drwxr-xr-x 2 root root 40 02/07 15:27 blkio
        lrwxrwxrwx 1 root root 11 02/07 15:27 cpu -> cpu,cpuacct
        lrwxrwxrwx 1 root root 11 02/07 15:27 cpuacct -> cpu,cpuacct
        drwxr-xr-x 2 root root 40 02/07 15:27 cpu,cpuacct
        drwxr-xr-x 2 root root 40 02/07 15:27 cpuset
        

        In this scenario, no additional actions are needed. You can proceed to the next step.

7. Execute the following command to set the inheritance property for subdirectories in the oceanbase directory.

   [admin@xxx /]$ sudo sh -c "echo 1 > /sys/fs/cgroup/cpu/oceanbase/cgroup.clone_children"
   

   After the command executes successfully, any cgroup subdirectories created under the oceanbase directory will inherit the properties of the parent directory.

Step 2: Deploy OceanBase Database

After the cgroup system directory is configured, you can deploy OceanBase Database Enterprise Edition. For the deployment procedure, see Deploy OceanBase Database.

Step 3: Establish a soft link to OceanBase Database

After OceanBase Database is installed, establish a soft link between the installation directory of OceanBase Database and the cgroup system directory.

1. Log in to the OBServer node as the admin user.

2. Manually establish a soft link between the installation directory of OceanBase Database and the cgroup system directory.

   [admin@xxx /home/admin]$ cd /home/admin/oceanbase/
   
   [admin@xxx /home/admin]
   $ ln -sf /sys/fs/cgroup/cpu/oceanbase/ cgroup
   

   Here, /home/admin/oceanbase/ is the installation directory of OceanBase Database.

   The execution result is as follows:

   [admin@xxx /home/admin/oceanbase]
   $ll cgroup
   lrwxrwxrwx 1 admin admin 29 Dec  8 11:09 cgroup -> /sys/fs/cgroup/cpu/oceanbase/
   

3. Restart the observer process.

   You must first stop the observer process and then restart it. For more information, see Restart a node.

   If the observer process detects that a soft link has been established, it will create the cgroup directory in the /sys/fs/cgroup/cpu/oceanbase/ directory.

Step 4: Enable the cgroup feature

In OceanBase Database, the cluster-level enable_cgroup parameter specifies whether to enable the cgroup feature for the OBServer node. The default value is True, which specifies to enable this feature. If this feature is disabled, perform the following steps to enable it.

1. Log in to the sys tenant of the cluster as the root user.

2. Execute the following statement to enable the cgroup feature:

   obclient> ALTER SYSTEM SET enable_cgroup=true;
   

   or

   obclient> ALTER SYSTEM SET enable_cgroup=1;
   

   or

   obclient> ALTER SYSTEM SET enable_cgroup=ON;
   

Configure cgroups in OceanBase Database Community Edition

Step 1: Configure the cgroup system directory

This section describes how to configure the cgroup system directory on one OBServer node as the usercg user. If the OceanBase cluster consists of multiple OBServer nodes, you must configure the cgroup system directory on each OBServer node.

1. Log in to the OBServer server as the usercg user.

2. Run the following command to mount the /sys/fs/cgroup directory.

   Note

   If the /sys/fs/cgroup directory already exists, skip this step.

   [usercg@xxx /]$ sudo mount -t tmpfs cgroups /sys/fs/cgroup
   

   Here, cgroups is a user-defined name for identification when you view the mount information.

   The mounting result is as follows:

   $df
   Filesystem      1K-blocks       Used Available Use% Mounted on
   /               293601280   28055472 265545808  10% /
   /dev/v01d      2348810240 2113955876 234854364  91% /data/1
   /dev/v02d      1300234240 1170211208 130023032  91% /data/log1
   shm              33554432          0  33554432   0% /dev/shm
   /dev/v04d       293601280   28055472 265545808  10% /home/usercg/logs
   cgroups         395752136          0 395752136   0% /sys/fs/cgroup
   

3. Create a directory named /sys/fs/cgroup/cpu and change its owner. This directory is used for mounting the cpu subsystem later.

   [usercg@xxx /]$ sudo mkdir /sys/fs/cgroup/cpu
   
   [usercg@xxx /]$ sudo chown usercg:usercg -R /sys/fs/cgroup/cpu
   

   Note

   If the /sys/fs/cgroup/cpu directory already exists and is empty, skip this step.

4. Mount the cpu subsystem.

   Create a hierarchy named cpu, attach the cpu subsystem to this hierarchy, and mount it to the /sys/fs/cgroup/cpu directory.

   [usercg@xxx /]$ sudo mount -t cgroup -o cpu cpu /sys/fs/cgroup/cpu
   

5. Create a subdirectory named oceanbase and change its owner to usercg.

   [usercg@xxx /]$ sudo mkdir /sys/fs/cgroup/cpu/oceanbase
   
   [usercg@xxx /]$ sudo chown usercg:usercg -R /sys/fs/cgroup/cpu/oceanbase
   

6. Execute the following commands to configure the oceanbase directory to inherit the CPU and memory configurations from its parent directory, and set it to automatically inherit these configurations for its subdirectories.

   Notice

   Currently, mounting the cpu, cpuset, and cpuacct subsystems on different directories is not supported. If these three subsystems are mounted on different directories on your machine, you need to clear the mount information first, and then execute the command sudo mount -t cgroup -o cpuset,cpu,cpuacct cpu /sys/fs/cgroup/cpu to remount these three subsystems in the same directory.

   After confirming that the cpu, cpuset, and cpuacct subsystems are mounted in the same directory, execute the following commands:

   [usercg@xxx /]$ sudo sh -c "echo `cat /sys/fs/cgroup/cpu/cpuset.cpus` > /sys/fs/cgroup/cpu/oceanbase/cpuset.cpus"
   
   [usercg@xxx /]$ sudo sh -c "echo `cat /sys/fs/cgroup/cpu/cpuset.mems` > /sys/fs/cgroup/cpu/oceanbase/cpuset.mems"
   
   [usercg@xxx /]$ sudo sh -c "echo 1 > /sys/fs/cgroup/cpu/oceanbase/cgroup.clone_children"
   

Step 2: Deploy OceanBase Database

After the cgroup system directory is configured, you can deploy OceanBase Database Community Edition. For more information about how to deploy OceanBase Database Community Edition, see Deploy OceanBase Database on CLI in a production environment.

Step 3: Establish a soft link to OceanBase Database

After OceanBase Database is installed, establish a soft link between the installation directory of OceanBase Database and the cgroup system directory.

1. Log in to the OBServer node as the usercg user.

2. Manually establish a soft link between the installation directory of OceanBase Database and the cgroup system directory.

   [usercg@xxx /home/usercg]$ cd /home/usercg/oceanbase/
   
   [usercg@xxx /home/usercg]
   $ ln -sf /sys/fs/cgroup/cpu/oceanbase/ cgroup
   

   Here, /home/usercg/oceanbase/ is the installation path of OceanBase Database.

   The execution result is as follows:

   [usercg@xxx /home/usercg/oceanbase]
   $ll cgroup
   lrwxrwxrwx 1 usercg usercg 29 Dec  8 11:09 cgroup -> /sys/fs/cgroup/cpu/oceanbase/
   

3. Restart the observer process.

   You must first stop the observer process and then restart it. For more information, see Restart a node.

   If the observer process detects that a soft link has been established, it will create the cgroup directory in the /sys/fs/cgroup/cpu/oceanbase/ directory.

Step 4: Enable the cgroup feature

In OceanBase Database, the cluster-level enable_cgroup parameter specifies whether to enable the cgroup feature for the OBServer node. The default value is True, which specifies to enable this feature. If this feature is disabled, perform the following steps to enable it.

1. Log in to the sys tenant of the cluster as the root user.

2. Execute the following statement to enable the cgroup feature:

   obclient> ALTER SYSTEM SET enable_cgroup=true;
   

   or

   obclient> ALTER SYSTEM SET enable_cgroup=1;
   

   or

   obclient> ALTER SYSTEM SET enable_cgroup=ON;
   

What to do next

After you configure the cgroup system directory and enable the cgroup feature, in the case of emergencies, you can control the utilization of CPU resources in a tenant by using the cpu.cfs_period_us, cpu.cfs_quota_us, and cpu.shares files in the directory of the tenant. Generally, we recommend that you do not implement resource isolation in this way.

We recommend that you use the files in the cgroup system directory to call the CREATE_CONSUMER_GROUP subprogram in the DBMS_RESOURCE_MANAGER package to create resource groups for resource isolation. For more information, see Configure resource isolation within a tenant.

References

Clear cgroup configurations