This topic describes how to configure resource isolation in MySQL-compatible mode.
Configure resource isolation for CPU or IOPS resources
Prerequisites
Before you configure resource isolation, we recommend that you understand basic concepts such as resource groups, resource management plans, and their contents. For more information about resource isolation concepts and application scenarios, see Resource isolation overview.
CPU resource isolation depends on cgroup. To control CPU resource isolation, you must configure the cgroup directory and enable the cgroup feature before configuring resource isolation. For more information about how to configure the cgroup directory and enable the cgroup feature, see Configure cgroup.
When configuring user-level or function-level resource isolation, if you do not need to control CPU resource isolation (only IOPS resource isolation), you do not need to configure cgroup. When configuring SQL-level resource isolation, you must configure cgroup regardless of whether you need to control CPU resource isolation.
Before performing IOPS resource isolation, you can perform disk performance calibration. For more information about how to perform disk performance calibration, see Disk performance calibration.
Note
In the current version, IOPS resource isolation is no longer heavily dependent on disk performance calibration.
Ensure that you have created the user for whom resource isolation is required. For more information about how to create a user, see Create a user.
If you need to configure SQL-level resource isolation, ensure that you have created the database, table, and column for which resource isolation is required.
If you need to configure database-level resource isolation, ensure that database-level resource isolation is enabled. That is, the value of the tenant-level hidden parameter
_enable_database_isolation_modehas been set toTrue.In a user tenant, the statement to enable database-level resource isolation is as follows:
obclient(root@mysql001)[(none)]> ALTER SYSTEM SET _enable_database_isolation_mode = True;After modification, it takes effect only after a restart.
Background information
Resource isolation is divided into three types: user-level resource isolation, SQL-level resource isolation, and function-level resource isolation. For more information about these three types of resource isolation, see Resource isolation overview.
Step 1 (Optional): Configure the tenant's MAX_IOPS and MIN_IOPS to valid values
Note
If you have already set MAX_IOPS and MIN_IOPS to the IOPS values corresponding to 16 KB reads when creating the tenant's unit specification, or if you do not need to control IOPS resource isolation, skip this step.
After completing disk calibration, before configuring a resource isolation plan, you must ensure that the MAX_IOPS and MIN_IOPS values in the tenant's unit specification are valid. Valid values refer to the IOPS values corresponding to 16 KB reads, which serve as reference values for the tenant's IOPS configuration.
Log in to the
systenant of the cluster as therootuser.Run the following command to view the unit specification of the tenant for which resource isolation is to be configured.
obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_UNIT_CONFIGS;An example of the query result is as follows:
+----------------+-----------------+----------------------------+----------------------------+---------+---------+-------------+---------------+---------------------+---------------------+-------------+ | UNIT_CONFIG_ID | NAME | CREATE_TIME | MODIFY_TIME | MAX_CPU | MIN_CPU | MEMORY_SIZE | LOG_DISK_SIZE | MAX_IOPS | MIN_IOPS | IOPS_WEIGHT | +----------------+-----------------+----------------------------+----------------------------+---------+---------+-------------+---------------+---------------------+---------------------+-------------+ | 1 | sys_unit_config | 2023-12-19 13:55:04.463295 | 2023-12-19 13:56:08.969718 | 3 | 3 | 2147483648 | 3221225472 | 9223372036854775807 | 9223372036854775807 | 3 | | 1001 | small_unit | 2023-12-19 13:56:09.851665 | 2023-12-19 13:56:09.851665 | 1 | 1 | 2147483648 | 6442450944 | 9223372036854775807 | 9223372036854775807 | 1 | | 1002 | medium_unit | 2023-12-19 13:56:10.030914 | 2023-12-19 13:56:10.030914 | 8 | 4 | 8589934592 | 25769803776 | 9223372036854775807 | 9223372036854775807 | 4 | | 1003 | large_unit | 2023-12-19 13:56:10.112115 | 2023-12-19 13:56:10.112115 | 16 | 8 | 21474836480 | 64424509440 | 9223372036854775807 | 9223372036854775807 | 8 | +----------------+-----------------+----------------------------+----------------------------+---------+---------+-------------+---------------+---------------------+---------------------+-------------+ 4 rows in setBased on the query results, if both the tenant's
MAX_IOPSandMIN_IOPSare the default valueINT64_MAX(9223372036854775807), you need to replan the IOPS resources available to the tenant.Run the following command to confirm on which OBServer nodes the tenant is deployed.
obclient [oceanbase]> SELECT DISTINCT SVR_IP, SVR_PORT FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE tenant_id = xxxx;An example of the query result is as follows:
+----------------+----------+ | SVR_IP | SVR_PORT | +----------------+----------+ | xx.xxx.xxx.xx1 | xxxx1 | | xx.xxx.xxx.xx1 | xxxx2 | | xx.xxx.xxx.xx1 | xxxx3 | +----------------+----------+ 3 rows in setRun the following command to confirm the disk calibration values on the OBServer node where the tenant to be isolated resides. Use the disk calibration value for a 16 KB read as the upper limit for setting the node's IOPS.
obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_IO_BENCHMARK WHERE MODE='READ' AND SIZE=16384;An example of the query result is as follows:
+----------------+----------+--------------+------+-------+-------+------+---------+ | SVR_IP | SVR_PORT | STORAGE_NAME | MODE | SIZE | IOPS | MBPS | LATENCY | +----------------+----------+--------------+------+-------+-------+------+---------+ | xx.xxx.xxx.xx1 | xxxx1 | DATA | READ | 16384 | 48162 | 752 | 331 | | xx.xxx.xxx.xx1 | xxxx2 | DATA | READ | 16384 | 47485 | 741 | 336 | | xx.xxx.xxx.xx1 | xxxx3 | DATA | READ | 16384 | 48235 | 753 | 331 | +----------------+----------+--------------+------+-------+-------+------+---------+ 3 rows in setBased on the query results, use the obtained disk calibration values for each node as the upper limits to plan the IOPS available to the tenant. Since multiple tenants may be deployed on the same OBServer node in the cluster, you need to allocate these IOPS based on your actual business requirements.
Suppose there are two tenants in a cluster, both deployed on the same OBServer node. The disk IOPS baseline for a 16 KB read is the same for both OBServer nodes, at 20000 IOPS. Also, assume the load for both tenants is roughly equal. Then, you can divide the 20000 IOPS equally between the two tenants (you can adjust the IOPS allocated to each tenant based on your actual business needs), setting both tenants'
MAX_IOPSandMIN_IOPSto 10000. Alternatively, based on your business considerations, you can setMIN_IOPSto a value less thanMAX_IOPS.Run the following command to modify the tenant's
MAX_IOPSandMIN_IOPSvalues.It is recommended to modify
MIN_IOPSfirst, thenMAX_IOPS.ALTER RESOURCE UNIT unit_name MIN_IOPS = xxx;ALTER RESOURCE UNIT unit_name MAX_IOPS = xxx;
Step 2: Configure a resource isolation plan
Assume there are already two users, tp_user and ap_user, in the current tenant.
You can follow these steps to configure a resource isolation plan to control different users or background tasks from using different CPU or IOPS resources.
Log in to the MySQL-compatible tenant of the cluster as a tenant administrator.
Call the
CREATE_CONSUMER_GROUPsubprogram in theDBMS_RESOURCE_MANAGERsystem package to create a resource group.The statement is as follows:
CALL DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP( CONSUMER_GROUP => 'group_name' , COMMENT => 'coments' );The relevant parameters are described as follows:
CONSUMER_GROUP: Defines the name of the resource group.COMMENT: Provides remarks for the resource group.
For example, create the following two resource groups:
interactive_groupandbatch_group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP( CONSUMER_GROUP => 'interactive_group' , COMMENT => 'TP' );obclient [test]> CALL DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP( CONSUMER_GROUP => 'batch_group' , COMMENT => 'AP' );After successful creation, you can query the
oceanbase.DBA_RSRC_CONSUMER_GROUPSview to confirm it. For detailed information about theoceanbase.DBA_RSRC_CONSUMER_GROUPSview, see oceanbase.DBA_RSRC_CONSUMER_GROUPS.Call the
CREATE_PLANsubprogram in theDBMS_RESOURCE_MANAGERsystem package to create a resource management plan.The statement is as follows:
CALL DBMS_RESOURCE_MANAGER.CREATE_PLAN( PLAN => 'plan_name', comment => 'coments');The relevant parameters are described as follows:
PLAN: Defines the name of the resource management plan.COMMENT: Provides remarks for the resource management plan.
For example, create a resource management plan named
daytimeand add some remarks.obclient [test]> CALL DBMS_RESOURCE_MANAGER.CREATE_PLAN( PLAN => 'daytime', comment => 'TPFirst');After successful creation, you can query the
oceanbase.DBA_RSRC_PLANSview to confirm it. For detailed information about theoceanbase.DBA_RSRC_PLANSview, see oceanbase.DBA_RSRC_PLANS.Call the
CREATE_PLAN_DIRECTIVEsubprogram in theDBMS_RESOURCE_MANAGERsystem package to create the content of the resource management plan. This is used to limit the CPU and IOPS resources used by a resource group when the resource management plan is enabled.The statement is as follows:
CALL DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE ( PLAN => 'plan_name', GROUP_OR_SUBPLAN => 'group_name' , COMMENT => 'comments', MGMT_P1 => int_value, UTILIZATION_LIMIT => int_value, MIN_IOPS => int_value, MAX_IOPS => int_value, WEIGHT_IOPS => int_value, MAX_NET_BANDWIDTH => int_value, NET_BANDWIDTH_WEIGHT => int_value);The relevant parameters are described as follows:
PLAN: Specifies the resource management plan name.GROUP_OR_SUBPLAN: Specifies the resource group.COMMENT: Provides remarks for the resource management plan content. The default value isNULL.MGMT_P1: Specifies the maximum relative CPU utilization when the system is at full load. The default value is100.UTILIZATION_LIMIT: Specifies the upper limit of CPU resources that a resource group can use. The default value of this parameter is100, and its value range is (0, 100].100indicates that the tenant can use all CPU resources. If the value is70, it means the tenant can use up to 70% of the CPU resources.MIN_IOPS: Reserves IOPS resources for the resource group in case of I/O contention. The total reserved IOPS cannot exceed 100. The default value is0.MAX_IOPS: Specifies the maximum total IOPS that the resource group can use. The total IOPS can exceed 100. The default value is100.WEIGHT_IOPS: Specifies the weight value of IOPS. The total weight can exceed 100. The default value is0.MAX_NET_bandWIDTH: Specifies the maximum network bandwidth resources that can be used. The total bandwidth can exceed 100. The default value is100.NET_BANDWIDTH_WEIGHT: Specifies the weight value of network bandwidth. The total weight can exceed 100, and the bandwidth is allocated proportionally. The default value is0.
Examples:
Specify the resource plan as
daytime, bind it to the resource groupinteractive_group, and set the upper limit of available CPU resources to 80% of the tenant's total CPU resources. Also, specify that the minimum available IOPS resources are 30%, the upper limit of available IOPS resources are 90% of the total IOPS resources, the IOPS resource weight is 80, the available network bandwidth resources are 40%, and the network bandwidth weight is 40%.obclient [test]> CALL DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE( PLAN => 'daytime', GROUP_OR_SUBPLAN => 'interactive_group' , COMMENT => '', UTILIZATION_LIMIT =>80, MIN_IOPS => 30, MAX_IOPS => 90, WEIGHT_IOPS => 80, MAX_NET_BANDWIDTH => 40, NET_BANDWIDTH_WEIGHT => 40);Specify the resource plan as
daytime, bind it to the resource groupbatch_group, and set the upper limit of available CPU resources to 40% of the tenant's total CPU resources. Also, specify that the minimum available IOPS resources are 40%, the upper limit of available IOPS resources are 80% of the total IOPS resources, the IOPS resource weight is 70, the available network bandwidth resources are 30%, and the network bandwidth weight is 30.obclient [test]> CALL DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE( PLAN => 'daytime', GROUP_OR_SUBPLAN => 'batch_group' , COMMENT => '', UTILIZATION_LIMIT => 40, MIN_IOPS => 40, MAX_IOPS => 80, WEIGHT_IOPS => 70, MAX_NET_BANDWIDTH => 30, NET_BANDWIDTH_WEIGHT => 30);
After successful creation, you can query the
oceanbase.DBA_RSRC_PLAN_DIRECTIVESview and theoceanbase.DBA_OB_RSRC_IO_DIRECTIVESview to confirm the settings.For more information about the
oceanbase.DBA_RSRC_PLAN_DIRECTIVESview, see oceanbase.DBA_RSRC_PLAN_DIRECTIVES.For more information about the
oceanbase.DBA_OB_RSRC_IO_DIRECTIVESview, see oceanbase.DBA_OB_RSRC_IO_DIRECTIVES.Call the
SET_CONSUMER_GROUP_MAPPINGsubprogram in theDBMS_RESOURCE_MANAGERsystem package to create resource isolation matching rules based on your actual usage scenario.The statement is as follows:
CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'column | user | function | database', VALUE => 'values', CONSUMER_GROUP => 'group_name');The relevant parameters are described as follows:
ATTRIBUTE: Specifies the attribute type. Attribute names are case-insensitive.column: Indicates SQL-level resource isolation.user: Indicates user-level resource isolation.function: Indicates function-level resource isolation.database: Indicates database-level resource isolation.
VALUE: Specifies the attribute value.If the attribute type is
column, you need to specify information such as the database name, table name, column name, constant value, and username.Among these:
The database name and username are optional. The default database name is the current database name. If no username is specified, it applies to all users, including those created subsequently in the current tenant.
The table name, column name, and constant value are required, and only one value can be specified for each. When specifying a constant value, only numeric or string values are supported.
When specifying a table name, column name, or username, the specified table, column, and user must exist.
If the attribute type is
user, you only need to specify the username here. Currently, only one user is supported.If the attribute type is
function, you need to specify one of the following background tasks corresponding to the DAG thread: compaction_high, ha_high, compaction_mid, ha_mid, compaction_low, ha_low, ddl, ddl_high, clog_high, and opt_stats. Currently, only one task is supported. For detailed descriptions of each task, see Resource isolation overview.If the attribute type is
database, you need to specify the database name here. Currently, only one database name is supported.
CONSUMER_GROUP: Specifies the resource group to bind. This indicates which resource group the statement will be bound to when the SQL matches the rule set inVALUE. Currently, only binding to one resource group is supported.If no resource group is specified, the system's built-in
OTHER_GROUPSis used by default. The resources corresponding to the system's built-inOTHER_GROUPSare as follows:MIN_IOPS = 100 - SUM(the sum of other resource groups within the tenant)
MAX_IOPS = 100
WEIGHT_IOPS = 100
Examples:
Create an SQL-level resource isolation matching rule.
Specify that when a user
tp_userexecutes an SQL statement containing theWHEREconditiontest.t.c3 = 3, this SQL statement will be bound to a resource group namedbatch_groupfor execution, using the CPU and IOPS resources constrained by that resource group.Notice
When executing an SQL statement, the statement does not necessarily have to contain
test.t.. As long asc3is ultimately resolved totest.t.c3, the SQL statement will be bound to the resource group namedbatch_groupfor execution. For example:SELECT * FROM test.t WHERE c3 = 1;.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'column', VALUE => 'test.t.c3=3 for tp_user', CONSUMER_GROUP => 'batch_group');Specify that when executing an SQL statement containing the
WHEREconditiont.c3=5, this SQL statement will be bound to a resource group namedinteractive_groupfor execution, using the CPU and IOPS resources constrained by that resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'column', VALUE => 't.c3=5', CONSUMER_GROUP => 'interactive_group');
Besides binding resource groups by calling the
SET_CONSUMER_GROUP_MAPPINGsubprogram, OceanBase Database also supports binding resource groups using hints. You can use hints to flexibly commit the SQL statements to be executed to a specified resource group. For example, if the SQL statement to be executed isSELECT * FROM Tand you want this SQL to use the resources constrained by thebatch_groupresource group, an example of binding a resource group using hints is as follows:obclient [test]> SELECT /*+resource_group('batch_group')*/ * FROM t;Note
After specifying a resource group using a hint, if the resource group does not exist, the default resource group
OTHER_GROUPSis used during statement execution.Create user-level resource isolation matching rules
Bind the SQL statements executed by the
tp_useruser to the resource group namedinteractive_groupfor execution, and use the CPU and IOPS resources constrained by this resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'user', VALUE => 'tp_user', CONSUMER_GROUP => 'interactive_group');Bind the SQL statements executed by the
ap_useruser to the resource group namedbatch_groupfor execution, and use the CPU and IOPS resources constrained by this resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'user', VALUE => 'ap_user', CONSUMER_GROUP => 'batch_group');
Create a function resource isolation matching rule.
When executing tasks with the
compaction_highpriority, bind the system to the resource group namedinteractive_groupfor execution, and use the CPU and IOPS resources constrained by this resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'function', VALUE => 'compaction_high', CONSUMER_GROUP => 'interactive_group');When executing tasks with the
ddl_highpriority, bind the system to the resource group namedbatch_groupfor execution, and use the CPU and IOPS resources constrained by this resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'function', VALUE => 'ddl_high', CONSUMER_GROUP => 'batch_group');
Create a database resource isolation matching rule.
When operating on the database
db1, bind the system to the resource group namedinteractive_groupfor execution, and use the CPU and IOPS resources constrained by this resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'database', VALUE => 'db1', CONSUMER_GROUP => 'interactive_group');When operating on the database
db2, bind the system to the resource group namedbatch_groupfor execution, and use the CPU and IOPS resources constrained by this resource group.obclient [test]> CALL DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING( ATTRIBUTE => 'database', VALUE => 'db2', CONSUMER_GROUP => 'batch_group');
After successful creation, you can query the
oceanbase.DBA_RSRC_GROUP_MAPPINGSview to confirm it. For detailed information about theoceanbase.DBA_RSRC_GROUP_MAPPINGSview, see oceanbase.DBA_RSRC_GROUP_MAPPINGS.Enable an appropriate resource management plan for the resource group.
Since different resource management plans may impose different constraints on the same resource group, you need to enable an appropriate resource management plan for the resource group.
obclient [test]> SET GLOBAL resource_manager_plan = 'daytime';Note
If resource constraints are not required, you can disable all resource plans by using the
SET GLOBAL resource_manager_plan = '';statement.
Considerations after configuration
After a resource isolation matching rule is added, if a user is deleted and then recreated, the resource isolation matching rule still applies.
A resource isolation matching rule does not take effect immediately after being added. It is expected to start taking effect within 10 seconds, subject to the actual environment.
In the same tenant, if isolation strategies of multiple levels are configured simultaneously, their priorities are as follows:
SQL-level resource isolation > Database-level resource isolation > User-level resource isolation > Function-level resource isolation
After a resource isolation matching rule is added, it currently only takes effect on statements such as
SELECT,INSERT,UPDATE, andDELETE. It does not take effect on DDL and DCL statements, nor on PL statements. However, it does take effect in prepared statements.
Performance impact after parameter configuration
User-level resource isolation and function-level resource isolation can determine which resource group's resources to use before SQL parsing is complete, so they have no impact on performance.
The performance impact of SQL-level resource isolation mainly comes from retries. Unlike user-level and function-level resource isolation, which determine which resource group's resources to use before an SQL statement is parsed, SQL-level resource isolation determines which resource group's resources to use during SQL parsing or when the plan cache is hit. If it is found that the resource group to be used is different from the one currently in use, the system will perform a retry and use the resource group corresponding to the matching rule to process the SQL.
The performance impact of SQL-level resource isolation is mainly divided into the following three scenarios:
When an SQL statement does not match any rule, there is almost no impact on performance.
When an SQL statement matches a rule, assuming the resource group specified by that rule is
batch_group, not only will this SQL statement ultimately be executed using the resources ofbatch_group, but the next SQL statement will also first use the resources ofbatch_group. The system will only retry when it encounters another rule that requires switching to a different resource group. For scenarios where a batch of SQL statements are executed consecutively and all these SQL statements are bound to the same resource group, using this strategy can achieve retries only for the first SQL statement in the batch, with subsequent SQL statements not requiring retries. This minimizes retries and has a minimal impact on performance.When the expected resource group for each SQL statement is different from the previous one, each SQL statement needs to be retried once, which has a significant impact on performance.
(Optional) Configure resource isolation for DDL worker threads
In the current version, DDL requests compete with DML requests for front-end worker threads. When a large number of DDL requests arrive in a short period, it can severely impact normal user requests. To minimize performance and stability issues caused by thread resource contention, OceanBase Database has separated DDL worker thread resources from DML worker thread resources. You can choose whether to enable resource isolation for DDL worker threads based on your actual business needs.
OceanBase Database uses the tenant-level hidden parameter _enable_ddl_worker_isolation to control the switch for DDL thread resource isolation. The possible values and meanings of this parameter are as follows:
False: Indicates that resource isolation for DDL worker threads is disabled. In this scenario, DDL requests share worker threads with DML requests, which can affect each other.True: Indicates that resource isolation for DDL worker threads is enabled. In this scenario, DDL requests use independent worker threads, and the number of DDL requests that can be executed simultaneously for a tenant is also limited to avoid excessive impact on DML requests.
To enable resource isolation for DDL worker threads, perform the following steps:
Log in to the MySQL-compatible tenant of the cluster as the
rootuser.A sample connection string is as follows:
obclient -h172.30.xxx.xxx -P2883 -uroot@mysqltenant#obdemo -pxxxx -ACheck the value of the current parameter
_enable_ddl_worker_isolation.obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_PARAMETERS WHERE NAME LIKE '%_enable_ddl_worker_isolation%';A sample query result is as follows:
+----------------+----------+-------+--------+-----------+------------------------------+-----------+-------+------------------------------------------+--------------+-------------------+---------------+-----------+ | SVR_IP | SVR_PORT | ZONE | SCOPE | TENANT_ID | NAME | DATA_TYPE | VALUE | INFO | SECTION | EDIT_LEVEL | DEFAULT_VALUE | ISDEFAULT | +----------------+----------+-------+--------+-----------+------------------------------+-----------+-------+------------------------------------------+--------------+-------------------+---------------+-----------+ | 172.xx.xxx.xxx | 2882 | zone1 | TENANT | 1002 | _enable_ddl_worker_isolation | NULL | True | a switch controling ddl thread isolation | ROOT_SERVICE | DYNAMIC_EFFECTIVE | False | NO | +----------------+----------+-------+--------+-----------+------------------------------+-----------+-------+------------------------------------------+--------------+-------------------+---------------+-----------+ 1 row in setModify the value of the parameter
_enable_ddl_worker_isolation.obclient [oceanbase]> ALTER SYSTEM SET _enable_ddl_worker_isolation = True;
