A thread that processes SQL and transaction requests is referred to as a worker thread. Each tenant has its own worker threads.
Multi-tenant thread pool
Overview
A multi-tenant thread pool is shared by multiple tenants. Each tenant requests worker threads from this thread pool. Threads in the pool are initialized and terminated along with tenants. When tenants are initialized, they request a number of threads. The thread pool can be expanded when the tenants are running.
The initial size of the thread pool is related to the number of CPU cores on the OBServer node and the number of threads reserved by the sys tenant and the virtual tenants. Some threads can be configured by using the corresponding parameters. The maximum number of threads that can be added in a thread pool is limited by both the maximum number of threads on the OBServer node and that for tenants, which can be set by specifying the corresponding parameters.
Thread pool-related parameters
_ob_max_thread_num
The maximum number of threads on an OBServer node.
The value range is [4096,10000], with the default value of 9999. A restart is required for the modification to take effect.
system_cpu_quota
The number of virtual CPU cores of the sys tenant. This parameter is not directly related to the physical CPU. It only affects the initial and maximum size of the multi-tenant thread pool.
The value range is [0,16], with the default value of 10. A restart is required for the modification to take effect.
server_cpu_quota_min
The minimum number of virtual CPU cores of the sys tenant. This parameter is not directly related to the physical CPU. It only affects the initial size of the multi-tenant thread pool.
The value range is [0,16], with the default value of 1. A restart is required for the modification to take effect.
server_cpu_quota_max
The maximum number of virtual CPU cores of the sys tenant. This parameter is not directly related to the physical CPU. It only affects the maximum size of the multi-tenant thread pool.
The value range is [0,16], with the default value of 1. A restart is required for the modification to take effect.
election_cpu_quota
The number of virtual CPU cores of the election tenant. This parameter is not directly related to the physical CPU. It only affects the initial and maximum size of the multi-tenant thread pool.
The value range is [0,16], with the default value of 3. A restart is required for the modification to take effect.
location_cache_cpu_quota
The number of virtual CPU cores of the location cache tenant. This parameter is not directly related to the physical CPU. It only affects the initial and maximum size of the multi-tenant thread pool.
The value range is [0,16], with the default value of 5. A restart is required for the modification to take effect.
Tenant threads
Types
A tenant has seven dedicated threads that process nested requests and several general threads that process general requests. The status of a tenant thread can be active or suspended. This section describes only the general threads, because you do not need to manage the threads that are dedicated to nested requests.
Active threads
In contrast to suspended threads, active threads can normally process requests. The number of active threads of a tenant limits the CPU utilization of the tenant. When a tenant is running, it keeps a constant number of active threads. The number of active threads of a tenant is determined both by the following parameters and the unit specifications. Number of active threads = unit_min_cpu × cpu_quota_concurrency.
Thread suspension for large queries
The SQL queries that a user sends to an OBServer node are divided into general and large queries. A general query accesses and operates on a small amount of data and is executed fast. A large query accesses or writes a large amount of data and takes a long time to execute.
The large queries are processed on the assumption of user efficiency. From the perspective of Quality of Service (QoS) that affects the number of users and latency, it is reasonable to infer that the impact of a delay in 1,000 small queries is more serious than that of a delay in one large query. Therefore, instead of spending 1s to process a large query, it is better to spend the same amount of CPU time to process 1,000 small queries.
A request is processed as a large query if the time required to process it exceeds the large query threshold. To modify the threshold, you can specify the corresponding parameter.
Some of the tenant threads that process large queries have the permission to continue the execution. Other threads must be suspended. You can specify the percentage by using the corresponding parameter.
The maximum number of threads
Threads that process large queries can be suspended. Therefore, a tenant needs to dynamically request threads from the multi-tenant thread pool to keep the number of active threads constant. The maximum number of threads limits the memory usage of a tenant. The total number of threads of a tenant is determined both by the following parameters and the unit specifications. Maximum number of threads = unit_max_cpu × worker_per_cpu_quota.
Related parameters
cpu_quota_concurrency
The ratio between the number of active threads of a tenant and the unit specifications of the tenant.
The value range is [1, 20], with the default value of 4. This parameter takes effect immediately after it is modified.
worker_per_cpu_quota
The ratio between the maximum number of threads of a tenant and the unit specifications of the tenant.
The value range is [2, 20], with the default value of 10. The parameter takes effect immediately after it is modified.
large_query_worker_percentage
The percentage of threads that are authorized to continue the execution of large queries.
The value range is [0, 100], with the default value of 30. The parameter takes effect immediately after it is modified.
large_query_threshold
The threshold of processing time for a request to be processed as a large query.
The value range is [[1ms, +∞]), with the default value of 5s. If you specify a value without a unit, the value is treated as seconds. The parameter takes effect immediately after it is modified.
For more information about how to query and modify parameters, see Set parameters.
Log diagnosis
You can run the grep 'dump tenant info' observer.log command to obtain the information of worker threads of a tenant and requests in queue, as shown in the following example:
grep 'dump tenant info.tenant={id:1002' log/observer.log | sed 's/,/,\n/g'
[2022-07-20 14:55:40.774143] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:1993) [80700][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=621]
dump tenant info(tenant={id:1002, // The tenant ID.
tenant_meta:{unit:{tenant_id:1002, // The tenant metadata, including the unit configuration information, SuperBlcok, and create_status information.
unit_id:1001, //The unit ID.
has_memstore:true, // Indicates whether a MemTable is contained.
unit_status:"NORMAL", // The status of the unit, which can be UNIT_NORMAL, UNIT_MIGRATE_IN, UNIT_MIGRATE_OUT, UNIT_MARK_DELETING, UNIT_WAIT_GC_IN_OBSERVER, UNIT_DELETING_IN_OBSERVER, or UNIT_ERROR_STAT.
config:{unit_config_id:1003, // The ID of the unit configuration.
name:"2c2g", // The name of the unit configuration.
resource:{min_cpu:2, // The minimum number of CPU cores of the unit configuration.
max_cpu:2, // The maximum number of CPU cores of the unit configuration.
memory_size:"1.5GB", // The memory size of the unit configuration.
log_disk_size:"5.4GB", // The log disk size of the unit configuration.
min_iops:20000, // The minimum disk IOPS of the unit configuration.
max_iops:20000, // The maximum disk IOPS of the unit configuration.
iops_weight:2}}, // The IOPS weight of the unit configuration.
mode:1, // The compatibility mode. Valid values: 0 and 1. The value 0 indicates the MySQL mode. The value 1 indicates the Oracle mode.
create_timestamp:1658298418435426, // The time when the unit was created.
is_removed:false}, // Indicates whether the unit is marked for deletion.
super_block:{tenant_id:1002, // The ObTenantSuperBlock information.
replay_start_point:ObLogCursor{file_id=1, // The replay point of the tenant slog.
log_id=440,
offset=343322},
ls_meta_entry:[140](ver=0, // The entry point of the linked table that stores slog checkpoints of the log stream metadata of the tenant.
mode=0,
seq=139),
tablet_meta_entry:[141](ver=0, // The slog checkpoint entry of the tablet metadata of the tenant.
mode=0,
seq=140)},
create_status:1}, // The creation status of the tenant. Valid values: 0: CREATING; 1: CREATE_COMMIT; 2: CREATE_ABORT; 3: DELETING; 4: DELETE_COMMIT.
unit_min_cpu:"2.000000000000000000e+00", // The minimum number of CPU cores guaranteed.
unit_max_cpu:"2.000000000000000000e+00", // The maximum number of CPU cores allowed.
slice:"0.000000000000000000e+00",
slice_remain:"0.000000000000000000e+00",
token_cnt:8, // The number of tokens allocated by the scheduler. Each token is converted into a worker thread.
sug_token_cnt:8,
ass_token_cnt:8, // The number of tokens allocated to the tenant. You can determine the number of tokens based on the token_cnt field. Typically, the values of the two fields are the same.
lq_tokens:2, // The number of tokens for large requests. The value of this field is equal to the value of the token_cnt field multiplied by the proportion of large requests.
used_lq_tokens:0, // The number of worker threads that hold large query tokens.
stopped:false, // Indicates whether resource units of the tenant are being deleted.
idle_us:6076629, // The total idle time of the worker threads in one round (10s). The idle time includes only the wait time in queues.
recv_hp_rpc_cnt:1, // The cumulative number of RPC requests received by the tenant at different levels, including hp(High), np(Normal), and lp(Low).
recv_np_rpc_cnt:5,
recv_lp_rpc_cnt:0,
recv_mysql_cnt:24, // The cumulative number of MySQL requests received by the tenant.
recv_task_cnt:1, // The cumulative number of internal tasks received by the tenant.
recv_large_req_cnt:0, // The cumulative number of large requests predicted by the tenant. The value of this field is only incremented but not cleared. The value is incremented upon a retry.
tt_large_quries:0, // The cumulative number of large requests processed by the tenant. The value of this field is only incremented but not cleared. The value is incremented upon a check.
pop_normal_cnt:1024555,
actives:8, // The number of active worker threads. The value is equal to the number of active worker threads of the tenant plus the number of threads suspended due to large queries, which is often equal to the value of the workers parameter. The difference between the values of the actives and workers parameters is the number of inactive threads of the tenant that are waiting to be collected with a delay.
workers:8, // The number of worker threads held by the tenant, which is equal to the sum of the number of active worker threads of the tenant, the number of threads suspended due to large queries, and the number of inactive threads of the tenant that are waiting to be collected with a delay.
nesting workers:7, // The number of dedicated threads held by the tenant for nested requests. A total of seven threads correspond to seven nesting levels.
lq waiting workers:0, // The number of worker threads intended for processing large queries and waiting to be scheduled.
req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , // The worker queues with different priorities. A smaller value indicates a higher priority.
large queued:0, // The number of large requests predicted by the tenant.
multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 , // The worker queues to accommodate nested requests. The values 1 to 7 correspond to the seven nesting levels. queue[0] is invalid. queque[5] accommodate inner SQL requests.
recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 ,
group_map:group_id = 1, // The threads of the group that corresponds to each group_id in the group_map and their queuing status.
queue_size = 0, // The queuing status of the group queue.
recv_req_cnt=13526, // The cumulative number of requests in the group that are pushed to the queue.
pop_req_cnt=13526, // The cumulative number of requests popped out by threads in the group.
token_cnt = 2, // The number of tokens allocated to the group. Each token is converted into a worker thread.
min_token_cnt = 2,
max_token_cnt = 2,
ass_token_cnt = 2 , // The number of tokens allocated to the group. You can determine the number of tokens based on the token_cnt field. Typically, the values of the two fields are the same.
rpc_stat_info: pcode=0x150a:cnt=1489 pcode=0x150b:cnt=1091}) // The RPC pcodes that have been received most frequently by the tenant in a period of time. The statistical period is 10 seconds, and the top five RPC pcodes are printed.