A thread that processes SQL and transaction requests is referred to as a worker thread. Each tenant has its own worker threads.
Multi-tenant thread pool
Overview
A multi-tenant thread pool is shared by multiple tenants. Each tenant requests worker threads from this thread pool. Threads in the pool are initialized and terminated along with tenants. When tenants are initialized, they request a number of threads. The thread pool can be expanded when the tenants are running.
The initial size of the thread pool is related to the number of CPU cores for the OBServer and the number of threads reserved by the SYS tenants and the virtual tenants. Some threads can be configured by using the corresponding parameters. The maximum number of threads in an expanded thread pool is limited by both the maximum number of threads for the observer process and that for tenants. To set the maximum number of threads, specify the corresponding parameters.
Thread pool-related parameters
_ob_max_thread_num
The maximum number of threads for the observer process.
Value range: [4096, 10000]. Default value: 4096. A restart is required for the modification to take effect.
system_cpu_quota
The number of virtual CPU cores of the SYS tenant. This parameter is not directly related to the physical CPU. It only affects the initial and maximum size of the multi-tenant thread pool.
Value range: [0, 16]. Default value: 10. A restart is required for the modification to take effect.
server_cpu_quota_min
The minimum number of virtual CPU cores of the SYS500 tenant. This parameter is not directly related to the physical CPU. It only affects the initial size of the multi-tenant thread pool.
Value range: [0, 16]. Default value: 2.5. A restart is required for the modification to take effect.
server_cpu_quota_max
The maximum number of virtual CPU cores of the SYS500 tenant. This parameter is not directly related to the physical CPU. It only affects the maximum size of the multi-tenant thread pool.
Value range: [0, 16]. Default value: 5. A restart is required for the modification to take effect.
election_cpu_quota
The number of virtual CPU cores of the election tenant. This parameter is not directly related to the physical CPU. It only affects the initial and maximum size of the multi-tenant thread pool.
Value range: [0, 16]. Default value: 3. A restart is required for the modification to take effect.
location_cache_cpu_quota
The number of virtual CPU cores of the location cache tenant. This parameter is not directly related to the physical CPU. It only affects the initial and maximum size of the multi-tenant thread pool.
Value range: [0, 16]. Default value: 5. A restart is required for the modification to take effect.
Tenant threads
Types
A tenant has seven dedicated threads that process nested requests and several general threads that process general requests. The status of a tenant thread can be active or suspended. This section describes only the general threads, because you do not need to manage the threads that are dedicated to nested requests.
Active threads
In contrast to suspended threads, active threads can normally process requests. The number of active threads of a tenant limits the CPU utilization of the tenant. When a tenant is running, it keeps a constant number of active threads. The number of active threads of a tenant is determined both by the following parameters and the unit specifications. Number of active threads = unit_min_cpu × cpu_quota_concurrency.
Thread suspension for large queries
The SQL queries that a user sends to an OBServer are divided into small and large queries. Small queries involve the access and operation of a small amount of data and are executed fast. Large queries require the access or write of a large amount of data and take a long time to execute.
The large queries are processed on the assumption of user efficiency. From the perspective of Quality of Service (QoS) that affects the number of users and latency, it is reasonable to infer that the impact of a delay in 1,000 small queries is more serious than that of a delay in one large query. Therefore, instead of spending 1s to process a large query, it is better to spend the same amount of CPU time to process 1,000 small queries.
A request is processed as a large query if the time required to process it exceeds the large query threshold. To modify the threshold, you can specify the corresponding parameter.
Some of the tenant threads that process large queries have the permission to continue the execution. Other threads must be suspended. You can specify the percentage by using the corresponding parameter.
The maximum number of threads
Threads that process large queries can be suspended. Therefore, a tenant needs to dynamically request threads from the multi-tenant thread pool to keep the number of active threads constant. The maximum number of threads limits the memory usage of a tenant. The total number of threads of a tenant is determined both by the following parameters and the unit specifications. Maximum number of threads = unit_max_cpu × worker_per_cpu_quota.
Related parameters
cpu_quota_concurrency
The ratio between the number of active threads of a tenant and the unit specifications of the tenant.
Value range: [1, 10]. Default value: 4. The parameter takes effect immediately after it is modified.
worker_per_cpu_quota
The ratio between the maximum number of threads of a tenant and the unit specifications of the tenant.
Value range: [2, 20]. Default value: 10. The parameter takes effect immediately after it is modified.
large_query_worker_percentage
The percentage of threads that are authorized to continue the execution of large queries.
Value range: [0, 100]. Default value: 30. The parameter takes effect immediately after it is modified.
large_query_threshold
The threshold of processing time for a request to be processed as a large query.
Value range: [1ms, +∞). Default value: 5s. If you specify a value without a unit, the value is treated as seconds. The parameter takes effect immediately after it is modified.
Log diagnosis
You can run the grep 'dump tenant info' observer.log command to obtain the information of worker threads of a tenant and requests in queue, as shown in the following example:
grep 'dump tenant info.*tenant={id:1002' log/observer.log.*
[2021-05-10 16:56:22.564978] INFO [SERVER.OMT] ob_multi_tenant.cpp:803
[48820][2116][Y0-0000000000000000] [lt=5] dump tenant info
(tenant={id:1002, compat_mode:1, unit_min_cpu:"1.000000000000000000e+01",
unit_max_cpu:"1.500000000000000000e+01", slice:"0.000000000000000000e+00",
slice_remain:"0.000000000000000000e+00", token_cnt:30, ass_token_cnt:30,
lq_tokens:3, used_lq_tokens:3, stopped:false, idle_us:4945506,
recv_hp_rpc_cnt:2420622, recv_np_rpc_cnt:7523808, recv_lp_rpc_cnt:0,
recv_mysql_cnt:4561007, recv_task_cnt:337865, recv_large_req_cnt:1272,
tt_large_quries:3648648, actives:35, workers:35, nesting workers:7,
lq waiting workers:5, req_queue:total_size=48183 queue[0]=47888 queue[1]=0 queue[2]=242 queue[3]=5 queue[4]=48 queue[5]=0 ,
large queued:12, multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 ,
recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=165652 cnt[6]=10 cnt[7]=0 })
The following parameters provide useful information on worker threads and are helpful in the diagnosis:
unit_min_cpu and unit_max_cpu: the unit specifications of the tenant.
token_cnt: the target number of active threads of the tenant.
ass_token_cnt: the actual number of active threads of the tenant.
lq_tokens: the maximum number of threads of the tenant that have the permission to continue the execution of large queries.
used_lq_tokens: the number of threads that the tenant has used to execute large queries.
lq waiting workers: the number of threads of the tenant that process large queries and are suspended.