You can search logs by using the keywords "dump tenant info" to view tenant information, such as resource specifications, threads, queues, and request statistics. The logs are updated for each tenant every 10s. You can check these logs to troubleshoot problems such as request execution hanging and request timeouts. You can also check these logs to determine whether changes to tenant specifications take effect. Examples:
grep 'dump tenant info.*id:1002' log/observer.log.*
[2021-05-10 16:56:22.564978] INFO [SERVER.OMT] ob_multi_tenant.cpp:803 [48820][2116][Y0-0000000000000000] [lt=5] dump tenant info(tenant={id:1002, compat_mode:1, unit_min_cpu:"1.000000000000000000e+01", unit_max_cpu:"1.500000000000000000e+01", slice:"0.000000000000000000e+00", slice_remain:"0.000000000000000000e+00", token_cnt:30, ass_token_cnt:30, lq_tokens:3, used_lq_tokens:3, stopped:false, idle_us:4945506, recv_hp_rpc_cnt:2420622, recv_np_rpc_cnt:7523808, recv_lp_rpc_cnt:0, recv_mysql_cnt:4561007, recv_task_cnt:337865, recv_large_req_cnt:1272, tt_large_quries:3648648, actives:35, workers:35, nesting workers:7, lq waiting workers:5, req_queue:total_size=48183 queue[0]=47888 queue[1]=0 queue[2]=242 queue[3]=5 queue[4]=48 queue[5]=0 , large queued:12, multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=165652 cnt[6]=10 cnt[7]=0 })
Field description:
| Attribute | Description |
|---|---|
| id | The ID of the tenant. |
| unit_min_cpu | The minimum number of CPU cores to be provided. |
| unit_max_cpu | The maximum number of CPU cores available. |
| slice | This parameter has no practical significance. |
| slice_remain | This parameter has no practical significance. |
| token_cnt | The number of tokens allocated by the scheduler. Each token is converted into a worker thread. |
| ass_token_cnt | The number of tokens confirmed by the tenant. You can confirm the number of tokens based on the token_cnt parameter. Typically, the values of the two parameters are the same. |
| lq_tokens | The number of tokens for large requests. The value of this field is equal to the value of the token_cnt field multiplied by the proportion of large requests. |
| used_lq_tokens | The number of worker threads that hold large query tokens. |
| stopped | Indicates whether resource units of the tenant are being deleted. |
| idle_us | The total idle time of the worker threads in one round (10s). The idle time includes only the waiting time in queues. |
| recv_hp/np/lp_rpc_cnt | The cumulative number of RPC requests received by the tenant at different levels, including hp(High), np(Normal), and lp(Low). |
| recv_mysql_cnt | The cumulative number of MySQL requests received by the tenant. |
| recv_task_cnt | The cumulative number of internal tasks received by the tenant. |
| recv_large_req_cnt | The cumulative number of large requests predicted by the tenant. The value of this parameter is only incremented but not cleared. The value is incremented upon retries. |
| tt_large_queries | The cumulative number of large requests processed by the tenant. The value of this parameter is only incremented but not cleared. The value is incremented upon checks. |
| actives | The number of active worker threads. |
| workers | The number of worker threads held by the tenant, which equals the size of the workers_ list. |
| nesting workers | The number of dedicated threads held by the tenant for nested requests. A total of seven threads correspond to seven nesting levels. |
| lq waiting workers | The number of worker threads waiting for scheduling. |
| req_queue | The worker queues with different priorities. A smaller value indicates a higher priority. This field indicates the number of queued requests at each priority. |
| large queued | The number of predicted large requests. |
| multi_level_queue | The worker queues to accommodate nested requests. The values 1 to 7 correspond to the seven nesting levels. Queue [0] is invalid. This field indicates the number of queued requests at each level. |
req_queue helps you troubleshoot SQL execution hanging issues, SQL execution timeouts, and other issues. If a queue is stuck for a long time, a deadlock probably occurs.