Collect host logs containing the dump tenant info keyword

Description

You can query logs containing the dump tenant info.tenant keyword from the observer.log file.

Statement

Query INFO logs containing the dump tenant info.tenant={id: keyword from the observer.log file in the observer log directory, which is /home/admin/oceanbase/log by default.

Here is a sample command, where ob_tenant_id must be replaced with the actual tenant ID:

grep 'dump tenant info.tenant={id:${ob_tenant_id}' ${observer.log.path}/observer.log

Here, the sys500 tenant is used as an example. If the tenant queue is full, the following logs containing the resource specifications, threads, queues, and request statistics of the tenant are returned:

observer.log.20171129181406:[2017-11-29 18:13:54.020089] INFO  [SERVER.OMT] ob_multi_tenant.cpp:533 [116676][Y0-0000000000000000] [lt=42] dump tenant info(tenant={id:500, 			unit_min_cpu:"10.000000000000", unit_max_cpu:"10.000000000000", slice:"0.000000000000", slice_remain:"0.000000000000", token_cnt:40, ass_token_cnt:40, lq_tokens:12, used_lq_tokens:0, stopped:false, idle_us:0, recv_hp_rpc_cnt:380082895, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, actives:40, workers:40, lq waiting workers:0, req_queue:total_size=65537 queue[0]=65537 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , large queued:0})

You can identify backlog of the tenant queue from this type of logs. The fields in the logs are described as follows:

• id: the ID of the tenant.
• unit_min_cpu: the minimum number of CPU cores for the tenant.
• unit_max_cpu: the maximum number of CPU cores for the tenant.
• slice: N/A.
• slice_remain: N/A.
• token_cnt: the number of tokens allocated by the scheduler. Each token is converted into a worker thread.
• ass_token_cnt: the number of tokens confirmed by the tenant. You can determine the number of tokens based on the token_cnt field. Typically, the values of the two fields are the same.
• lq_tokens: the number of tokens for large requests. The value of this field is equal to the value of the token_cnt field multiplied by the proportion of large requests.
• used_lq_tokens: the number of worker threads that hold large query tokens.
• stopped: indicates whether resource units of the tenant are being deleted.
• idle_us: the total idle time of the worker threads in one round (10s). The idle time includes only the wait time in queues.
• recv_hp/np/lp_rpc_cnt: the cumulative number of RPC requests received by the tenant at different levels, including hp (high), np (normal), and lp (low).
• recv_mysql_cnt: the cumulative number of MySQL requests received by the tenant.
• recv_task_cnt: the cumulative number of internal tasks received by the tenant.
• recv_large_req_cnt: the cumulative number of large requests predicted by the tenant. The value of this field is only incremented but not cleared. The value is incremented upon retries.
• tt_large_quries: the cumulative number of large requests processed by the tenant. The value of this field is only incremented but not cleared. The value is incremented upon checks.
• actives: the number of active worker threads.
• workers: the number of worker threads held by the tenant, which equals the size of the workers_ list.
• nesting workers: the number of dedicated threads held by the tenant for nested requests. A total of seven threads correspond to seven nesting levels.
• lq waiting workers: the number of worker threads waiting to be scheduled.
• req_queue: the worker queues with different priorities. A smaller value indicates a higher priority. This field indicates the number of queued requests at each priority.
• large queued: the number of large requests predicted by the tenant.
• multi_level_queue: the worker queues to accommodate nested requests. The values 1 to 7 correspond to the seven nesting levels. queue[0] is invalid. This field indicates the number of queued requests at each level.