In end-to-end tracing, different requests have corresponding request links. A complete request link (trace) consists of several request procedures (spans). In OceanBase Database, each internal processing procedure is defined as a span. A request link can be understood as a tree structure that contains parent span nodes and child span nodes. This information is printed to trace logs and parent-child nodes are associated by parent_id and id. In trace logs, each request procedure includes the actual executed SQL information, which is named tag. Different tags record the details of the current operation. Spans and tags are recorded in trace logs to help O&M engineers understand the internal processing logic of the database and locate issues. The entire access link is divided into three parts for description.
Key concepts
- Trace: In the end-to-end tracing process, a Trace can be understood as a transaction in OceanBase Database.
- Span: Belongs to a specific Trace. A Trace can have multiple spans. A span can be a statement, function, or anonymous block.
- Tag: A key-value pair that belongs to a specific span. A span can have multiple tags.
- Log: A key-value pair with timestamp that belongs to a specific span. A span can have multiple logs.
Client
Applications connect to OceanBase Database by using the OBClient or JDBC driver. Client-related spans are obclient and JDBC.
obclient: access through the OBClient.JDBC: connect through the JDBC driver.
Client-related tags are command_name and client_ip. The tag information of OBClient and JDBC is consistent.
command_namemarks requests of different types, such as text, prepare, and execute.client_ipmarks the IP address and port of the obclient that sends the request.
ODP
This phase is divided into three levels of spans. The level here corresponds to the level variable in the Trace-related functions of DBMS_MONITOR.
Level 1
Two types of spans are defined: ob_proxy and ob_proxy_server_process_req.
ob_proxyindicates the total time consumed by ODP to process an SQL request, that is, the entire link from database request start to data processing completion and result return, excluding the frontend driver layer of OceanBase Database.ob_proxy_server_process_reqindicates the time consumed by ODP to process an SQL request and the round-trip network overhead.
Level 2
Two types of spans are defined: ob_proxy_server_response_read and ob_proxy_cluster_resource_create.
ob_proxy_server_response_readindicates the time consumed in reading the response.ob_proxy_cluster_resource_createindicates the time consumed by ODP to prepare cluster resources in the request forwarding phase.
Level 3
Four types of spans are defined: ob_proxy_partition_location_lookup, ob_proxy_do_observer_open, ob_proxy_client_response_write, and ob_proxy_server_request_write.
ob_proxy_partition_location_lookupindicates the time consumed by ODP to obtain the partition location for routing in the request forwarding phase.ob_proxy_do_observer_openindicates the time consumed by ODP to select an OBServer, connect, and other procedures in the request forwarding phase.ob_proxy_client_response_writeindicates the time consumed by ODP to forward the response to the client after receiving the response from OBServer.ob_proxy_server_request_writeindicates the time consumed by ODP to forward the client request to OBServer.
OBServer
Span
OBServer classifies the received requests into text SQL, preprocess statement, and stored procedure. The three types also contain transactions, internal SQL, and access stored in the database.
The following figure shows the access links for different request types and the parent-child relationship between spans. Example: the parent span of mpquery_single_stmt is com_query_entry; the child span of com_query_entry is mpquery_single_stmt.
The related spans are described as follows:
- com_query_entry: the query procedure.
- mpquery_single_stmt: the access path of a single statement.
- sql_compile: compile SQL.
- pc_get_plan: obtain the execution plan.
- hard_parse: hard parsing.
- parse: soft parsing.
- resolve: parse the semantics of the syntax tree and generate a statement.
- rewrite: rewrite SQL.
- optimize: perform cost-based optimization and generate execution plan logs.
- code_generate: generate a physical execution plan based on the execution plan logs.
- pc_add_plan: add the generated execution plan to the plan cache.
- sql_execute: execute the physical execution plan.
- open: open the execution plan.
- response_result: the procedure and result of the execution plan.
- px_schedule: schedule tasks by PX.
- px_task: execute PX subtasks.
- close: close the execution plan.
- cmd_execute: execute the command.
- cmd_open: enable the cmd plan.
- ps_prepare: preprocessing of preprocess statement.
- ps_execute: execution of preprocess statement.
- ps_close: close the preprocess statement.
- pl_entry: stored procedure processing.
- pl_compile: compile the stored procedure object.
- pc_get_pl_object: obtain the stored procedure object from the plan cache.
- pc_add_pl_object: store the stored procedure object in the plan cache.
- pl_execute: execute the stored procedure.
- pl_spi_query: execute SPI statements in the stored procedure.
- pl_spi_prepare: preprocessing phase of stored procedure.
- pl_spi_execute: execute SPI statements in the stored procedure.
- inner_prepare: preprocessing phase of internal SQL.
- inner_execute: execution phase of internal SQL.
- inner_execute_read: read internal SQL.
- inner_execute_write: write internal SQL.
- inner_commit: commit internal SQL transaction.
- inner_rollback: roll back internal SQL transaction.
Tags
To enrich the information of some spans, the following tags are defined:
- com_query_entry
- log_trace_id: the trace ID of the current request in logs.
- err_code: the error code of the current request.
- sql_compile
- sess_id: the session ID.
- sql_text: the SQL text.
- sql_id: the SQL ID.
- hit_plan: indicates that the execution plan hits the plan cache.
- px_task
- task_id: the logical ID of the parallel task.
- dfo_id: the ID of the data flow operator.
- sqc_id: the ID of the subquery coordinator.
- qc_id: the ID of the query coordinator.
- group_id: the ID of the resource group.
- px_schedule
- dfo_id: the ID of the data flow operator.
- used_worker_cnt: the number of PX worker threads in use.
- qc_id: the ID of the query coordinator.
- ps_close
- ps_id: the ID of the preprocess statement.