What is end-to-end tracing?
OceanBase Database is a distributed system with complex call chains. When timeout issues occur, it is difficult for O&M engineers to quickly determine whether the issues stems from OceanBase Database components or the network. Their only recourse is to analyze the issues based on experience and OBServer logs. To enhance diagnostic efficiency, this version introduces a new end-to-end tracing mechanism. It traces the relevant information of user SQL requests as they pass through different components and stages in the database call chain, and presents this data to users in a visual format. This feature helps users quickly locate the source of problems and accurately identify internal component issues.
End-to-end tracing paths
End-to-end tracing covers two main data flow paths:
The first path starts with a request from the application, which is then transmitted through a client (such as JDBC or OCI) to OceanBase Database Proxy (ODP), and forwarded to the OBServer nodes, with the final result returned to the application.
The second path involves a request sent directly from the application through the client to the OBServer nodes, with the result returned directly.
The purpose of end-to-end tracing is to locate issues within all components along these two paths. Users can connect to OceanBase Database through the client. When sending a request through the client, the request may be routed through ODP to OBServer nodes, or it may directly reach OBServer nodes. With the end-to-end tracing functionality, O&M engineers can use methods from the PL/SQL DBMS_MONITOR package to control and monitor whether the application has enabled end-to-end tracing, as well as the printing details of the tracing information as needed.
End-to-end tracing logs
The logs for end-to-end tracing, also known as trace logs, are determined by the data access path. If the database is accessed through ODP, the trace information is recorded in both the ODP and OBServer log files. If the access is direct to OBServer nodes, the trace information is only recorded in the OBServer log file. Specifically, the trace log file for ODP is named obproxy_trace.log, and for OBServer it is named trace.log. Each log file is limited to 256 MB. When a log reaches this size limit, the system creates a new log file and archives the old one. The number of archived files is controlled by the parameter max_syslog_file_count, and any excess files are automatically deleted.
By collecting and analyzing all trace logs, O&M engineers can track the execution time and other relevant information of each transaction or SQL query across the entire call chain. This not only provides important clues for accurate issue locating, but also greatly improves operational efficiency.