What is end-to-end tracing?
OceanBase Database is a distributed system with complex call chains. When timeout issues occur, it is difficult for O&M engineers to quickly determine whether the issues stems from OceanBase Database components or the network. Their only recourse is to analyze the issues based on experience and OBServer logs. To enhance diagnostic efficiency, this version introduces a new end-to-end diagnostic mechanism. It is capable of tracing the pertinent information of user SQL requests as they traverse different components and stages within the database call chain, and presenting this data to the user in a visual format. This feature assists users in rapidly pinpointing the source of the problem and accurately identifying internal component issues.
End-to-end tracing paths
End-to-end diagnosis covers two main data flow paths:
The first path starts with a request from the application, which is then transmitted through a client (such as JDBC or OCI) to OceanBase Database Proxy (ODP), and forwarded to the OBServer nodes, with the final result returned to the application.
The second path involves a request sent directly from the application through the client to the OBServer nodes, with the result returned directly.
The purpose of end-to-end diagnosis is to locate issues within all components along these two paths. Users can connect to OceanBase Database through the client. When sending a request through the client, the request may be routed through ODP to OBServer nodes, or it may directly reach OBServer nodes. With the end-to-end diagnosis functionality, O&M engineers can use methods from the PL/SQL DBMS_MONITOR package to control and monitor whether the application has enabled end-to-end tracing, as well as the printing details of the tracing information as needed.
End-to-end diagnosis logs
The logs for end-to-end diagnosis, also known as trace logs, are determined by the data access path. If the database is accessed through ODP, the trace information is recorded in both the ODP and OBServer log files. If the access is direct to OBServer nodes, the trace information is only recorded in the OBServer log file. Specifically, the trace log file for ODP is named obproxy_trace.log, and for OBServer it is named trace.log. Each log file is limited to a size of 256 MB, and when the log reaches this size limit, the system creates a new log file and archives the old one. The number of archived files is controlled by specific parameters, and any excess files are automatically deleted.
By collecting and analyzing all trace logs, O&M engineers can track the execution time and other relevant information of each transaction or SQL query across the entire call chain. This not only provides important clues for accurate issue locating, but also greatly improves operational efficiency.