System logs (SysLogs) are logs generated automatically by OceanBase Database during runtime. They are used for monitoring, alerting, and troubleshooting. System logs complement monitoring metrics, and together they provide comprehensive observability for OceanBase Database.
Although system logs and transaction logs are both sometimes referred to simply as "logs," they serve entirely different purposes:
System logs (SysLogs): Logs generated by OceanBase Database processes during runtime. They are used for monitoring, alerting, and troubleshooting, and are part of the observability system.
Transaction logs (commit logs, or clogs): Write-ahead logs (WALs) that OceanBase Database persists before the corresponding data. They are used to provide transactional guarantees and are part of the data consistency system.
Log modules
Program logs are organized into modules, with two levels: parent modules and child modules.
Parent modules in OceanBase Database include
CLIENT,CLOG,COMMON,ELECT,LIB,PROXY,RPC,RS,SERVER,SHARE,SQL,STORAGE, andTLOG.A child module is represented in the format
parent_module.child_module. For example,SQL.PARSERrefers to thePARSERchild module of theSQLmodule, whileSQL.*refers to all child modules of theSQLmodule.
Log files
The log files belonging to the log modules of OceanBase Database are divided into five types: observer.log, election.log, rootservice.log, trace.log, and alert.log. By default, logs above INFO level are printed.
Except for the trace.log and alert.log files, each type of log file automatically generates a WARNING log file with the .wf suffix (observer.log.wf, election.log.wf, rootservice.log.wf), which prints only logs above WARN level. Whether to generate WARNING log files is controlled by the cluster parameter enable_syslog_wf.
Log Name |
Log Path |
|---|---|
Startup and runtime logs (observer.log, observer.log.wf) |
In the $work_dir/log directory of the OBServer server. |
Election logs (election.log, election.log.wf) |
In the $work_dir/log directory of the OBServer server. |
Root Service logs (rootservice.log, rootservice.log.wf) |
In the $work_dir/log directory of the OBServer server. |
| End-to-end tracing log (trace.log) | In the $work_dir/log directory of the OBServer server. |
| Warning log (alert.log) | In the $work_dir/log/alert directory of the OBServer server. |
The size of a single log file in OceanBase Database does not exceed 256 MB. When a log file reaches 256 MB, log rotation occurs. The original log file is appended with a timestamp suffix (in the format yyyyMMddHHmmss), which is the generation time of the last log entry in the file, and a new log file is created. During log rotation, the .wf log files are rotated even if they have not reached 256 MB. This means the xxx.log.wf files and xxx.log files always correspond one-to-one, and the .wf files are generally much smaller than 256 MB.
A typical log file directory is as follows:
log
├── election.log
├── election.log.wf
├── observer.log
├── observer.log.20220427154619
├── observer.log.wf
├── observer.log.wf.20220427154619
├── rootservice.log
├── rootservice.log.20220427165438
├── rootservice.log.wf
├── rootservice.log.wf.20220427165438
└── log
├── alert.log
└── alert.log.20220304102928236
OceanBase Database natively supports log archiving (controlled by the cluster parameter enable_syslog_recycle). When the total number of log files reaches max_syslog_file_count, the oldest log file is deleted during log rotation. The maximum space usage for log files can be estimated as follows: max_syslog_file_count * 256M * 3 * (enable_syslog_wf ? 2 : 1).
The native log archiving feature of OceanBase Database archives logs based on the number of files. When OceanBase Database is deployed together with an external management and control system (such as OCP), the log archiving feature provided by the external system, which is based on disk usage, is typically used.
Log format
A typical log line and its format are as follows.
A general log line:
[2025-12-14 14:00:49.908850] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:370) [119938][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001})
Decomposing the log reveals its format:
- time:
[2025-12-14 14:00:49.908850] - log_level:
INFO - module:
[STORAGE.TRANS] - function:
check_all_readonly_tx_clean_up - file_name:line_number:
(ob_ls_tx_service.cpp:370) - thread_id:
[119938] - thread_name:
[T1004_TxLoopWor] - coroutine_id:
[T1004] - trace_id:
[Y0-0000000000000000-0-0] - log_used_time:
[lt=13] - info:
wait_all_readonly_tx_cleaned_up cleaned up success - parameter:
(ls_id={id:1001})
The log format consists of two parts:
- header:
[2025-12-14 14:00:49.908850] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:370) [119938][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=13] - message:
wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001})
As shown in the above format, a log consists of a header and a message body:
Header section:
- time: The time when this log was printed.
- log_level: The level of this log. Currently, the following levels are supported (from high to low): ERROR, USER_ERR, WARN, INFO, TRACE, DEBUG.
- module: The name of the module that prints the log. It consists of a parent module and child modules. OceanBase Database supports specifying log levels by module.
- function: The name of the function that prints the log.
- file_name:line_number: The name of the source code file where the log is printed and the line number.
- thread_id: The thread ID that prints the log. In version 4.x, the thread ID, thread name, and coroutine ID fields are arranged adjacent to each other.
- thread_name: The name of the thread where the log is printed. For example,
T1004_TxLoopWorindicates theTxLoopWorkerthread under tenantT1004. - coroutine_id: The ID of the coroutine that prints the log.
- trace_id: The trace ID used to track a task, which is unique at the task level. This trace ID is passed between OBServer nodes via RPCs, allowing all log data for a task to be retrieved based on the trace ID. This corresponds to the
TRACE_IDfield in theGV$OB_SQL_AUDITview. - log_used_time: The processing time of the previous log (in asynchronous logging scenarios, this includes the time taken to write the log to a file).
Message body: The message body generally consists of information (info) and parameters (parameter).
- info: The specific content of the log.
- parameter: A key-value (KV) list in the
name=valueformat. If the value is a primitive type, it is displayed directly. If it is a complex type, it is displayed as text similar to JSON format (to improve readability, the double quotes around keys in JSON format are removed).