Syslogs are the application logs of OceanBase Database and are automatically recorded for monitoring, alerting, and diagnostics when OceanBase Database is running. Syslogs are a great supplement to metrics. The two features ensure the observability of OceanBase Database.
Syslogs and commit logs (clogs) are sometimes referred to as "logs" for short. However, they are different.
Syslogs are recorded for monitoring, alerting, and diagnostics when the observer process is running. Syslogs enhance database observability.
Clogs are write-ahead logs (WALs) generated before data persistence in OceanBase Database. Clogs ensure transactional performance and data consistency.
Log modules
Program logs involve two levels of modules: parent modules and submodules.
OceanBase Database provides the following parent modules:
CLIENT,CLOG,COMMON,ELECT,LIB,PROXY,RPC,RS,SERVER,SHARE,SQL,STORAGE, andTLOG.If a parent module contains submodules, it is represented as
parent module.submodule. For example,SQL.PARSERrepresents the PARSER submodule of the SQL module, whileSQL.*represents all submodules of the SQL module.
Log files
OceanBase Database has three types of log files: observer.log, election.log, rootservice.log, trace.log, and alert.log. By default, these log files record logs at or above the INFO level.
Except for the trace.log and alert.log files, each type of log file automatically generates a WARNING log file with a .wf suffix (observer.log.wf, election.log.wf, rootservice.log.wf), which only records logs of WARN level and above. The generation of WARNING log files is controlled by the cluster configuration item enable_syslog_wf.
| Log | Path |
|---|---|
Startup and runtime logs (observer.log, observer.log.wf) |
Located in the $work_dir/log directory of the OBServer server. |
Election logs (election.log, election.log.wf) |
Located in the $work_dir/log directory of the OBServer server. |
Root Service logs (rootservice.log, rootservice.log.wf) |
Located in the $work_dir/log directory of the OBServer server. |
End-to-end tracing logs (trace.log) |
Located in the $work_dir/log directory of the OBServer server. |
Alert logs (alert.log) |
Located in the $work_dir/log/alert directory of the OBServer server. |
In OceanBase Database, a single log file cannot exceed 256 MB in size. When the size of a log file reaches 256 MB, OceanBase Database performs log rotation. During log rotation, OceanBase Database appends the generation timestamp of the last log entry in the log file to the file name in the yyyyMMddHHmmss format, and then creates another log file with the same file name. In a log rotation, the .wf log file is also rotated even if it does not reach 256 MB. In other words, a xxx.log file and the corresponding xxx.log.wf file are rotated as a pair. In most cases, .wf files are much smaller than 256 MB.
The following is a typical directory of log files:
log
├── election.log
├── election.log.wf
├── observer.log
├── observer.log.20220427154619
├── observer.log.wf
├── observer.log.wf.20220427154619
├── rootservice.log
├── rootservice.log.20220427165438
├── rootservice.log.wf
├── rootservice.log.wf.20220427165438
└── log
├── alert.log
└── alert.log.20220304102928236
OceanBase Database natively supports log archiving (controlled through the cluster parameter enable_syslog_recycle). When the total number of log files reaches max_syslog_file_count, the oldest log files are deleted during log rotation. The maximum space usage of log files can be estimated as follows: max_syslog_file_count * 256M * 3 * (enable_syslog_wf ? 2 : 1).
The native log recycling feature of OceanBase Database performs log recycling based on the number of log files. If OceanBase Database is deployed with an external management system, such as OceanBase Cloud Platform (OCP), the external log recycling feature based on disk space usage is often used.
Log format
Logs of OceanBase Database are typical one-line logs.
The format of a sample log entry is as follows:
[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=4, b_exist=false)
By analyzing the log entry, we can extract the following components and their corresponding format:
- time:
[2022-04-29 16:39:55.186527] - log_level:
WARN - module:
[COMMON] - function:
get_file_id_range - file_name:line_number:
(ob_log_file_group.cpp:127) - thread_id:
[103594] - coroutine_id:
[0] - trace_id:
[Y0-0000000000000000-0-0] - log_used_time:
[lt=17] - dropped_msg_count:
[dc=0] - info:
max file does not exist - parameter:
(max_file_id=4, b_exist=false)
The log format consists of two parts:
- header:
[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0] - message:
max file does not exist(max_file_id=4, b_exist=false)
As shown in the preceding example, a log consists of a header and a message body.
Fields in the header:
- time: the time when the log was recorded.
- log_level: the level of the log. Supported log levels are ERROR, USER_ERR, WARN, INFO, TRACE, and DEBUG.
- module: the name of the module that records the log. The module name consists of a parent module name and a submodule name. OceanBase Database allows you to specify the log level by module.
- function: the name of the function that records the log.
- file_name:line_number: the name of the source code file of the log and the line number.
- thread_id: the ID of the thread that records the log.
- coroutine_id: the ID of the coroutine that records the log.
- trace_id: the unique ID for tracing a task. The trace_id is passed from one OBServer node to another by using remote procedure calls (RPCs), so that all log data of a task can be obtained based on the trace_id. This field corresponds to the
TRACE_IDfield in theGV$OB_SQL_AUDITview. - log_used_time: the processing duration of the previous log. In asynchronous logging, this duration includes the time consumed to write the log to the file.
- dropped_msg_count: The value is in the
dc=xxformat, wherexxindicates the number of logs discarded between the generation time of the current log and the previous log. If no logs are discarded, dc=0.
Fields in the message body:
- info: the specific log content.
- parameter: a list of key-value pairs in the
name=valueformat. The value of a simple data type is directly displayed. The value of a complex type is displayed as a text string in a JSON-like format. The quotation marks (") that enclose the key is omitted to improve readability.