Syslogs are the application logs of OceanBase Database and are automatically recorded for monitoring, alerting, and diagnostics when OceanBase Database is running. Syslogs are a great supplement to metrics. The two features ensure the observability of OceanBase Database.
Syslogs and commit logs (clogs) are sometimes referred to as "logs" for short. However, they are different.
Syslogs are recorded for monitoring, alerting, and diagnostics when the observer process is running. Syslogs enhance database observability.
Clogs are write-ahead logs (WALs) generated before data persistence in OceanBase Database. Clogs ensure transactional performance and data consistency.
Log modules
Program logs involve two levels of modules: parent modules and submodules.
OceanBase Database provides the following parent modules:
CLIENT,CLOG,COMMON,ELECT,LIB,PROXY,RPC,RS,SERVER,SHARE,SQL,STORAGE, andTLOG.If a parent module contains a submodule, the submodule is expressed in the
Parent module.Submoduleformat. For example, in the expression ofSQL.PARSER,SQLindicates the SQL module, andSQL.PARSERindicates the PARSER submodule under the SQL module.SQL.*indicates all submodules under the SQL module.
Log files
OceanBase Database has three types of log files: observer.log, election.log, and rootservice.log. By default, these log files record logs at or above the INFO level.
A WARNING log file with the .wf file name extension is automatically generated for each type of log file: observer.log.wf, election.log.wf, and rootservice.log.wf. These WARNING log files record logs at or above the WARN level. The cluster parameter enable_syslog_wf controls whether to generate WARNING log files.
Log |
Path |
|---|---|
Startup and operation logs: observer.log and observer.log.wf |
$work_dir/log on the OBServer node |
Election logs: election.log and election.log.wf |
$work_dir/log on the OBServer node |
RootService logs: rootservice.log and rootservice.log.wf |
$work_dir/log on the OBServer node |
In OceanBase Database, a single log file cannot exceed 256 MB in size. When the size of a log file reaches 256 MB, OceanBase Database performs log rotation. During log rotation, OceanBase Database appends the generation timestamp of the last log entry in the log file to the file name in the yyyyMMddHHmmss format, and then creates another log file with the same file name. In a log rotation, the .wf log file is also rotated even if it does not reach 256 MB. In other words, a xxx.log file and the corresponding xxx.log.wf file are rotated as a pair. In most cases, .wf files are much smaller than 256 MB.
The following is a typical directory of log files:
log
├── election.log
├── election.log.wf
├── observer.log
├── observer.log.20220427154619
├── observer.log.wf
├── observer.log.wf.20220427154619
├── rootservice.log
├── rootservice.log.20220427165438
├── rootservice.log.wf
└── rootservice.log.wf.20220427165438
OceanBase Database natively supports log recycling. You can use the cluster parameter enable_syslog_recycle to specify whether to enable syslog recycling. If syslog recycling is enabled, and the number of log files reaches the value of the max_syslog_file_count parameter, the oldest log files are deleted during log rotation. The maximum space for log storage can be evaluated by using this method: max_syslog_file_count × 256M × 3 × (enable_syslog_wf ? 2:1).
The native log recycling feature of OceanBase Database performs log recycling based on the number of log files. If OceanBase Database is deployed with an external management system, such as OceanBase Cloud Platform (OCP), the external log recycling feature based on disk space usage is often used.
Log format
Logs of OceanBase Database are typical one-line logs.
The format of a sample log entry is as follows:
[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=4, b_exist=false)
By analyzing the log entry, we can extract the following components and their corresponding format:
- time:
[2022-04-29 16:39:55.186527] - log_level:
WARN - module:
[COMMON] - function:
get_file_id_range - file_name:line_number:
(ob_log_file_group.cpp:127) - thread_id:
[103594] - coroutine_id:
[0] - trace_id:
[Y0-0000000000000000-0-0] - log_used_time:
[lt=17] - dropped_msg_count:
[dc=0] - info:
max file does not exist - parameter:
(max_file_id=4, b_exist=false)
The log format consists of two parts:
- header:
[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0] - message:
max file does not exist(max_file_id=4, b_exist=false)
As shown in the preceding example, a log consists of a header and a message body.
Fields in the header:
- time: the time when the log was recorded.
- log_level: the level of the log. Supported log levels include ERROR, USER_ERR, WARN, INFO, TRACE, and DEBUG.
- module: the name of the module that records the log. The module name consists of a parent module name and a submodule name. OceanBase Database allows you to specify the log level by module.
- function: the name of the function that records the log.
- file_name:line_number: the name of the source code file of the log and the line number.
- thread_id: the ID of the thread that records the log.
- coroutine_id: the ID of the coroutine that records the log.
- trace_id: the unique ID for tracing a task. The trace_id is passed from one OBServer node to another by using remote procedure calls (RPCs), so that all log data of a task can be obtained based on the trace_id. This field corresponds to the
TRACE_IDfield in theGV$OB_SQL_AUDITview. - log_used_time: the processing duration of the previous log. In asynchronous logging, this duration includes the time consumed to write the log to the file.
- dropped_msg_count: The value is in the
dc=xxformat, wherexxindicates the number of logs discarded between the generation time of the current log and the previous log. If no logs are discarded, dc=0.
Fields in the message body:
- info: the specific log content.
- parameter: a list of key-value pairs in the
name=valueformat. The value of a simple data type is directly displayed. The value of a complex type is displayed as a text string in a JSON-like format. The quotation marks (") that enclose the key is omitted to improve readability.