Overview

Syslogs are the application logs of OceanBase Database and are automatically recorded for monitoring, alerting, and diagnostics when OceanBase Database is running. Syslogs are a great supplement to metrics. The two features ensure the observability of OceanBase Database.

Syslogs and commit logs (clogs) are sometimes referred to as "logs" for short. However, they are different.

• Syslogs are recorded for monitoring, alerting, and diagnostics when the observer process is running. Syslogs enhance database observability.

• Clogs are write-ahead logs (WALs) generated before data persistence in OceanBase Database. Clogs ensure transactional performance and data consistency.

Log modules

Application logs involve two levels of modules: parent modules and submodules.

• OceanBase Database provides the following parent modules: CLIENT, CLOG, COMMON, ELECT, LIB, PROXY, RPC, RS, SERVER, SHARE, SQL, STORAGE, and TLOG.

• If a parent module contains a submodule, the submodule is expressed in the Parent module.Submodule format. For example, in the expression of SQL.PARSER, SQL indicates the SQL module, and SQL.PARSER indicates the PARSER submodule under the SQL module. SQL.* indicates all submodules under the SQL module.

Log files

OceanBase Database has three types of log files: observer.log, election.log, and rootservice.log. By default, these log files record logs at or above the INFO level.

A WARNING log file with the .wf file name extension is automatically generated for each type of log file: observer.log.wf, election.log.wf, and rootservice.log.wf. These WARNING log files record logs at or above the WARN level. The cluster parameter enable_syslog_wf controls whether to generate WARNING log files.

In OceanBase Database, a single log file cannot exceed 256 MB in size. When the size of a log file reaches 256 MB, OceanBase Database performs log rotation. During log rotation, OceanBase Database appends the generation timestamp of the last log entry in the log file to the file name in the yyyyMMddHHmmss format, and then creates another log file with the same file name. In a log rotation, the .wf log file is also rotated even if it does not reach 256 MB. In other words, a xxx.log file and the corresponding xxx.log.wf file are rotated as a pair. In most cases, .wf files are much smaller than 256 MB.

Here is a typical directory of log files:

log
├── election.log
├── election.log.wf
├── observer.log
├── observer.log.20220427154619
├── observer.log.wf
├── observer.log.wf.20220427154619
├── rootservice.log
├── rootservice.log.20220427165438
├── rootservice.log.wf
└── rootservice.log.wf.20220427165438

OceanBase Database natively supports log recycling. You can use the cluster parameter enable_syslog_recycle to specify whether to enable syslog recycling. If syslog recycling is enabled, and the number of log files reaches the value of the max_syslog_file_count parameter, the oldest log files are deleted during log rotation. The maximum space for log storage can be evaluated by using this method: max_syslog_file_count × 256M × 3 × (enable_syslog_wf ? 2:1).

The native log recycling feature of OceanBase Database performs log recycling based on the number of log files. If OceanBase Database is deployed with an external management system, such as OceanBase Cloud Platform (OCP), the external log recycling feature based on disk space usage is often used.

Log format

Logs of OceanBase Database are typical one-line logs.

Here is the format of a sample log entry:

[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0]     [Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=4, b_exist=false)

Fields of this log are as follows:

• time: [2022-04-29 16:39:55.186527]
• log_level: WARN
• module: [COMMON]
• function: get_file_id_range
• file_name:line_number: (ob_log_file_group.cpp:127)
• thread_id: [103594]
• coroutine_id: [0]
• trace_id: [Y0-0000000000000000-0-0]
• log_used_time: [lt=17]
• dropped_msg_count: [dc=0]
• info: max file does not exist
• parameter: (max_file_id=4, b_exist=false)

A log is divided into two parts:

• header: [2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0]
• message: max file does not exist(max_file_id=4, b_exist=false)

As shown in the preceding example, a log consists of a header and a message body.

• Fields in the header:
  • time: the time when the log was recorded.
  • log_level: the level of the log. Supported log levels: ERROR, USER_ERR, WARN, INFO, TRACE, and DEBUG.
  • module: the name of the module that records the log. The module name consists of a parent module name and a submodule name. OceanBase Database allows you to specify the log level by module.
  • function: the name of the function that records the log.
  • file_name:line_number: the name of the source code file of the log and the line number.
  • thread_id: the ID of the thread that records the log.
  • coroutine_id: the ID of the coroutine that records the log.
  • trace_id: the unique ID for tracing a task. The trace_id is passed from one OBServer node to another by using remote procedure calls (RPCs), so that all log data of a task can be obtained based on the trace_id. This field corresponds to the TRACE_ID field in the GV$OB_SQL_AUDIT view.
  • log_used_time: the amount of time spent recording the previous log. When synchronous logging is enabled, the value is the total time spent generating the previous log content and writing the log file. When asynchronous logging is enabled, the value is the total time spent generating the previous log content and submitting it to the asynchronous logging queue. You can use the cluster parameter enable_async_syslog to specify whether to enable asynchronous logging.
  • dropped_msg_count: The value is in the dc=xx format, where xx indicates the number of logs discarded between the generation time of the current log and the previous log. If no logs are discarded, dc=0.
• Fields in the message body:
  • info: the specific log content.
  • parameter: a list of key-value pairs in the name=value format. The value of a simple data type is directly displayed. The value of a complex type is displayed as a text string in a JSON-like format. The quotation marks (") that enclose the key is omitted to improve readability.