Logs

An OBServer node generates four types of logs: observer.log, rootservice.log, election.log, and trace.log. These logs are stored in the log directory under the working directory of the OBServer node.

Each log type, such as observer.log, can be further divided into the following types by the file name:

• observer.log

• observer.log.20210901123456

• observer.log.wf

• observer.log.wf.20210901123456

When the size of the observer.log file reaches 256 MB, it is renamed to the second type. The appended string of digits is a timestamp. For more information about the wf logs, see the description of the enable_syslog_wf parameter.

Log format

• Format

  [Time] Log level [Module] File:Row number [Thread ID] [0] [trace_id] [lt] [Number of discarded logs]
  

• Parameters

  Parameter	Description
  Time	A timestamp that indicates the date and time when the log was generated.
  Log level	The importance or level of the log.
  Module	The module or component where the log was generated.
  File:Row number	The source code file path and row number, which indicates the location of the source code based on which the log was generated.
  Thread ID	The unique identifier of the thread that generated the log.
  0	The transaction ID or other information related to the log. [0] indicates that the field value is 0.
  trace_id	The trace ID of the log event.
  lt	A tag, which indicates the execution time of the previous log, in microseconds.
  Number of discarded logs	The number of logs that were discarded or were not recorded due to some reasons before the current log was generated.

• Example

  [2021-09-01 11:31:18.605433] INFO  [STORAGE] ob_pg_sstable_garbage_collector.cpp:170 [38715][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] do one gc free sstable by queue(ret=0, free sstable cnt=0)
  

Log levels

OceanBase Database supports the following seven log levels in ascending order: DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, and ERROR.

The ERROR log level is special. For a log at this level, the stack where the log was generated is printed. The symbol table is required to parse logs at this level.

Module

Logs can be distinguished from one another by their modules.

Thread ID

The ID of the thread that generated a log can be used to track the thread behaviors and is useful for troubleshooting faults such as thread hanging and timeout.

trace_id

This indicates the ID of an internal SQL statement in OceanBase Database. The default value is Y0-0000000000000000-0-0. trace_id can be used to track the execution process of an SQL statement and is an important means of troubleshooting.

Log compaction

You can specify related parameters to configure syslog compression, including the compression algorithm, maximum disk space available for syslogs, and number of uncompressed log files for each log type. For more information, see Log compression and decompression.

Log decompression

You can view a compressed log file in any of the following ways: decompress it and then view its content, directly view it, or use obdiag to search for it. For more information, see Log compression and decompression.

Related parameters

This section describes the cluster-level parameters related to log management. The parameters must be used in the sys tenant. You can modify the parameters by using the following syntax:

obclient> ALTER SYSTEM SET enable_syslog_recycle = False;

• enable_syslog_recycle

  Specifies whether to enable syslog recycling. The default value is False.

  After syslog recycling is enabled, redundant logs are automatically deleted. For more information about the enable_syslog_recycle parameter, see enable_syslog_recycle.

• enable_syslog_wf

  Specifies whether to enable .wf logs. The default value is True.

  After .wf logs are enabled, logs above the WARN level of each type are copied to a .wf log file, such as observer.log.wf.

• enable_async_syslog

  Specifies whether to enable asynchronous writing of logs. The default value is True.

  If you set this parameter to False, logs are written in synchronous mode. This ensures that all logs are written before an OBServer node is shut down. However, the OBServer node performance is significantly reduced. We recommend that you set this parameter to True.

• max_syslog_file_count

  Specifies the maximum number of log files of each type. The default value is 0. This parameter takes effect only when its value is greater than 0 and greater than or equal to the value of syslog_file_uncompressed_count and enable_syslog_recycle is set to True.

• syslog_io_bandwidth_limit

  The maximum bandwidth for log I/O. The default bandwidth is 30 MB.

  Notice

  When the log printing speed exceeds the limit, the following message is printed.

  REACH SYSLOG RATE LIMIT
  

• syslog_level

  Specifies the lowest level of the logs to print. The default value is WDIAG.

• syslog_disk_size

  The maximum disk space available for syslogs. The default value is 0M. The disk space available for syslogs is subject to the value of syslog_disk_size and the actual disk space. The smaller of the two prevails. When the remaining disk space is less than 2 GB, the oldest log files are deleted. If log compression is enabled, log files are compressed when the remaining disk space is less than 4 GB.

• syslog_compress_func

  Specifies the compression algorithm for syslogs. The default value is none, which specifies to disable log compression. Supported compression algorithms are zstd_1.0 and zstd_1.3.8. You can set the parameter to a supported compression algorithm to enable log compression.

  Notice

  Only log files suffixed with .log.{timestamp} are compressed. Those suffixed with wf and those not suffixed with {timestamp} are not compressed. timestamp indicates the time when the log file was generated.

• syslog_file_uncompressed_count

  Specifies the number of uncompressed syslog files. This parameter takes effect only when syslog_compress_func is not set to none. The number of uncompressed syslog files is counted separately for each log type, excluding the log files not suffixed with {timestamp}. The relationships between the syslog_compress_func parameter and the syslog_file_uncompressed_count, syslog_disk_size, and max_syslog_file_count parametrs are as follows:

  • When syslog_compress_func =none is specified:

    1. syslog_file_uncompressed_count is invalid.
    2. syslog_disk_size and max_syslog_file_count control the maximum number and size of log files.

  • When syslog_compress_func !=none is specified:

    1. max_syslog_file_count is invalid.
    2. When the number of log files exceeds the value of syslog_file_uncompressed_count, the redundant log files are compressed. When the total size of log files approaches the value of syslog_disk_size, the oldest log files are deleted.