Logs

An OBServer node generates four types of logs: observer.log, rootservice.log, election.log, and trace.log. These logs are stored in the log directory under the installation directory of the OBServer node.

Each log type, such as observer.log, can be further divided into the following types by the file name:

• observer.log

• observer.log.20210901123456

• observer.log.wf

• observer.log.wf.20210901123456

When the size of the observer.log file reaches 256 MB, it is renamed to the second type. The appended string of digits is a timestamp. For more information about the wf logs, see the description of the enable_syslog_wf parameter.

Log format

• Format

  [Time] Log level [Module] File:Row number [Thread ID] [0] [trace_id] [lt] [Number of discarded logs]
  

• Parameters

  Parameter	Description
  Time	A timestamp that indicates the date and time when the log was generated.
  Log level	The importance or level of the log.
  Module	The module or component where the log was generated.
  File:Row number	The source code file path and row number, which indicates the location of the source code based on which the log was generated.
  Thread ID	The unique identifier of the thread that generated the log.
  0	The transaction ID or other information related to the log. [0] indicates that the field value is 0.
  trace_id	The trace ID of the log event.
  lt	A tag, which indicates the execution time of the previous log, in microseconds.
  Number of discarded logs	The number of logs that were discarded or were not recorded due to some reasons before the current log was generated.

• Example

  [2021-09-01 11:31:18.605433] INFO  [STORAGE] ob_pg_sstable_garbage_collector.cpp:170 [38715][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] do one gc free sstable by queue(ret=0, free sstable cnt=0)
  

Crash logs

Unlike other logs, a crash log is generated only when the system or application crashes. It records the details of a crash.

• Crash log format

  CRASH ERROR!!! IP=xxx, RBP=xxx, sig=xxx, sig_code=xxx, sig_addr=xxx, RLIMIT_CORE=xxx, timestamp=xxx, tid=xxx, tname=xxx, trace_id=xxx, extra_info=xxx, lbt=xxx, SQL_ID=xxx, SQL_STRING==xxx
  

• Crash log parameters

  Parameter	Description
  IP	The address of the instruction being executed.
  RBP	The base pointer of the current stack frame, which is used to locate the stack.
  sig	The signal that triggered the crash.
  sig_code	The details of the trigger signal, including its type and cause.
  sig_addr	The memory address of the trigger signal, which indicates the location where the crash occurred.
  RLIMIT_CORE	The maximum size of the core dump generated by the program.
  timestamp	The timestamp that records when the crash occurred.
  tid	The ID of the thread that triggered the crash.
  tname	The name of the thread that triggered the crash.
  trace_id	The trace ID of the crash event, such as YXXXXXXXX-000XXXXXXXXXX-0-0.
  extra_info	The additional information, such as other debugging or context information of the crash.
  lbt	The stack information of the thread that triggered the crash, which records the function call stack generated when the crash occurred.
  SQL_ID	The ID of the SQL statement that was being executed by the thread that triggered the crash.
  SQL_STRING	The string representation of the SQL statement that was being executed by the thread that triggered the crash.

• Example

  CRASH ERROR!!! IP=7f4b4e45b277, RBP=7f4ae82cbdd0, sig=6, sig_code=-6, sig_addr=0x1505ee0005978f, RLIMIT_CORE=unlimited, timestamp=1712749024610833, tid=505088, tname=T1004_PX_G0, trace_id=YXXXXXXXX-000XXXXXXXXXX-0-0, extra_info=(), lbt=0x35c17118 0x34dd048b 0x7f4b4e80261f 0x7f4b4e45b277 0x7f4b4e45c967 0xa059fee 0x999ce0c 0x35ac9e92 0x97bbee4 0x979ea08 0x977dcf5 0x1d3afab8 0x1d8dde16 0x1d3aeed8 0x94017c7 0x9388f30 0x17dcc38c 0x17dc5492 0x17dc2ab8 0x9388351 0x1a4bbc2c 0x1a4bc772 0x9388351 0x12172da8 0x9388351 0x1217c0c1 0x12412b67 0x1214fa3a 0x1216d567 0x1a63f595 0x1a63d304 0x1a63934b 0x1a63b20d 0x1a6bd01c 0xf73acd7 0x9b598d6 0xf73b301 0x35bf9d0e 0x35bf87c2 0x7f4b4e7fae24 0x7f4b4e522f1c, SQL_ID=AEAB26B02AF74B951385BC00D988E5AC, SQL_STRING=SELECT * FROM (SELECT IFNULL(NULL, (((CAST((NULL) AND (CAST(((-9223372036854775808,7.404792285453929,1729790173) IN ((-922337203
  

Log levels

OceanBase Database supports the following seven log levels in ascending order: DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, and ERROR.

The ERROR log level is special. For a log at this level, the stack where the log was generated is printed. The symbol table is required to parse logs at this level.

Module

Logs can be distinguished from one another by their modules.

Thread ID

The ID of the thread that generated a log can be used to track the thread behaviors and is useful for troubleshooting faults such as thread hanging and timeout.

trace_id

This indicates the ID of an internal SQL statement in OceanBase Database. The default value is Y0-0000000000000000-0-0. trace_id can be used to track the execution process of an SQL statement and is an important means of troubleshooting.

Log compression

You can specify related parameters to configure system log compression, including the compression algorithm, maximum disk space available for system logs, and number of uncompressed log files for each log type. For more information, see Log compression and decompression.

Log decompression

You can view a compressed log file in any of the following ways: decompress it and then view its content, directly view it, or use obdiag to search for it. For more information, see Log compression and decompression.

Related parameters

This section describes the cluster-level parameters related to log management. The parameters must be used in the sys tenant. You can modify the parameters by using the following syntax:

obclient> ALTER SYSTEM SET enable_syslog_recycle = False;

• enable_syslog_recycle

  Specifies whether to record the logs generated before the startup of the OBServer node. The default value is False. You can use this parameter with max_syslog_file_count to specify whether to include earlier log files in the recycling logic.

  If you set this parameter to True, redundant logs are automatically deleted.

• enable_syslog_wf

  Specifies whether to enable wf logs. The default value is True.

  If you set this parameter to True, logs of each type at the WARN level or above are copied to a .wf log file, such as observer.log.wf.

• enable_async_syslog

  Specifies whether to enable asynchronous writing of logs. The default value is True.

  If you set this parameter to False, logs are written in synchronous mode. This ensures that all logs are written before an OBServer node crashes. However, the OBServer node performance is significantly reduced. We recommend that you set this parameter to True.

• max_syslog_file_count

  The maximum number of log files of each type. The default value is 0. This parameter takes effect only when its value is greater than 0 and greater than or equal to the value of syslog_file_uncompressed_count and enable_syslog_recycle is set to True.

  Notice

  Although the value range of this parameter is [0, +∞), the OBServer source code specifies that the value of this parameter cannot exceed the value of the MAX_LOG_FILE_COUNT parameter, which is 10240. You can set a larger value, but the MAX_LOG_FILE_COUNT parameter prevails.

• syslog_io_bandwidth_limit

  The maximum bandwidth for log I/O. The default value is 30 MB.

  Notice

  When the log printing speed exceeds the limit, the following message is printed.

  REACH SYSLOG RATE LIMIT
  

• syslog_level

  The lowest level of the logs to print. The default value is WDIAG.

• syslog_disk_size

  The maximum disk space available for system logs. The default value is 0 MB. The disk space available for system logs is subject to the value of syslog_disk_size and the actual disk space. The smaller of the two prevails. When the remaining disk space is less than 2 GB, the earliest log files are deleted. If log compression is enabled, log files are compressed when the remaining disk space is less than 4 GB.

• syslog_compress_func

  The compression algorithm for system logs. The default value is none, which specifies to disable log compression. Supported compression algorithms are zstd_1.0 and zstd_1.3.8. You can set the parameter to a supported compression algorithm to enable log compression.

  Notice

  Only log files suffixed with .log.{timestamp} are compressed. Those suffixed with wf and those not suffixed with {timestamp} are not compressed. {timestamp} indicates the time when the log file was generated.

• syslog_file_uncompressed_count

  The number of uncompressed system log files. This parameter takes effect only when syslog_compress_func is not set to none. The number of uncompressed system log files is counted separately for each log type, excluding the log files not suffixed with {timestamp}. The relationships between the syslog_compress_func parameter and the syslog_file_uncompressed_count, syslog_disk_size, and max_syslog_file_count parameters are as follows:

  • When syslog_compress_func =none is specified:

    1. The syslog_file_uncompressed_count parameter is invalid.
    2. The syslog_disk_size and max_syslog_file_count parameters control the maximum number and size of log files.

  • When syslog_compress_func !=none is specified:

    1. The max_syslog_file_count parameter is invalid.
    2. When the number of log files exceeds the value of syslog_file_uncompressed_count, redundant log files will be compressed. When the total size of log files approaches the value of syslog_disk_size, the earliest log files are deleted.