Alert logs|V4.3.5| docs|Distributed Database

Alert logs

Last Updated：2025-06-25 08:02:52 Updated

OceanBase Database V4.2.3 and later support alert logs.

Overview

Alert logs include INFO, WARN, and ERROR logs generated during running of OBServer nodes. They record critical events of clusters to help you monitor the cluster status and solve common issues.

Format

Alert logs are stored in CSV files. Multiple log fields are separated with vertical lines (|). Log content is enclosed in double quotation marks ("").

Here is a sample alert log:

2023-12-21 13:46:57.650115|INFO|LOG|OB_LOG_SERVICE_START_SUCCESS|0|1001|51152|OBServer|Y5B690B7C0505-00060C87BB2B5971-0-0|start|ob_log_service.h:224|"OBServer log service start success, cost 638519 us."

An alert log contains 12 fields: time, level, module, event, errcode, tenant_id, thread_id, thread_name, trace_id, func_name, code_location, and message (including the solution). The following table describes the fields.

Field	Description	Example
time	The time when the log was generated.	2023-12-21 13:46:57.650115
level	The log level. Valid values are `INFO`, `WARN`, and `ERROR`.	INFO
module	The module to which the log belongs. Valid values are `STORAGE`, `SQL`, and `TRANS`.	STORAGE
event	The event ID, which uniquely identifies an event that occurs during running of the cluster.	OB_LOG_SERVICE_START_SUCCESS
errcode	The error code of the log. The error code of `INFO` logs is `0`.	-4012
tenant_id	The ID of the tenant to which the current task belongs.	1002
thread_id	The ID of the current thread.	51152
thread_name	The name of the current thread.	observer
trace_id	The trace ID of the current SQL statement, which is used to trace the execution process of the SQL statement.	Y5B690B7C0505-00060C87BB2B5971-0-0
func_name	The name of the function for which the log is generated.	start
code_location	The position of the log in the code.	ob_log_service.h:224
message	The description, cause, and solution of the log event.	"observer log service start success, cost 638519 us."

Storage

Alert logs are stored in the alert.log file under the {Installation directory of the OBServer node}/log/alert directory.

A single alert log file cannot exceed 256 MB in size. When the size of a log file reaches 256 MB, the system performs log rotation. During log rotation, the system appends the start timestamp of the rotation to the original file name in the yyyyMMddHHmmss format, for example, alert.log.20220304102928236.

Query

Query alert logs from an external table

The alert.log file is stored in the CSV format. You can query alert logs only from an external table in the sys tenant.

The external table contains two more fields than the alert.log file: ip and port, which specify the IP address and port number of the OBServer node where the alert occurs. You can execute the following statement to query alert logs:

obclient> SELECT * FROM sys_external_tbs.__all_external_alert_log_info;

A sample query result is as follows:

+----------------------------+-------+----------------+---------------------------------------+---------+----------------+------+-----------+-----------+-------------+-----------------------------------+-------------------+--------------------------+-----------------------------------------------------------+
| time                       | level | module         | event                                 | errcode | ip             | port | tenant_id | thread_id | thread_name | trace_id                          | func_name         | code_location            | message                                                   |
+----------------------------+-------+----------------+---------------------------------------+---------+----------------+------+-----------+-----------+-------------+-----------------------------------+-------------------+--------------------------+-----------------------------------------------------------+
| 2024-04-01 14:01:21.367457 | INFO  | SERVER         | OB_SERVER_INIT_BEGIN                  |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | init              | ob_server.cpp:263        | [server_start 1/18] observer init begin.                  |
| 2024-04-01 14:01:21.378711 | INFO  | LIB            | OB_SERVER_SYSLOG_SERVICE_INIT_BEGIN   |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | init              | ob_log.cpp:1491          | [server_start 2/18] observer syslog service init begin.   |
| 2024-04-01 14:01:21.381666 | INFO  | LIB            | OB_SERVER_SYSLOG_SERVICE_INIT_SUCCESS |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | init              | ob_log.cpp:1565          | [server_start 3/18] observer syslog service init success. |
| 2024-04-01 14:01:22.267567 | INFO  | SERVER         | OB_SERVER_INIT_SUCCESS                |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | init              | ob_server.cpp:540        | [server_start 4/18] observer init success.                |
| 2024-04-01 14:01:22.267599 | INFO  | SERVER         | OB_SERVER_START_BEGIN                 |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | start             | ob_server.cpp:854        | [server_start 5/18] observer start begin.                 |
| 2024-04-01 14:01:22.267613 | INFO  | SERVER         | OB_SERVER_INSTANCE_START_BEGIN        |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | start             | ob_server.cpp:872        | [server_start 6/18] observer instance start begin.        |
| 2024-04-01 14:01:22.300143 | INFO  | STORAGE_BLKMGR | OB_SERVER_BLOCK_MANAGER_START_BEGIN   |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | start             | ob_block_manager.cpp:199 | [server_start 7/18] block manager start begin.            |
| 2024-04-01 14:01:22.354769 | INFO  | STORAGE_BLKMGR | OB_SERVER_BLOCK_MANAGER_START_SUCCESS |       0 | xx.xx.xx.xx    | 2882 |         0 |     70604 | observer    | Y0-00000000xxxxxxxx-0-0           | start             | ob_block_manager.cpp:256 | [server_start 8/18] block manager start success.          |
| 2024-04-01 14:02:15.592144 | INFO  | BOOTSTRAP      | OB_BOOTSTRAP_PREPARE_BEGIN            |       0 | xx.xx.xx.xx    | 2882 |         1 |     71165 | T1_L0_G0    | YB42AC1E87E2-00061502xxxxxxxx-0-0 | prepare_bootstrap | ob_bootstrap.cpp:243     | bootstrap prepare begin.                                  |
| 2024-04-01 14:02:21.341029 | INFO  | BOOTSTRAP      | OB_BOOTSTRAP_PREPARE_SUCCESS          |       0 | xx.xx.xx.xx    | 2882 |         1 |     71165 | T1_L0_G0    | YB42AC1E87E2-00061502xxxxxxxx-0-0 | prepare_bootstrap | ob_bootstrap.cpp:275     | bootstrap prepare success.                                |
+----------------------------+-------+----------------+---------------------------------------+---------+----------------+------+-----------+-----------+-------------+-----------------------------------+-------------------+--------------------------+-----------------------------------------------------------+
10 rows in set, 6 warnings (0.008 sec)

If the size of the alert.log file reaches 256 MB, the system generates a new log file. In this case, you must manually refresh the file list in the external table.

obclient> ALTER EXTERNAL TABLE sys_external_tbs.__all_external_alert_log_info refresh;

Then you can query all log files.

Notice

To ensure tool compatibility and upgrade consistency, do not manually create the __all_external_alert_log_info table.

Troubleshooting based on alert logs

INFO logs

You can query INFO logs in the alert.log file for the progress of a specific process. In the following startup log of an OBServer node, [server_start 3/12] indicates that the observer process startup contains 12 steps and step 3 is in progress. The event ID OB_SERVER_SYSLOG_SERVICE_INIT_SUCCESS of this step indicates that the syslog service is successfully initialized.

2024-02-22 11:39:39.910566|INFO|SERVER|OB_SERVER_START_BEGIN|0|0|88288|OBServer|Y0-00000000xxxxxxxx-0-0|main|main.cpp:556|"[server_start 1/12] OBServer start begin."
2024-02-22 11:39:39.933725|INFO|LIB|OB_SERVER_SYSLOG_SERVICE_INIT_BEGIN|0|0|88288|OBServer|Y0-00000000xxxxxxxx-0-0|init|ob_log.cpp:1457|"[server_start 2/12] OBServer syslog service start begin."
2024-02-22 11:39:39.937152|INFO|LIB|OB_SERVER_SYSLOG_SERVICE_INIT_SUCCESS|0|0|88288|OBServer|Y0-00000000xxxxxxxx-0-0|init|ob_log.cpp:1531|"[server_start 3/12] OBServer syslog service start success."

WARN and ERROR logs

You can query WARN and ERROR logs in the alert.log file for the event description, cause, and solution. The following log shows that the clog module fails to resize the disk because the new disk size is insufficient to hold all tenants. It also recommends you to set the log_disk_size parameter to a value greater than 43526 MB to solve this problem.
```
2023-11-17 03:19:38.592090|ERROR|CLOG|OB_CLOG_ALLOCATE_DISK_SPACE_FAILED|-4290|0|2569344|OBServer|Y0-0000000000000000-0-0|resize_|ob_server_log_block_mgr.cpp:211|"new log_disk_size(21504MB) is not enough to hold all tenants. [suggestion] set log_disk_size greater than 43526MB."
```