obdiag is a CLI diagnostic tool designed for OceanBase Database. It performs comprehensive scans and collects crucial data, such as logs, SQL audit records, and process stack information of OceanBase. You may deploy your OceanBase cluster by using OceanBase Control Platform (OCP) or OceanBase Deployer (OBD), or manually deploy it based on the OceanBase documentation. Regardless of the deployment mode, you can use obdiag to gather diagnostic information with a few clicks. This powerful tool has now been officially open-sourced, further enhancing its accessibility and usability for developers and database administrators.
The obdiag team has compiled relevant experiences regarding OceanBase diagnosis and tuning and has commenced releasing a series of tutorial articles. In this article, we will learn how to install and configure obdiag, and use obdiag to gather and analyze logs of OceanBase clusters.
OceanBase Database, being a native distributed database, so root cause analysis for faults is often complex, as it may involve many factors such as server environment, configuration parameters, and operational loads. When troubleshooting issues, experts need to gather extensive information for fault analysis. This is where OceanBase Diagnostic Tool (obdiag) comes into play. It aims to efficiently gather and analyze information dispersed across various nodes.
obdiag is a CLI diagnostic tool designed for OceanBase, with the following features:
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y oceanbase-diagnostic-tool
source /usr/local/oceanbase-diagnostic-tool/init.sh
Download the obdiag package: https://www.oceanbase.com/softwarecenter
sudo yum install -y oceanbase-diagnostic-tool*.rpm
source /usr/local/oceanbase-diagnostic-tool/init.sh
You can create or edit a user-defined configuration file by running the obdiag config <option>
command. By default, the configuration file is named config.yml
and is stored in the ~/.obdiag/
directory. Template configuration files are stored in the ~/.obdiag/example
directory.
obdiag config -h <db_host> -u <sys_user> [-p password] [-P port]
The following table describes the parameters:
Parameter | Required? | Description |
db_host | Yes | The IP address used to connect to the sys tenant of the OceanBase cluster. |
sys_user | Yes | The username used to connect to the sys tenant of the OceanBase cluster. To avoid permission issues, we recommend that you use 'root@sys'. |
-p password | No | The password used to connect to the sys tenant of the OceanBase cluster. This parameter is left empty by default. |
-P port | No | The port of the sys tenant of the OceanBase cluster. Port 2881 is used by default. |
Here are some examples:
# A password is specified.
obdiag config -hxx.xx.xx.xx -uroot@sys -p***** -P2881
# No password is specified.
obdiag config -hxx.xx.xx.xx -uroot@sys -p"" -P2881
# Through obproxy.
obdiag config -hxx.xx.xx.xx -uroot@sys#obtest -p***** -P2883
You can run this command to gather logs of the specified OceanBase cluster:
obdiag gather log [options]
The options are explained as follows:
Option | Required? | Data type | Default value | Description |
--from | No | String | This option is left empty by default. | The start time of log collection in the yyyy-mm-dd hh:mm:ssformat. For example, 1970-01-01 12:00:00. |
--to | No | String | This option is left empty by default. | The end time of log collection in the yyyy-mm-dd hh:mm:ssformat. For example, 1970-01-01 13:00:00. |
--since | No | String | This option is left empty by default. | The most recent period for which logs are collected, in the \<n> <m\|h\|d>format, where n is a number, m indicates minutes, h indicates hours, and d indicates days. For example, 30mspecifies to collect logs of the last 30 minutes. |
--scope | No | String | all | The type of logs to be collected. Valid values: observer, election, rootservice, and all. |
--grep | No | String | This option is left empty by default. | The search keyword. |
--encrypt | No | String | false | Specifies whether to encrypt the returned files. Valid values: true and false. |
--store_dir | No | String | The default path is the current path in which the command is executed. | The local path where the results are stored. |
-c | No | String | ~/.obdiag/config.yml | The path of the configuration file. |
Here is an example:
obdiag gather log --scope observer --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep STORAGE --encrypt true
...
ZipFileInfo:
+-------------------+-----------+
| Node | LogSize |
+===================+===========+
| xxx.xxx.xxx.xxx | 36.184M |
+-------------------+-----------+
...ZipFileInfo:
+-------------------+-----------+
| Node | LogSize |
+===================+===========+
| xxx.xxx.xxx.xxx | 44.176M |
+-------------------+-----------+
...
Summary:
+-------------------+-----------+----------+------------------+--------+------------------------------------------------------------------------+
| Node | Status | Size | Password | Time | PackPath |
+===================+===========+==========+==================+========+========================================================================+
| xxx.xxx.xxx.xxx | Completed | 36.762M | HYmVourcUyRNP8Om | 19 s | gather_pack_20220701183246/result_xxx.xxx.xxx.xxx_20220701183247.zip |
+-------------------+-----------+----------+------------------+--------+------------------------------------------------------------------------+
| xxx.xxx.xxx.xxx | Completed | 638.200M | 1RicMaiLUUNfemnj | 718 s | gather_pack_20220701183246/result_xxx.xxx.xxx.xxx_20220701183918.zip |
+-------------------+-----------+----------+------------------+--------+------------------------------------------------------------------------+
Here are examples of gathering logs of a recent period:
# Collect logs of the last hour.
obdiag gather log --since 1h
# Collect logs of the last 30 minutes.
obdiag gather log --since 30m
# Collect logs of the last 30 minutes and specify the configuration file.
obdiag gather log --since 30m -c /root/config.yml
Filter and gather logs based on keywords:
# Collect logs from the last 30 minutes and filter by the keyword "TRACE_ID":
obdiag gather log --grep "TRACE_ID"
# Collect logs from the last 30 minutes and filter by multiple keywords, for example "AAAAA" and "BBBBB":
obdiag gather log --grep "AAAAA" --grep "BBBBB"
# Collect logs from a specified time range and filter by multiple keywords, for example "AAAAA" and "BBBBB":
obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB"
You can run the command obdiag analyze log [option]
to analyze logs of an OceanBase cluster online, or specify the --files
option to enable offline analysis.
Note:
obdiag analyze log [options]
The following table describes the options:
Option | Required? | Data type | Default value | Description |
--from | No | String | This option is left empty by default. | The start time of log collection in the yyyy-mm-dd hh:mm:ssformat. For example, 1970-01-01 12:00:00. |
--to | No | String | This option is left empty by default. | The end time of log collection in the yyyy-mm-dd hh:mm:ssformat. For example, 1970-01-01 13:00:00. |
--since | No | String | This option is left empty by default. | The most recent period for which logs are collected, in the \<n> <m\|h\|d>format, where n is a number, m indicates minutes, h indicates hours, and d indicates days. For example, 30mspecifies to collect logs of the last 30 minutes. |
--scope | No | String | all | The type of logs of the OceanBase cluster to be analyzed. Valid values: observer, election, rootservice, and all. |
--grep | No | String | This option is left empty by default. | The search keyword. |
--store_dir | No | String | The default path is the current path in which the command is executed. | The local path where the results are stored. |
--log_level | No | String | WARN | The level of logs of the OceanBase cluster to be analyzed. Valid values in ascending order: DEBUG, TRACE, INFO, WDIAG, WARN, EDIAG, and ERROR. Logs of the specified level and higher are analyzed. |
--files | No | String | This option is left empty by default. | If you specify the --filesoption, the offline log analysis mode is enabled. In offline analysis mode, you must specify the name or path of the log file of the OceanBase cluster, but do not need to specify the --from, --to, --since, or --ob_install_diroptions. |
-c | No | String | ~/.obdiag/config.yml | The path of the configuration file. |
Example of online log analysis:
obdiag analyze log --from "2023-10-08 10:25:00" --to "2023-10-08 11:30:00"
...
FileListInfo:
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node | LogList |
+================+=======================================================================================================================================================================================================================+
| xx.xx.xx.xx | ['observer.log.20231008104204260', 'observer.log.20231008111305072', 'observer.log.20231008114410668', 'observer.log.wf.20231008104204260', 'observer.log.wf.20231008111305072', 'observer.log.wf.20231008114410668'] |
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
...
Analyze OceanBase Online Log Summary:
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| Node | Status | FileName | ErrorCode | Message | Count |
+================+===========+==============================================================================+=============+===============================================================================================================================+=========+
| xx.xx.xx.xx | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008104204260 | -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use | 2 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| xx.xx.xx.xx | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008111305072 | -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use | 8 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| xx.xx.xx.xx | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008114410668 | -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use | 10 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| xx.xx.xx.xx | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008114410668 | -4009 | IO error | 20 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
For more details, please run cmd 'cat analyze_pack_20231008171201/result_details.txt'
Here are examples of analyzing logs of a recent period:
# Analyze the logs of the last hour online. When you run this command, obdiag pulls the logs of the last hour from the remote host for analysis to diagnose the errors that have occurred.
obdiag analyze log --since 1h
# Analyze the logs of the last 30 minutes online. When you run this command, obdiag pulls the logs of the last 30 minutes from the remote host for analysis to diagnose the errors that have occurred.
obdiag analyze log --since 30m
Example of offline log analysis:
ls -lh test/
-rw-r--r-- 1 admin staff 256M Oct 8 17:24 observer.log.20231008104204260
-rw-r--r-- 1 admin staff 256M Oct 8 17:24 observer.log.20231008111305072
-rw-r--r-- 1 admin staff 256M Oct 8 17:24 observer.log.20231008114410668
-rw-r--r-- 1 admin staff 18K Oct 8 17:24 observer.log.wf.20231008104204260
-rw-r--r-- 1 admin staff 19K Oct 8 17:24 observer.log.wf.20231008111305072
-rw-r--r-- 1 admin staff 18K Oct 8 17:24 observer.log.wf.20231008114410668
obdiag analyze log --files test/
Analyze OceanBase Offline Log Summary:
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| Node | Status | FileName | ErrorCode | Message | Count |
+===========+===========+=======================================================================+=============+===============================================================================================================================+=========+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008104204260 | -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use | 2 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008111305072 | -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use | 8 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008114410668 | -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use | 10 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008114410668 | -4009 | IO error | 20 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
For more details, please run cmd 'cat analyze_pack_20231008172144/result_details.txt'
Analyze the specified OBServer log file offline:
# Analyze the specified log file offline.
obdiag analyze log --files observer.log.20230831142211247
obdiag v1.2 supports one-click gathering of diagnostic information. A diagnostic tool not only needs the capability to gather information but also requires data analysis capabilities. Therefore, in obdiag v1.3.0, log analysis function has been introduced, enabling users to check for any abnormal situations with a few clicks.
The main architecture of obdiag log gathering and analysis relies on the centralized collection mode of obdiag. When a user initiates an obdiag analysis, data needs to be gathered from various nodes and then processed centrally.
The process of obdiag online log analysis:
Find the latest version of obdiag for free from the OceanBase Software Center.
Explore comprehensive usage guides and configuration details in the obdiag Documentation.
GitHub Repository
Review the source code, report issues, and contribute to the project on GitHub.