If OceanBase Diagnostic Tool (obdiag) is independently deployed, you can run the obdiag gather log command to collect logs of the specified OceanBase cluster.
If your OceanBase cluster is deployed by using OceanBase Deployer (obd), you can run obdiag commands on obd to collect diagnostic information of the cluster. For more information about the commands, see obdiag commands.
Syntax
obdiag gather log [options]
The following table describes the options.
| Option | Required | Data type | Default value | Description |
|---|---|---|---|---|
| --from | No | string | Empty | The start time of log collection in the yyyy-mm-dd hh:mm:ss format.
NoteIf the obdiag version is earlier than V.2.0.0, do not enclose the parameter value in quotation marks. Otherwise, an error is returned. This limitation does not apply to V2.0.0 or later. |
| --to | No | string | Empty | The end time of log collection in the yyyy-mm-dd hh:mm:ss format.
NoteIf the obdiag version is earlier than V.2.0.0, do not enclose the parameter value in quotation marks. Otherwise, an error is returned. This limitation does not apply to V2.0.0 or later. |
| --since | No | string | Empty | The most recent period of time for log collection, in the <n> <m|h|d> format, where n specifies the time value, m represents "minute", h represents "hour", and d represents "day". For example, 30m specifies to collect logs of the last 30 minutes. |
| --scope | No | string | all | The type of logs of the OceanBase cluster to be collected. Valid values: observer, election, rootservice, and all. |
| --grep | No | string | Empty | The search keyword. |
| --store_dir | No | string | The current path where the command is executed | The local path where the results are stored. |
| --temp_dir | No | string | /tmp |
The directory where temporary files generated by the remote node during log collection are stored. |
| --redact | No | string | Empty | The data desensitization rule, such as all_sql. If multiple rules are specified, separate them with commas (,). For more information about how to define a data desensitization rule, see the Define a data desensitization rule section in this topic.
NoteThe data desensitization results are stored in a .zip file, which is not encrypted. You can manually encrypt it. |
| -c | No | string | ~/.obdiag/config.yml |
The path of the configuration file. |
| --inner_config | No | string | Empty | The configurations of obdiag, in the format of --inner_config key1=value1 --inner_config key2=value2.
NoteFor information about the parameters supported by this option, see System configuration file. |
| --config | No | string | Empty | The configurations of the cluster diagnosed by obdiag, in the format of --config key1=value1 --config key2=value2.
NoteFor information about the parameters supported by this option, see Configure obdiag. |
Examples
Method 1: Use the command out-of-the-box without a configuration file
Collect logs of the specified period
obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" \ --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \ --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \ --config obcluster.servers.global.ssh_username=test \ --config obcluster.servers.global.ssh_password=****** \ --config obcluster.servers.global.home_path=/home/admin/oceanbaseThe output is as follows:
... ZipFileInfo: +-------------------+-----------+ | Node | LogSize | +===================+===========+ | xxx.xxx.xxx.xxx | 36.184M | +-------------------+-----------+ ... ZipFileInfo: +-------------------+-----------+ | Node | LogSize | +===================+===========+ | xxx.xxx.xxx.xxx | 44.176M | +-------------------+-----------+ ... Summary: +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+ | Node | Status | Size | Time | PackPath | +===================+===========+==========+========+===============================================================================+ | xxx.xxx.xxx.xxx | Completed | 36.762M | 19 s | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183247.zip | +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+ | xxx.xxx.xxx.xxx | Completed | 638.200M | 718 s | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183918.zip | +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+Notice
If encryption is enabled, collected logs are encrypted when being compressed. The value in the
Passwordcolumn indicates the password for decrypting the .zip package. Encryption is disabled by default.Collect logs of a recent period
# Collect logs of the last hour with the target nodes accessible to obdiag without a password. obdiag gather log --since 1h \ --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \ --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \ --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \ --config obcluster.servers.global.home_path=/home/admin/oceanbase # Collect logs of the last hour with the target nodes accessible to obdiag without a password, and specify to collect at most 300 MB of logs stored in at most five log files from a single node. Note that if a node contains more than 300 MB of logs or more than five log files, obdiag does not collect logs from the node. obdiag gather log --since 1h \ --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \ --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \ --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \ --config obcluster.servers.global.home_path=/home/admin/oceanbase \ --inner_config obdiag.basic.file_number_limit=5 --inner_config obdiag.basic.file_size_limit=300M # Collect logs of the last hour with the target nodes accessible to obdiag without a password, and specify to collect at most 1 GB of logs stored in at most five log files from a single node. Note that if a node contains more than 1 GB of logs or more than five log files, obdiag does not collect logs from the node. obdiag gather log --since 1h \ --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \ --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \ --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \ --config obcluster.servers.global.home_path=/home/admin/oceanbase \ --inner_config obdiag.basic.file_number_limit=5 --inner_config obdiag.basic.file_size_limit=1G # Collect logs of the last 30 minutes with the target nodes accessible to obdiag without a password. obdiag gather log --since 30m \ --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \ --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \ --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \ --config obcluster.servers.global.home_path=/home/admin/oceanbase # Collect logs of the last 30 minutes with the information on different OBServer nodes varying. obdiag gather log --since 30m \ --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \ --config obcluster.servers.nodes[0].ssh_username=test1 \ --config obcluster.servers.nodes[0].ssh_password=****** \ --config obcluster.servers.nodes[0].home_path=/home/test1/oceanbase \ --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 --config obcluster.servers.nodes[1].ssh_username=test2 \ --config obcluster.servers.nodes[1].ssh_password=****** \ --config obcluster.servers.nodes[1].home_path=/home/test2/oceanbase # Collect logs of the last hour with the connection information of the sys tenant specified. In this case, obdiag automatically obtains the IP addresses of all nodes in the cluster. obdiag gather log --since 1h \ --config db_host=xx.xx.xx.xx \ --config db_port=xxxx \ --config tenant_sys.user=root@sys \ --config tenant_sys.password=*** \ --config obcluster.servers.global.ssh_username=test \ --config obcluster.servers.global.ssh_password=****** \ --config obcluster.servers.global.home_path=/home/admin/oceanbaseCollect logs filtered by keyword
# Collect logs of the last 30 minutes with logs filtered by "TRACE_ID". obdiag gather log --grep "TRACE_ID" \ --config db_host=xx.xx.xx.xx \ --config db_port=xxxx \ --config tenant_sys.user=root@sys \ --config tenant_sys.password=*** \ --config obcluster.servers.global.ssh_username=test \ --config obcluster.servers.global.ssh_password=****** \ --config obcluster.servers.global.home_path=/home/admin/oceanbase # Collect logs of the last 30 minutes with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB". obdiag gather log --grep "AAAAA" --grep "BBBBB" \ --config db_host=xx.xx.xx.xx \ --config db_port=xxxx \ --config tenant_sys.user=root@sys \ --config tenant_sys.password=*** \ --config obcluster.servers.global.ssh_username=test \ --config obcluster.servers.global.ssh_password=****** \ --config obcluster.servers.global.home_path=/home/admin/oceanbase # Collect logs of the specified period with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB". obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB" \ --config db_host=xx.xx.xx.xx \ --config db_port=xxxx \ --config tenant_sys.user=root@sys \ --config tenant_sys.password=*** \ --config obcluster.servers.global.ssh_username=test \ --config obcluster.servers.global.ssh_password=****** \ --config obcluster.servers.global.home_path=/home/admin/oceanbaseNotice
If logs are filtered by multiple keywords, the keywords are evaluated by using the
ANDoperator. In other words, only logs that contain all the specified keywords are collected.
Method 2: Use the command with a configuration file
Before you use obdiag, make sure that you have configured the login information of the target nodes in the config.yml file in the ~/.obdiag directory. For more information, see Configure obdiag.
Collect logs of the specified period
obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00"The output is as follows:
... ZipFileInfo: +-------------------+-----------+ | Node | LogSize | +===================+===========+ | xxx.xxx.xxx.xxx | 36.184M | +-------------------+-----------+ ... ZipFileInfo: +-------------------+-----------+ | Node | LogSize | +===================+===========+ | xxx.xxx.xxx.xxx | 44.176M | +-------------------+-----------+ ... Summary: +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+ | Node | Status | Size | Time | PackPath | +===================+===========+==========+========+===============================================================================+ | xxx.xxx.xxx.xxx | Completed | 36.762M | 19 s | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183247.zip | +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+ | xxx.xxx.xxx.xxx | Completed | 638.200M | 718 s | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183918.zip | +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+Notice
If encryption is enabled, collected logs are encrypted when being compressed. The value in the
Passwordcolumn indicates the password for decrypting the .zip package. Encryption is disabled by default.Collect logs of a recent period
# Collect logs of the last hour. obdiag gather log --since 1h # Collect logs of the last 30 minutes. obdiag gather log --since 30m # Collect logs of the last 30 minutes with the configuration file of the target cluster specified. obdiag gather log --since 30m -c /root/config.ymlCollect logs filtered by keyword
# Collect logs of the last 30 minutes with logs filtered by "TRACE_ID". obdiag gather log --grep "TRACE_ID" # Collect logs of the last 30 minutes with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB". obdiag gather log --grep "AAAAA" --grep "BBBBB" # Collect logs of the specified period with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB". obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB"Notice
If logs are filtered by multiple keywords, the keywords are evaluated by using the
ANDoperator. In other words, only logs that contain all the specified keywords are collected.
Results
The command generates a folder in the specified directory. A sample folder name is obdiag_gather_pack_20240808105846.
Define a data desensitization rule
In most cases, data desensitization rules are defined on the business side. In other words, sensitive data is defined by users. obdiag supports custom data desensitization rules in V2.5.0 and later. Data desensitization rule files are stored in the ~/.obdiag/gather/redact/ directory. Each .py file corresponds to a data desensitization rule. The file name is the rule name.
This section describes how to define a data desensitization rule named a_test.
Create a rule file.
In the
~/.obdiag/gather/redact/directory, create a rule file nameda_test.py, which is also the rule name. In other words, a file with the absolute path~/.obdiag/gather/redact/a_test.pyis created.Create a rule class.
To facilitate management, you need to create a rule class to inherit the RedactBase class. You can define the class name, which is usually the same as the file name. Here is an example:
from handler.gather.plugins.redact import RedactBase class a_test(RedactBase):Create a rule method.
The rule class needs to implement the
redactmethod.textis the input file content, which is not split into lines. You can split the content as needed.text_lines = text.split('\n')Return the processed information as a string.
If the file content is split into lines, the processing time increases greatly. We recommend that you do not split the file content unless necessary.
Create a rule instance.
You need to set a variable named after the rule name for being called by the obdiag kernel. In this example, you need to set the
a_testvariable as follows:a_test = a_test()
Complete sample code
from handler.gather.plugins.redact import RedactBase
import re
class a_test(RedactBase):
def redact(self, text):
text_lines = text.split('\n')
redacted_log = ''
for line in text_lines:
if 'test' in line:
redacted_log = redacted_log + re.sub('test', '******', line)
return redacted_log
a_test = a_test()