obdiag gather log

If OceanBase Diagnostic Tool (obdiag) is independently deployed, you can run the obdiag gather log command to collect logs of the specified OceanBase cluster.

If your OceanBase cluster is deployed by using OceanBase Deployer (obd), you can run obdiag commands on obd to collect diagnostic information of the cluster. For more information about the commands, see obdiag commands.

Syntax

obdiag gather log [options]

The following table describes the options.

Examples

Method 1: Use the command out-of-the-box without a configuration file

• Collect logs of the specified period

  obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" \
      --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
      --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
      --config obcluster.servers.global.ssh_username=test \
      --config obcluster.servers.global.ssh_password=****** \
      --config obcluster.servers.global.home_path=/home/admin/oceanbase
  

  The output is as follows:

  ...
  ZipFileInfo:
  +-------------------+-----------+
  | Node              | LogSize   |
  +===================+===========+
  | xxx.xxx.xxx.xxx   | 36.184M   |
  +-------------------+-----------+
  ...
  
  ZipFileInfo:
  +-------------------+-----------+
  | Node              | LogSize   |
  +===================+===========+
  | xxx.xxx.xxx.xxx   | 44.176M   |
  +-------------------+-----------+
  ...
  
  Summary:
  +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+
  | Node              | Status    | Size     | Time   | PackPath                                                                      |
  +===================+===========+==========+========+===============================================================================+
  | xxx.xxx.xxx.xxx   | Completed | 36.762M  | 19 s   | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183247.zip   |
  +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+
  | xxx.xxx.xxx.xxx   | Completed | 638.200M | 718 s  | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183918.zip   |
  +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+
  

  Notice

  If encryption is enabled, collected logs are encrypted when being compressed. The value in the Password column indicates the password for decrypting the .zip package. Encryption is disabled by default.

• Collect logs of a recent period

  # Collect logs of the last hour with the target nodes accessible to obdiag without a password.
  obdiag gather log --since 1h \
    --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
    --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
    --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase
  
  # Collect logs of the last hour with the target nodes accessible to obdiag without a password, and specify to collect at most 300 MB of logs stored in at most five log files from a single node. Note that if a node contains more than 300 MB of logs or more than five log files, obdiag does not collect logs from the node.
  obdiag gather log --since 1h \
    --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
    --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
    --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase \
    --inner_config obdiag.basic.file_number_limit=5
    --inner_config obdiag.basic.file_size_limit=300M
  
  # Collect logs of the last hour with the target nodes accessible to obdiag without a password, and specify to collect at most 1 GB of logs stored in at most five log files from a single node. Note that if a node contains more than 1 GB of logs or more than five log files, obdiag does not collect logs from the node.
  obdiag gather log --since 1h \
    --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
    --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
    --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase \
    --inner_config obdiag.basic.file_number_limit=5
    --inner_config obdiag.basic.file_size_limit=1G
  
  # Collect logs of the last 30 minutes with the target nodes accessible to obdiag without a password.
  obdiag gather log --since 30m \
    --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
    --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
    --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase
  
  # Collect logs of the last 30 minutes with the information on different OBServer nodes varying.
  obdiag gather log --since 30m \
    --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
    --config obcluster.servers.nodes[0].ssh_username=test1 \
    --config obcluster.servers.nodes[0].ssh_password=****** \
    --config obcluster.servers.nodes[0].home_path=/home/test1/oceanbase \
    --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2
    --config obcluster.servers.nodes[1].ssh_username=test2 \
    --config obcluster.servers.nodes[1].ssh_password=****** \
    --config obcluster.servers.nodes[1].home_path=/home/test2/oceanbase
  
  # Collect logs of the last hour with the connection information of the sys tenant specified. In this case, obdiag automatically obtains the IP addresses of all nodes in the cluster.
  obdiag gather log --since 1h \
    --config db_host=xx.xx.xx.xx \
    --config db_port=xxxx \
    --config tenant_sys.user=root@sys \
    --config tenant_sys.password=*** \
    --config obcluster.servers.global.ssh_username=test \
    --config obcluster.servers.global.ssh_password=****** \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase
  

• Collect logs filtered by keyword

  # Collect logs of the last 30 minutes with logs filtered by "TRACE_ID".
  obdiag gather log --grep "TRACE_ID"  \
    --config db_host=xx.xx.xx.xx \
    --config db_port=xxxx \
    --config tenant_sys.user=root@sys \
    --config tenant_sys.password=*** \
    --config obcluster.servers.global.ssh_username=test \
    --config obcluster.servers.global.ssh_password=****** \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase
  
  # Collect logs of the last 30 minutes with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB".
  obdiag gather log --grep "AAAAA" --grep "BBBBB" \
    --config db_host=xx.xx.xx.xx \
    --config db_port=xxxx \
    --config tenant_sys.user=root@sys \
    --config tenant_sys.password=*** \
    --config obcluster.servers.global.ssh_username=test \
    --config obcluster.servers.global.ssh_password=****** \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase
  
  # Collect logs of the specified period with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB".
  obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB" \
    --config db_host=xx.xx.xx.xx \
    --config db_port=xxxx \
    --config tenant_sys.user=root@sys \
    --config tenant_sys.password=*** \
    --config obcluster.servers.global.ssh_username=test \
    --config obcluster.servers.global.ssh_password=****** \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase
  

  Notice

  If logs are filtered by multiple keywords, the keywords are evaluated by using the AND operator. In other words, only logs that contain all the specified keywords are collected.

Method 2: Use the command with a configuration file

Before you use obdiag, make sure that you have configured the login information of the target nodes in the config.yml file in the ~/.obdiag directory. For more information, see Configure obdiag.

• Collect logs of the specified period

  obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00"
  

  The output is as follows:

  ...
  ZipFileInfo:
  +-------------------+-----------+
  | Node              | LogSize   |
  +===================+===========+
  | xxx.xxx.xxx.xxx   | 36.184M   |
  +-------------------+-----------+
  ...
  
  ZipFileInfo:
  +-------------------+-----------+
  | Node              | LogSize   |
  +===================+===========+
  | xxx.xxx.xxx.xxx   | 44.176M   |
  +-------------------+-----------+
  ...
  
  Summary:
  +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+
  | Node              | Status    | Size     | Time   | PackPath                                                                      |
  +===================+===========+==========+========+===============================================================================+
  | xxx.xxx.xxx.xxx   | Completed | 36.762M  | 19 s   | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183247.zip   |
  +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+
  | xxx.xxx.xxx.xxx   | Completed | 638.200M | 718 s  | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183918.zip   |
  +-------------------+-----------+----------+--------+-------------------------------------------------------------------------------+
  
  

  Notice

  If encryption is enabled, collected logs are encrypted when being compressed. The value in the Password column indicates the password for decrypting the .zip package. Encryption is disabled by default.

• Collect logs of a recent period

  # Collect logs of the last hour.
  obdiag gather log --since 1h
  
  # Collect logs of the last 30 minutes.
  obdiag gather log --since 30m
  
  # Collect logs of the last 30 minutes with the configuration file of the target cluster specified.
  obdiag gather log --since 30m -c /root/config.yml
  

• Collect logs filtered by keyword

  # Collect logs of the last 30 minutes with logs filtered by "TRACE_ID".
  obdiag gather log --grep "TRACE_ID"
  
  # Collect logs of the last 30 minutes with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB".
  obdiag gather log --grep "AAAAA" --grep "BBBBB"
  
  # Collect logs of the specified period with logs filtered by multiple keywords, such as "AAAAA" and "BBBBB".
  obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB"
  

  Notice

  If logs are filtered by multiple keywords, the keywords are evaluated by using the AND operator. In other words, only logs that contain all the specified keywords are collected.

Results

The command generates a folder in the specified directory. A sample folder name is obdiag_gather_pack_20240808105846.

Define a data desensitization rule

In most cases, data desensitization rules are defined on the business side. In other words, sensitive data is defined by users. obdiag supports custom data desensitization rules in V2.5.0 and later. Data desensitization rule files are stored in the ~/.obdiag/gather/redact/ directory. Each .py file corresponds to a data desensitization rule. The file name is the rule name.

This section describes how to define a data desensitization rule named a_test.

1. Create a rule file.

   In the ~/.obdiag/gather/redact/ directory, create a rule file named a_test.py, which is also the rule name. In other words, a file with the absolute path ~/.obdiag/gather/redact/a_test.py is created.

2. Create a rule class.

   To facilitate management, you need to create a rule class to inherit the RedactBase class. You can define the class name, which is usually the same as the file name. Here is an example:

   from handler.gather.plugins.redact import RedactBase
   class a_test(RedactBase):
   

3. Create a rule method.

   The rule class needs to implement the redact method. text is the input file content, which is not split into lines. You can split the content as needed.

   text_lines = text.split('\n')
   

4. Return the processed information as a string.

   If the file content is split into lines, the processing time increases greatly. We recommend that you do not split the file content unless necessary.

5. Create a rule instance.

   You need to set a variable named after the rule name for being called by the obdiag kernel. In this example, you need to set the a_test variable as follows:

   a_test = a_test()
   

Complete sample code

from handler.gather.plugins.redact import RedactBase
import re
class a_test(RedactBase):
    def redact(self, text):
        text_lines = text.split('\n')
        redacted_log = ''
        for line in text_lines:
            if 'test' in line:
                redacted_log = redacted_log + re.sub('test', '******', line)
        return redacted_log
a_test = a_test()