Use obdiag to inspect an OceanBase cluster

If OceanBase Diagnostic Tool (obdiag) is independently deployed, you can use the obdiag check run commands to inspect the status of an OceanBase cluster. The inspection process analyzes the cluster from multiple perspectives such as the system kernel parameters and internal tables, identifies the causes of existing or possible exceptions, and provides O&M suggestions.

Prerequisites

check commands

Overview

# List all inspection packages
obdiag check list

# Perform a full inspection (most commonly used)
obdiag check run

# Perform a proof of concept (POC) check for columnar storage
obdiag check run --cases=column_storage_poc

# Check the version of OceanBase Database Proxy (ODP)
obdiag check run --obproxy_cases=proxy

# Check the deployment environment
obdiag check run --cases=build_before

# List the inspection tasks when a Sysbench benchmark is run
obdiag check run --cases=sysbench_run

# List the inspection tasks before a Sysbench benchmark is run
obdiag check run --cases=sysbench_free

Syntax

obdiag check run [options]

The following table describes the options.

Here is an example:

obdiag check run

'cat ./check_report/check_report_2023-10-30-16:15:52.table', export type is table
For more details, please run cmd 'cat ./check_report/check_report_2023-10-30-16:15:52.table'
If you want to view detailed obdiag logs, please run:' obdiag display-trace --trace_id a7674ecb-0d99-36fe-b584-3b707b4647bc'

Tutorial for writing a task

A task is a specific inspection scenario. obdiag runs a task based on the dedicated script file in the YAML format.

The script file of a task contains a declaration at the beginning, which helps you implement professional inspections on an OceanBase cluster.

Preparations

Before you write the script file of a task, you must specify the path of the file.

The file must be stored in the directory specified by the CHECK.tasks_base_path parameter in the config.yml configuration file. Check whether the task falls into an existing category in the directory. If not, create a folder to declare the category.

Here is an example:

# Go to `${CHECK.tasks_base_path}/observer`, create a folder named `test`, and then create a sample task script file named `test.yaml`.
cd ~/.obdiag/check/tasks/observer
mkdir test
cd test
touch test.yaml

Write a task script

A task script is a list that declares the steps to be executed in an inspection. You can specify steps for different versions of OceanBase Database, so the script can be used for inspections of OceanBase clusters of the specified versions.

Here is a sample task script:

info: testinfo
task:
  - version: "[3.1.0,3.2.4]"
    steps:
      {steps_object}
  - version: "[4.2.0.0,4.3.0.0]"
    steps:
      {steps_object}

The parameters are described as follows:

steps

steps is a list of multiple execution processes.

An element of steps is a single step that involves the following parameters.

In the following examples, steps: serves only as a mark and has no actual meaning.

• get_system_parameter

steps:
- type: get_system_parameter
  parameter: parameter
  result:
    set_value: servervm.max_map_count

• ssh

Remotely execute the instruction and obtain the corresponding return value.

steps:
- type: ssh
  ssh: wc -l /proc/${task_OBServer_pid}/maps | awk '{print $1}'
  result:
    set_value: observerMaps

• sql

Execute an SQL statement and obtain the corresponding value.

steps:
- type: sql
  sql: select tenant_name from oceanbase.table_name from where tenant_id=${taskTenantId};
  result:
    set_value: tenant_name

Result & verification

The verification feature depends on information in the result section to verify the task result.

The format is as follows:

result:
      set_value: {set_value}
      verify_type: {verify_type}
      report_type: {report_type}
      verify: {verify}
      err_msg: {err_msg}

The parameters are described as follows:

Judgment types supported by verify_type:

At present, the judgment types supported by verify_type apply only to the base and int types.

• between: checks whether the value of the parameter specified by the set_value parameter falls within the range specified by the verify parameter.
• max: checks whether the value of the parameter specified by the set_value parameter is smaller than the value specified by the verify parameter.
• min: checks whether the value of the parameter specified by the set_value parameter is greater than the value specified by the verify parameter.
• equal: checks whether the value of the parameter specified by the set_value parameter is equal to the value specified by the verify parameter.

Description of base

You can replace new_expr with the expression specified by the verify parameter to manually verify the logic in your local environment.

if ${new_expr}; then
    echo "true"
else
    echo "false"
fi

Write a package script

At present, a package script comprises the observer_check_pakage.yaml and obproxy_check_package.yaml files, which respectively correspond to the inspection packages of OceanBase Database and ODP.

A package is a large dictionary with the following structure:

{case_name}:
  info_en: "" # The English description of the package.
  info_cn: "" # The Chinese description of the package.
  tasks:
    {tasks_names} # A list structure of names of one or more tasks.

After you write the package script, run the obdiag check list command to view the list of available cases.

Manually update tasks

We will update these tasks from time to time to provide more effective inspection tasks. To update your inspection package to the latest version, copy the ${code_path}/handler/check/tasks folder and the *check_package.yaml file from the code repository of obdiag to respectively replace the original tasks folder and *check_package.yaml file in the ~/.obdiag/check directory.