Troubleshooting logic

This topic describes the troubleshooting rules and logic.

Describe the issue

When you encounter an issue, you must first clearly describe it. For example, you can describe the issue in the QA module in the OceanBase open source community. The template is as follows:

[Environment] Production environment or test environment
[OceanBase Database or other components]
[Version]
[Description] Clearly describe the issue .
[Recurrence path] Describe the actions taken after the issue occurred.
[Symptom and impact]

[Attachment]

We recommend that you describe issues based on the preceding template. Make sure that the following content is included in the description:

• Issue occurrence time: For an issue that is difficult to reproduce, the time information can help locate and analyze the issue.

• Client errors: Client errors contain rich information such as the SQL statement, stack, and execution time. Such information is very helpful in troubleshooting. When a conclusion is drawn, the information can also be used for comparison to further verify the conclusion.

Analyze the issue

Before troubleshooting engineers analyze the issue, they must first globally understand the database access link, including the following modules:

• Application

• Load balancer (LB)

• OceanBase Database Proxy (ODP)

• OBServer

You can look at the issue from the global perspective and then from the module perspective to help clarify your mind. ODP interacts with all modules. We recommend that you learn more about applications, LBs, and OBServers.

When you analyze an issue, you can start from the applications. First locate the module where the issue occurred and then identify the issues of the module.

The following are some inefficient troubleshooting methods. We recommend that you do not use these methods.

• If an error occurs when an application uses the database, search for logs of the WARN and ERROR levels in the log directory of ODP. If the printed logs are not standard or if too many WARN/ERROR logs exist, this method in inefficient and may be even unrelated to the issue.

• When a database connection is interrupted, directly check issues with OceanBase Database. You may fail to find any issue if you use this method.

Classify issues

This section describes how to classify common ODP issues to facilitate issue analysis. ODP issues are classified into the following categories from the business perspective:

• Logon failure: A logon failure easily occurs due to configuration errors. It is easy to reproduce and troubleshoot a logon failure.

• SQL execution

  • Error code returned: The backend service returns an error code. You can check the OBServer and ODP based on the error code.

  • Slow SQL: Determine the time consumption of each module, find the bottleneck, and perform performing tuning accordingly.

  • Disconnection: Send a FIN packet to locate the module that initiates the disconnection, and then check the module.

    • Disconnection upon timeout

    • Disconnection upon exceptions

Solve the problem

Three tools are available to resolve OceanBase Database Proxy (ODP) issues: ODP logs, Linux commands (including network commands, system commands, and text commands), and OceanBase Cloud Platform (OCP).

• For more information about ODP log, see Logs.

• Reference materials about Linux commands and OCP are common on the Internet. We recommend that you learn the related knowledge to facilitate troubleshooting.

• For more information about OCP, see OceanBase Cloud Platform.

Prometheus

In addition to a log system, ODP further provides the Prometheus system for integration with different monitoring platforms. Prometheus provides information such as the total number of requests, execution time, number of frontend connections, and number of backend connections. The following table describes the monitoring metrics.