Application exception -- error message contains an OceanBase error code

During database operation, various exceptions may occur, such as application errors, database connection errors, database permission issues, database resource issues, and network problems. One such exception is an application exception where the error message contains an OceanBase error code.

To help you quickly identify the root cause of such exceptions and efficiently resolve them, this topic provides a clear and practical troubleshooting process for application exceptions with error messages containing OceanBase error codes. The process offers step-by-step guidance to enhance issue resolution efficiency and minimize business impact, thereby supporting your daily O&M work.

The following figure shows the troubleshooting process for an application exception where the error message contains an OceanBase error code.

Process

When an application exception occurs and the error message contains an OceanBase error code, you can troubleshoot the issue by using the following process.

First, determine whether you can use OBClient to connect to the database and manually reproduce the issue:

• If yes, connect to the database after the issue is reproduced for further troubleshooting.

• If no, troubleshoot the data source information in the program configuration.

Troubleshoot the issue by reproducing it

1. Use OBClient to connect to the OceanBase cluster at port 2881 (the default SQL port. If you used a custom port, replace 2881 with the actual port number).

2. Manually execute the SQL statement that caused the error to reproduce the issue.

3. Execute the following statement to obtain the trace_id.

   Notice

   You must execute the following statement immediately after the SQL error is triggered; otherwise, the trace_id of the SQL statement that caused the error cannot be obtained.

   MySQL mode

   Oracle mode

   The following statement is used to obtain the trace_id in MySQL mode:

   obclient> SELECT last_trace_id();
   

   The following statement is used to obtain the trace_id in Oracle mode:

   obclient> SELECT last_trace_id() FROM DUAL;
   

4. Obtain the host information where the SQL statement is actually executed.

   OceanBase clusters are generally deployed across multiple nodes. You can execute the following SQL statement to obtain the node where the SQL statement is actually executed, and then filter the logs.

   MySQL mode

   Oracle mode

   Execute the following statement in MySQL mode:

   obclient> SELECT * FROM oceanbase.GV$OB_SQL_AUDIT WHERE trace_id=last_trace_id;
   

   Replace last_trace_id with the trace_id obtained in the previous step.

   Execute the following statement in Oracle mode:

   obclient> SELECT * FROM SYS.GV$OB_SQL_AUDIT WHERE trace_id=last_trace_id;
   

   Replace last_trace_id with the trace_id obtained in the previous step.

   In the GV$OB_SQL_AUDIT view, the host corresponding to svr_ip is the host where the SQL statement is actually executed.

5. Use the ssh command to log in to the host based on the obtained host information.

6. Go to the log directory.

   Assume that the installation directory of OceanBase Database is /home/admin/oceanbase. The following example shows the specific path where logs are stored.

   cd /home/admin/oceanbase/log
   

7. Execute the following command to filter the required information from the logs.

   grep "${trace_id}" observer.log
   

   grep "${trace_id}" observer.log.xxx
   

   Replace ${trace_id} with the trace_id obtained in the previous steps. observer.log.xxx is a log file with a timestamp. Replace xxx with the actual timestamp.

8. Analyze the issue based on the information provided in the logs and the error messages.

   For more information about logs, see Log overview. If the information in the logs is unclear, contact technical support for further troubleshooting.

Troubleshoot the data source configuration

1. Identify the access link based on the data source.

2. The access link contains the following information:

   • The number and addresses of OBProxies

   • The number and addresses of OBServer hosts

3. Determine whether OBProxy is involved.

   • If yes, follow the steps below to troubleshoot the issue:

     1. Run the ssh command to log in to the corresponding OBProxy node.

     2. Run the cd command to go to the log directory of the OBProxy. The OBProxy logs are stored in the /log directory under its installation directory.

     3. Run the following command to filter out the error logs:

        grep "Error text" obproxy_error.log obproxy_error.log.xxx | grep "Error time point"
        

     4. Identify the OBServer host that the error SQL statement is routed to based on the filtered log entries.

     5. Run the ssh command to log in to the corresponding OBServer host for troubleshooting.

   • If no, follow the steps below to troubleshoot the issue:

     1. Run the ssh command to log in to the corresponding OBProxy node.

     2. Go to the log directory.

        Assume that the installation directory of OceanBase Database is /home/admin/oceanbase. The specific log path varies by actual environment.

        cd /home/admin/oceanbase/log
        

     3. Run the following command to filter out the error logs:

        grep "sending error" observer.log observer.log.xxx | grep "Error code"
        

     4. Obtain the trace_id from the filtered logs, and then filter the logs again based on the trace_id.

        grep "trace id" observer.log observer.log.xxx
        

     5. Analyze the issue based on the information provided in the logs and the related error messages.

        For more information about logs, see Overview. If the information in the logs is unclear, contact technical support for assistance.

Case studies

References

• Find logs of a specific SQL request

• Find logs of the last SQL request based on the Trace feature

• Log overview