Application error -- the error message contains an OceanBase error code

During database operation, various exceptions may occur, such as application errors, database connection errors, database permission issues, database resource shortages, and network problems. One type of exception is application exceptions, where the error message contains an OceanBase error code.

To help you quickly identify the root cause of such exceptions and efficiently resolve them, this topic summarizes a clear and practical troubleshooting process for application exceptions with OceanBase error codes. This process provides specific steps to enhance troubleshooting efficiency and minimize business impact, supporting your daily O&M work.

Procedure

If an application fails to run and the error message contains an OceanBase error code, you can troubleshoot the issue by using the following procedure.

First, determine whether you can use OBClient to connect to the database and manually reproduce the issue:

• If yes, connect to the database after the issue is reproduced and troubleshoot the issue.

• If not, troubleshoot the data source information in the program configuration.

Troubleshoot the issue by reproducing it

1. Use OBClient to connect to the OceanBase cluster at port 2881 (the default SQL port). If you use a custom port, replace 2881 with the actual port number.

2. Manually execute the SQL statement that caused the error to reproduce the issue.

3. Execute the following statement to obtain the trace_id.

   Notice

   You must execute the following statement immediately after the SQL error is triggered; otherwise, the trace_id of the SQL statement that caused the error cannot be obtained.

   MySQL mode

   Oracle mode

   The following example shows how to obtain the trace_id in MySQL mode:

   obclient> SELECT last_trace_id();
   

   The following example shows how to obtain the trace_id in Oracle mode:

   obclient> SELECT last_trace_id() FROM DUAL;
   

4. Obtain the host information where the SQL statement is actually executed.

   Generally, an OceanBase cluster has multiple nodes. You can execute the following SQL statement to obtain the node where the SQL statement is actually executed, and then filter the logs.

   MySQL mode

   Oracle mode

   Execute the following statement in MySQL mode:

   obclient> SELECT * FROM oceanbase.GV$OB_SQL_AUDIT WHERE trace_id=last_trace_id;
   

   Replace last_trace_id with the trace_id obtained in the previous step.

   Execute the following statement in Oracle mode:

   obclient> SELECT * FROM SYS.GV$OB_SQL_AUDIT WHERE trace_id=last_trace_id;
   

   Replace last_trace_id with the trace_id obtained in the previous step.

   According to the query result of the GV$OB_SQL_AUDIT view, the host corresponding to the svr_ip column is the host where the SQL statement is actually executed.

5. Use the ssh command to log in to the host based on the obtained host information.

6. Go to the log directory.

   Assume that the installation directory of OceanBase Database is /home/admin/oceanbase. The following example shows how to go to the log directory. The timestamped log file name, observer.log.xxx, is used in the following examples. Replace xxx with the actual timestamp based on the time when the SQL statement caused the error during the reproduction.

   cd /home/admin/oceanbase/log
   

7. Run the following command to filter the required information from the logs.

   grep "${trace_id}" observer.log
   

   grep "${trace_id}" observer.log.xxx
   

   Replace ${trace_id} with the trace_id obtained in the previous steps; replace observer.log.xxx with the actual timestamped log file name.

8. Analyze the issue based on the information provided in the logs and the error message.

   For more information about logs, see Log overview. If the information in the logs is unclear, contact technical support for assistance.

Troubleshoot the data source configuration

1. Based on the data source, identify the access path.

2. The access path contains the following information:

   • The number and IP addresses of OBProxies

   • The number and IP addresses of OBServer hosts

3. Determine whether OBProxy is involved.

   • If yes, follow these steps to troubleshoot the issue:

     1. Run the ssh command to log in to the corresponding OBProxy node.

     2. Run the cd command to go to the log directory of OBProxy. The OBProxy logs are stored in the /log directory under its installation directory.

     3. Run the following command to filter out the error logs:

        grep "Error text" obproxy_error.log obproxy_error.log.xxx | grep "Error time"
        

     4. Identify the OBServer host that the error SQL statement is routed to based on the filtered log entries.

     5. Run the ssh command to log in to the corresponding OBServer host for troubleshooting.

   • If not, follow these steps to troubleshoot the issue:

     1. Run the ssh command to log in to the corresponding OBProxy node.

     2. Go to the log directory.

        Assume that the installation directory of OceanBase Database is /home/admin/oceanbase. The specific log path depends on your actual setup.

        cd /home/admin/oceanbase/log
        

     3. Run the following command to filter out the error logs:

        grep "sending error" observer.log observer.log.xxx | grep "Error code"
        

     4. Obtain the trace_id from the filtered logs and then filter the logs again using trace_id.

        grep "trace id" observer.log observer.log.xxx
        

     5. Analyze the issue based on the information provided in the logs and the error messages.

        For more information about logs, see Overview. If the logs are unclear, contact technical support for assistance.

Case studies

References

• Find logs of a specific SQL request

• Find logs of the last SQL request based on the Trace feature

• Log overview