Troubleshoot application disconnection issues

During database operation, various exceptions may occur, such as application errors, database connection errors, database permission issues, database resource issues, and network issues. Application disconnection is one such scenario.

To help you quickly identify the root cause of this issue and resolve it efficiently, we have summarized a clear and practical troubleshooting process for application disconnection issues. This process provides specific steps to improve the efficiency of issue resolution, minimize the impact on business operations, and provide strong support for daily operations and maintenance.

Procedure overview

When an application disconnection occurs, follow this procedure to troubleshoot the issue.

First, determine the programming language used by the application.

Java

If you confirm that the application is developed in Java, you can check the error logs of the Java application to determine the reason for the disconnection. Here's how:

1. Check if the error log contains the Communications link failure exception.

   • If the error log does not contain the Communications link failure exception, you can try to reproduce the scenario:

     • If you can reproduce the scenario, you can analyze the cause of the error in the program code by debugging the program code or analyzing the network packets.
     • If you cannot reproduce the scenario, you can try to infer the cause of the error based on the existing information and the logic of the program.

   • If the error log contains the Communications link failure exception, proceed to the next step.

2. Check the information corresponding to the Caused by section of the exception log.

   • If this section is not printed, the stack trace is incomplete, and the cause cannot be determined. You need to reproduce the issue after the application prints the complete stack trace and then analyze it.

   • If this section is printed, you can determine the cause of the disconnection based on the output information in the Caused by section:

     • If the content is Connection reset"、"can not read response from server. Expected to read 4 bytes,read 0 bytes before connection was unexpectedly lost or unexpected end of stream, read XXX bytes from xxx (socket was closed by server), the connection from the application side was disconnected. You need to determine the cause based on the error message. For more information, see Application-side disconnection in this topic.

     • If the content is read timed out" or "Connection timed out (Read failed), the slow SQL triggered the socketTimeout time set in the application, causing the program to disconnect. In this case, you need to determine whether the exception SQL is a slow SQL. If it is, you can optimize the SQL as needed. If there are no slow SQLs, you can consider adjusting the socketTimeout. For more information, see Database connection pool configuration.

     • If the content is connect timed out or Connection timed out: connect, the TCP handshake failed. The destination address and port of the application are unreachable, which may be due to network issues or incorrect IP and port configurations in the program.

     • If the content is Connection refused, the connection failed. The destination address and port of the application are reachable, but the connection was rejected due to some system or network issue.

C

If you confirm that the application is developed in C, you can check the error logs of the C application to determine the reason for the disconnection.

• If the error message is end-of-file on communication channel or Lost connection to MySQL server during query, the connection from the application side was disconnected. You need to determine the cause based on the error message. For more information, see Application-side disconnection in this topic.

• If the error message is reading initial communication packet or reading authorization packet, the connection to the database failed. The failure may be due to network issues or issues with the OceanBase Database service. You need to analyze the logs, and packet capture may be necessary.

• If the application logs do not contain obvious error messages, you can try to reproduce the scenario.

  • If you can reproduce the scenario, you can analyze the cause of the error in the program code by debugging the program code or analyzing the network packets.
  • If you cannot reproduce the scenario, you need to try to infer the cause of the error based on the existing information and the logic of the program.

Application-side disconnection

When the application-side connection is disconnected, check the logs for a conn id identifier or the SQL statement executed when the exception occurred. If neither is present, critical information is missing and you need to contact the application team for more details. If either piece of information is present, identify the data source configuration in the application and proceed as follows:

1. Identify the access path based on the data source. The access path includes the number and addresses of the following components:

   • OBProxy instances and their addresses

   • OBServer hosts and their addresses

2. Determine the current access path and check whether the application connects through OBProxy.

   • The application connects to the database through OBProxy

     1. Use the ssh command to log in to the corresponding OBProxy node.

     2. Use the ps command to check when the OBProxy process started.

     3. Check whether OBProxy restarted during the exception period.

        If OBProxy restarted, the disconnection was likely caused by the OBProxy restart.

        If OBProxy did not restart, proceed to the next step.

     4. If OBProxy did not restart, use the cd command to go to the OBProxy log directory. OBProxy logs are stored in the /log directory under the OBProxy installation directory.

     5. Check whether the logs contain a conn_id. If not, check whether they contain SQL text related to the exception. If neither is present, contact the application team for more information.

        • If a conn_id is present, run the following command to filter OBProxy logs from the exception period:

          grep "xxxxxx" obproxy.log
          

          Replace xxxxx with the actual conn_id.

        • If SQL text related to the exception is present, substitute part of the SQL text into the following commands to filter OBProxy logs from the exception period:

          grep "error text" obproxy.log.xxxx | grep "time of error"
          

          grep "error text" obproxy_error.log.xxxx | grep "time of error"
          

     6. From the filtered log entries, find the OBProxy trace_id from the exception period.

     7. Substitute the trace_id into the following command to filter OBProxy logs from the exception period:

        grep "xxxxxx" obproxy.log.xxxx | grep "time of error"
        

        Replace xxxxx with the actual trace_id.

     8. Based on the OBProxy log output, determine how the connection was disconnected.

        • If OBProxy actively disconnected the connection, analyze the cause from the filtered logs. If necessary, review the OBProxy source code.

        • If the connection was terminated by OBServer, which caused the application disconnection, find the connected OBServer node and the OBServer session_id from the OBProxy log entry, then follow Use SSH to directly log in to the corresponding OBServer node below for further diagnosis.

        • If a network device between the application server and OBProxy (such as F5, SLB, or other network devices) disconnected the link, troubleshoot the network devices between the application and OBProxy.

   • Use SSH to directly log in to the corresponding OBServer node

     1. Use the ssh command to log in to the corresponding OBServer node.

     2. Use the ps command to check when the OBServer process started.

     3. Check whether OBServer restarted during the exception period.

        If OBServer restarted, the disconnection was likely caused by the OBServer restart.

        If OBServer did not restart, proceed to the next step.

     4. If OBServer did not restart, use the cd command to go to the log directory.

        The following example assumes that OceanBase Database is installed in /home/admin/oceanbase. Use the actual log path in your environment.

        cd /home/admin/oceanbase/log
        

     5. Run the following commands to filter logs related to session id:

        grep "observer session id" observer.log 
        

        grep "observer session id" observer.log.xxx 
        

     6. Based on the filtered logs, determine the connection status.

        • If OBServer actively disconnected the connection, analyze the cause from the filtered logs. If necessary, review the OBServer source code.
        • If a network device between OBProxy and OBServer disconnected the link, troubleshoot the network devices between OBProxy and OBServer.

Case studies