For the physical connections of a client, OceanBase Database Proxy (ODP) maintains its connections to multiple OBServer nodes at the backend. It also adopts the incremental synchronization solution based on version numbers to maintain the session status of connections to each OBServer node. This ensures efficient client access to each OBServer node. ODP also helps keep connections alive. When an OBServer node is down, being upgraded, or being restarted, the connection between the relevant client and ODP is not terminated. In addition, ODP rapidly switches to a normal OBServer node. This process is transparent to applications.
Create a connection
ODP sessions are classified into client sessions and server sessions. A client session is a connection established between a client and ODP. A server session is a connection established between ODP and an OBServer node. When ODP forwards a client request to an OBServer node, if no connection is established between ODP and the OBServer node, a session instance needs to be initialized.
Authentication is required when a session is being established. ODP cannot decide which OBServer node this session will access. Therefore, ODP selects an arbitrary OBServer node for authentication. ODP forwards the authentication packet sent from the client to the selected OBServer node and forwards the result returned by the OBServer node to the client. ODP also caches the authentication packet of the client within the client session. This way, when ODP establishes a server session that is associated with this client session to another OBServer node, ODP can forward the authentication packet to the target OBServer node for authentication.
Store a connection
When a client sends a request to ODP, ODP queries a client session based on the client connection information. In non-connection pool mode, a server session cannot independently exist without being associated with a client session. You can query the server sessions that are associated with a client session or check whether a server session is associated with a client session.
When ODP receives an SQL request from a client, ODP queries the partition table cache for the IP address of the OBServer node, and then checks whether a server session saved in the client session is associated with this OBServer node. If yes, the server session is used. If not, ODP establishes a connection to this OBServer node and adds the server session to the server session storage structure of the client session.
Maintain the transaction status
In the default configuration or non-distributed transaction mode, ODP binds client sessions to server sessions based on transactions. When the first statement of a transaction arrives at ODP, ODP selects a server session and binds it to the client session. All subsequent requests in the transaction are forwarded through this server session to the OBServer node. Switching the server session is not allowed. Therefore, ODP records the transaction status in the client session to maintain the binding.
In non-distributed transaction mode, the main purpose of transaction status maintenance is to ensure that the server session is not switched during the transaction. Specifically:
If an OBServer node goes down during a transaction, ODP needs to detect the node status and terminate the unfinished transaction. Because the MySQL protocol has no timeout mechanism, ODP must proactively notify the client that the transaction is terminated.
OBServer nodes do not implement transaction synchronization, and transaction migration is not supported. A transaction can only be processed by the initial OBServer node. Switching nodes will cause failure. Therefore, during primary/standby switchover, ODP must ensure that ongoing transactions are completed or return a special error to notify the client to terminate the transaction.
Another purpose is during ODP upgrade: before killing the old ODP to stop services, ODP must wait for all transactions to complete to minimize the impact on users.
If distributed transaction routing is enabled (parameters enable_ob_protocol_v2=true and enable_transaction_internal_routing=true), switching the server session within a transaction is allowed. In this mode, ODP dynamically selects the optimal OBServer node based on the partition locations involved in the SQL, enabling cross-node routing within a transaction. In this case, the purpose of transaction status maintenance shifts to ensuring routing consistency and fault recovery: ODP still needs to record transaction status to handle node failures or primary/standby switchover, but since cross-node routing is supported, transaction migration restrictions are partially relaxed, and ODP can coordinate multi-node transactions through internal mechanisms.
When a transaction starts, the transaction start status is recorded in the client session. When the transaction ends, the transaction status is reset. The criteria for determining transaction end are:
If autocommit is used, the return packet from the OBServer node must be parsed for each request, and the transaction status is reset immediately after data forwarding is complete.
For long transactions, the return packet is parsed only after a Commit or Rollback request to determine whether the transaction has ended.
Manage connection variables
A client session records all variables that are set by the client in this session. Each time a variable is modified, the modification time is recorded in the client session. When ODP is selecting a server session to forward requests, ODP first checks whether the modification time of variables in this server session is later than that recorded in the client session. If not, this server session does not use the latest session variable settings. In this case, ODP first resets all session variables in the server session. Then, ODP applies the session variable settings recorded in the current client session to the server session and forwards requests by using the server session. If the modification time of variables in the server session is later than that recorded in the client session, ODP directly forwards requests by using the server sessions.
The client may frequently set commonly used session variables, such as autocommit. Therefore, ODP cannot decide whether to reset multiple session variables at a time only based on the modification time of the variables. Instead, each server session stores the values of commonly used session variables. ODP checks whether the variable values in the client session and those in the server session are consistent for each request, and resets the variable values if they are inconsistent.
Avoid connection interruption
Connection interruption avoidance ensures that server session exceptions between ODP and OBServer nodes are imperceptible to clients, client sessions are normal, and clients can properly read and write data. The following part describes the scenarios where server session exceptions must be handled:
ODP does not obtain the new leader during the follower-to-leader switchover. Before the follower-to-leader switchover is performed, make sure that the ongoing transaction is ended. ODP only forwards a request to the OBServer node where the data is located. After the leader OBServer node is switched, the leader OBServer node needs to process the request and returns the result to ODP even if the data is not on the leader OBServer node.
The OBServer node is down. The connection between ODP and the OBServer node is closed. If the current server session is processing a transaction, ODP needs to send an error response to the client. If the current server session is idle, ODP needs only to disassociate the server session from the associated client session. Then, new requests are no longer forwarded by using this server session.
The communication between ODP and the OBServer node times out. The MySQL protocol does not have a timeout mechanism, but OBServer nodes do. Therefore, the OBServer node first notifies ODP of the timeout error, and ODP then notifies the client of the error. If the OBServer node notices that the server session has been idle for a long period of time, the OBServer node kills this server session. In this case, the handling strategy for the second scenario applies as well.
The network connection between ODP and the OBServer node is closed, or the network is partitioned. In this case, the handling strategy for the second scenario applies as well. If the server session is processing a transaction when network partitioning occurs, the OBServer node terminates this server session after a period of time. Then, ODP reports an error to the client and ends the uncompleted transaction. This avoids longtime occupancy of the session.
After an ODP upgrade is performed, the new ODP is responsible for new sessions initiated by the clients, and the number of sessions on the original ODP is decreased. If the number of sessions on the original ODP is smaller than the specified threshold, the sessions will be terminated, and the original ODP will be stopped. The sessions will be terminated after all ongoing transactions are completed. Long-running transactions may consume a long period of time. Transactions that time out must be forcibly killed. During this process, client connection interruption is unavoidable.
The current ODP is down. In this case, all connections to the current ODP are interrupted. The current ODP may be restarted within a short period of time, or the connections will be taken over by another ODP. Connection interruption is also unavoidable during the process.
If you use ODP, you can avoid connection interruption in most cases and minimize the impact on clients. Normal client connections can be kept during backend maintenance operations, such as the follower-to-leader switchover of OBServer nodes, primary/standby cluster switchover, and ODP upgrade.
Kill sessions
MySQL has no timeout mechanism, but it kills a session that has been idle for a long period of time. If a client does not receive a response from the OBServer node, the client keeps waiting. In this case, a DBA will intervene in the process and call the kill session command of MySQL to close the session.
OBServer nodes use a timeout mechanism. Generally, ODP can notify the client session of a transaction processing failure in the event of server downtime, network partitioning, and ODP disconnection from the OBServer node. If a client session is suspended because a long timeout period is specified or ODP misses specific objects to be processed, the client will send a request to kill the session. To kill a session, ODP needs to process the recorded client session status and notify the OBServer node to close the server session.