Blocklist mechanism

OceanBase Database Proxy (ODP, also known as OBProxy) uses the blocklist mechanism to implement self-adaptive access control for operations such as off-peak major compaction, upgrade, leader switchover, downtime, startup, and shutdown on OBServer nodes.

Background information

ODP supports three types of blocklists: status blocklist, detection blocklist, and alive-but-unavailable blocklist. You can manage blocklists by setting the corresponding parameters.

You can use a blocklist to implement access control in the following scenarios:

• Ensure that a zone involved in an off-peak major compaction cannot be accessed.

• Ensure that an OBServer node or a zone in the cluster that is undergoing an upgrade cannot be accessed.

• Ensure that a failed OBServer node cannot be accessed.

• Ensure that, after a partition migration, requests cannot be sent to the original OBServer node of the partition.

• Ensure that, when an OBServer node to be accessed does not have tenant resources, ODP retries accessing other OBServer nodes.

• Ensure that, when an OBServer node becomes alive but unavailable (for example, when the memory usage exceeds the specified threshold, the timeout period elapses, the OBServer node is initialized, or the observer process exits), ODP retries accessing other OBServer nodes.

Status blocklist

The status blocklist depends on the status changes of OBServer nodes. Due to historical reasons, the DETECT_ALIVE and DETECT_DEAD states are not related to the status blocklist. The changes of other states can be obtained from views of OBServer nodes.

When ODP obtains the latest status of an OBServer node by using a scheduled task, ODP performs the corresponding operation based on the status of the OBServer node.

• ACTIVE: ODP removes the OBServer node from the status blocklist.

• INACTIVE/REPLAY: ODP adds the OBServer node to the status blocklist.

• DELETED/DELETING: ODP updates the OBServer list in the memory and no longer forwards SQL requests to this OBServer node.

• UPGRADE: ODP neither adds the OBServer node to the status blocklist nor forwards SQL requests to the OBServer node. This is equivalent to adding the OBServer node to the status blocklist.

Detection blocklist

For the status blocklist, ODP obtains information from the RootService of OceanBase Database. Such information sometimes may be unable to reflect the conditions between ODP and OBServer nodes. As shown in the following figure, ODP learns from the RootService that OBServer1 is in the ACTIVE state, but the network between ODP and OBServer1 is disconnected.

Therefore, ODP implements the detection blocklist based on the status blocklist. ODP sends a detection SQL statement to an OBServer node to determine its status. ODP sends a detection SQL statement select 'detect server alive' from dual to the sys tenant of the OBServer node and set the timeout period to 5s. If no result is returned within the timeout period, ODP sends a detection statement again. If the detection fails for three consecutive times, ODP sets the status of this OBServer node to DETECT_DEAD. If a result is returned, ODP clears the number of detection failures and sets the status of this OBServer node to DETECT_ALIVE. After ODP detects a status change, ODP performs the corresponding operation.

• DETECT_ALIVE: ODP removes the OBServer node from the detection blocklist.

• DETECT_DEAD: ODP adds the OBServer node to the detection blocklist and closes all connections to this OBServer node.

Alive-but-unavailable blocklist

In the alive-but-unavailable mechanism, ODP defines the status of an OBServer node based on the execution results of business SQL statements and adds the OBServer node to or removes it from the blocklist based on the status. ODP is more cautious about adding an OBServer node to or removing an OBServer node from the alive-but-unavailable blocklist than with the status blocklist and the detection blocklist. This aims to avoid misjudgments that are made based on a single SQL execution result. The operation performed by ODP on an OBServer node vary with the actual situation:

• ODP will record a failure event in any of the following cases:

  • ODP sends a detection SQL statement to the OBServer node and does not receive any response after the timeout period specified by the ob_query_timeout parameter expires.

  • The OBServer node returns the OB_SERVER_IS_INIT, OB_SERVER_IS_STOPPING, OB_PACKET_CHECKSUM_ERROR, or OB_ALLOCATE_MEMORY_FAILED error code.

  • ODP fails to connect to the OBServer node, parse packets, or transmit data.

• If the OBServer node has five alive-but-unavailable records within 120s, ODP will add it to the alive-but-unavailable blocklist. This behavior is controlled by the parameters congestion_fail_window (error statistics period) and congestion_failure_threshold (number of failures).

• ODP will attempt to resend a detection SQL statement to the OBServer node 20s later after it adds the OBServer node to the alive-but-unavailable blocklist. The retry interval is controlled by the parameter congestion_retry_interval.

• If the detection SQL statement is successfully executed, ODP will remove the OBServer node from the blocklist. This value is controlled by the parameter min_keep_congestion_interval.

There is no priority among the three types of blocklists mentioned above. If an OBServer node is in any of the blocklists, it will be unable to receive requests. In other words, the node will only receive requests when it is not in any blocklist.

When no OBServer nodes are available, the system force accesses OBServer nodes in the blocklist.

Blocklist parameters

1. Log on to the sys tenant as the root user from a client through ODP.

   obclient> obclient -h10.10.10.1 -uroot@sys#obdemo -P2883 -p -c
   

   In this example, the IP address of ODP is 10.10.10.1, the name of the cluster is obdemo, and the use connects to the cluster through OBClient. For detailed operations on connecting to the OceanBase tenant, see Connection methods (MySQL mode) and Connection methods (Oracle mode).

2. Execute the following statement to view the configuration parameters of ODP.

   obclient> SHOW PROXYCONFIG;
   

3. To modify a parameter, use the ALTER proxyconfig SET key=value statement. The modification takes effect immediately.

   The following table describes the configuration parameters that are related to blocklists.

   Parameter	Description
   enable_congestion	Specifies whether to enable the blocklist mechanism. By default, the blocklist mechanism is enabled. When the blocklist mechanism malfunctions, you can temporarily disable it by modifying this parameter so that ODP can properly work.
   congestion_failure_threshold	Specifies whether to add an OBServer node to the blocklist. This parameter is used with congestion_fail_window. If the number of errors from an OBServer node exceeds this value within the congestion_fail_window period, ODP will add the OBServer node to the blocklist. When this parameter’s value is less than 0, all servers in the blocklist will be removed.
   congestion_fail_window	The interval at which the number of errors is reset to zero. Unit: seconds. The default value of this parameter is 120 seconds. If, within this cycle, the number of errors exceeds the value specified by congestion_failure_threshold, the OBServer node is added to the blocklist. You can modify the value of this parameter in real time. Once the parameter is modified, the current error count will be reset, and will be recalculated based on the new configuration.
   congestion_retry_interval	The interval at which ODP retries to access an alive-but-unavailable OBServer node in the blocklist. Default value: 20. Unit: seconds.
   min_keep_congestion_interval	The period after which an OBServer node is removed from the blocklist.
   min_congested_connect_timeout	The timeout period of client connections. When a connection times out, the OBServer node is added to the alive-but-unavailable blocklist. Default value: 100. Unit: ms.

Query blocklists

1. Log on to the sys tenant as the root user from a client through ODP.

   obclient> obclient -h10.10.10.1 -uroot@sys#obdemo -P2883 -p -c
   

   In this example, the IP address of ODP is 10.10.10.1, the name of the cluster is obdemo, and the use connects to the cluster through OBClient. For detailed operations on connecting to the OceanBase tenant, see Connection methods (MySQL mode) and Connection methods (Oracle mode).

2. Execute the following statement to view the blocklist information.

   The syntax is as follows:

   SHOW PROXYCONGESTION [all] [clustername];
   

   The options in the command are described as follows:

   • If no option is specified, the blocklists of all clusters are retrieved.

   • If only the all option is specified, OBServer nodes of all clusters, including OBServer nodes in and those not in the blocklists, are retrieved.

   • If only the clustername option is specified, the blocklists of the specified cluster are retrieved.

   • If both the all and clustername options are specified, the OBServer nodes of the specified cluster are retrieved.

Examples

Here is an example of viewing the blocklist information for the single-node cluster obcluster.

obclient> show proxycongestion 'obcluster'\G

The output is as follows:

*************************** 1. row ***************************
         cluster_name: obcluster
            zone_name: zone1
          region_name: sys_region
           zone_state: ACTIVE
            server_ip: xxx.xxx.xxx.xxx:2881
           cr_version: 5
         server_state: ACTIVE
      alive_congested: 0
 last_alive_congested: 0
       dead_congested: 0
  last_dead_congested: 0
  stat_alive_failures: 0
   stat_conn_failures: 0
  conn_last_fail_time: 0
  conn_failure_events: 0
 alive_last_fail_time: 0
 alive_failure_events: 0
            ref_count: 2
     detect_congested: 0
last_detect_congested: 0