OceanBase Database Proxy (ODP, also known as OBProxy) uses the blocklist mechanism to implement self-adaptive access control for operations such as off-peak major compaction, upgrade, leader switchover, downtime, startup, and shutdown on OBServer nodes.
Background
ODP supports two types of blocklists: down blocklist and live-but-unavailable blocklist. You can manage blocklists by setting the corresponding parameters.
You can use a blocklist to implement access control in the following scenarios:
Ensure that a zone involved in an off-peak major compaction cannot be accessed.
Ensure that an OBServer node or a zone in the cluster that is undergoing an upgrade cannot be accessed.
Ensure that a failed OBServer node cannot be accessed.
Ensure that, after a partition migration, requests cannot be sent to the original OBServer node of the partition.
Ensure that, when an OBServer node to be accessed does not have tenant resources, ODP retries accessing other OBServer nodes.
Ensure that, when an OBServer node becomes alive but unavailable (for example, when the memory usage exceeds the specified threshold, the timeout period elapses, the OBServer node is initialized, or the observer process exits), ODP retries accessing other OBServer nodes.
Down blocklist
If an OBServer node is detected to be down or has suspended service provision after you execute the stop server command, the OBServer node is added to the down blocklist. When this OBServer node is considered normal, it is removed from the blocklist. Applications do not access OBServer nodes in the blocklist.
An OBServer node is added to the down blocklist in one of the following cases:
The value of the
start_service_timeparameter is 0.The value of the
statusparameter isinactive.
Alive-but-unavailable blocklist
The status of an OBServer node in a live-but-unavailable blocklist is temporary or uncertain. This differentiates it from a down blocklist. The system retries accessing it periodically. It is removed from the blocklist after it is found to be normal again.
An OBServer node is added to the live-but-unavailable blocklist in one of the following cases:
The local zone is undergoing a major compaction or an upgrade.
The OBServer node is undergoing a major compaction or an upgrade.
When an OBServer node is alive, errors that occur are returned in error packages. The alive-but-unavailable count of an OBServer node increases by 1 if the OBServer node returns a system error in one of the following cases: the OBServer node reaches the resource limit, connection limit, or memory limit, encounters a packet verification error, a response timeout error, a client connection timeout error, or an ODP timeout error, is being initialized, or the observer process is exiting. If the alive-but-unavailable count of an OBServer node reaches M within N seconds, this OBServer node is added to the alive-but-unavailable blocklist.
After an off-peak major compaction begins, Root Service first updates internal tables to set the major compaction status of the relevant zone, and then switches the leaders from this zone to other zones. ODP periodically updates the zone status table to set the zone status. After the status of the relevant zone is updated, all OBServer nodes in this zone are set to the alive-but-unavailable state. During an off-peak major compaction, all OBServer nodes in a zone that is undergoing a major compaction are in the alive-but-unavailable blocklist.
Blocklist priorities
The down blocklist has a higher priority than the alive-but-unavailable blocklist. The priority of a major compaction or an upgrade of a zone is higher than that of an OBServer node.
Force access OBServer nodes in the blocklist
When no OBServer nodes are available, the system force accesses OBServer nodes in the blocklist.
Blocklist parameters
Log on to the
systenant as the root user from a client.Execute the following statement to view the configuration parameters of ODP:
obclient> SHOW PROXYCONFIG;To modify a parameter, use the
ALTER proxyconfig SET key=valuestatement. The modification takes effect immediately.The following table describes the configuration parameters that are related to blocklists.
Parameter Description enable_congestion Specifies whether the blocklist mechanism is enabled. By default, the blocklist mechanism is enabled. When the blocklist mechanism malfunctions, you can temporarily disable it by modifying this parameter so that ODP can properly work. congestion_failure_threshold Specifies the number of failures within N seconds after which an OBServer node is added to the blocklist. If the value of this parameter is smaller than 0, all OBServer nodes are removed from the blocklist. congestion_fail_window The interval at which the number of errors is reset to zero. Unit: seconds. The default value of this parameter is 120 seconds. If, within this cycle, the number of errors exceeds the value specified by congestion_failure_threshold, the OBServer node is added to the blocklist. You can modify the value of this parameter in real time. After you modify this parameter, the current error count is reset to zero in the new fail window.congestion_retry_interval The interval at which ODP retries to access an alive-but-unavailable OBServer node in the blocklist. Default value: 20. Unit: seconds. min_keep_congestion_interval The period after which an OBServer node is removed from the blocklist. min_congested_connect_timeout The timeout period of client connections. When a connection times out, the OBServer node is added to the alive-but-unavailable blocklist. Default value: 100. Unit: ms.
Query blocklists
Log on to the
systenant as therootuser from a client.Execute the following statement to view the blocklist information.
Sample statement:
SHOW PROXYCONGESTION [all] [clustername];Parameters:
If no option is specified, the blocklists of all clusters are retrieved.
If only the
alloption is specified, OBServer nodes of all clusters, including OBServer nodes in and those not in the blocklists, are retrieved.If only the
clusternameoption is specified, the blocklists of the specified cluster are retrieved.If both the
allandclusternameoptions are specified, the OBServer nodes of the specified cluster are retrieved.
Sample code:
obclient> show proxyinfo binary\G *************************** 1. row *************************** name: binary info info: ObProxy-OceanBase 1.1.2-459.el6 version:RELEASE_6U MD5 REVISION:459-local-5ba2e3d33ea201fb675eb704627162a584485cdd BUILD_TIME:Sep 29 2016 11:50:15 BUILD_FLAGS:-D__STDC_LIMIT_MACROS -D_OB_VERSION=1000 -D__STDC_CONSTANT_MACROS -D_NO_EXCEPTION -g -Wall -Werror -Wextra -Wunused-parameter -Wformat -Wconversion -Wno-invalid-offsetof -Wno-deprecated -fno-strict-aliasing -fno-omit-frame-pointer -mtune=generic -Wno-psabi -Wno-sign-compare -DGCC_52 -DUSE_POSIX_FALLOCATE -DSUPPORT_SSE4_2 -DHAVE_SCHED_GETCPU -DHAVE_REALTIME_COARSE -DOB_HAVE_EVENTFD -DHAVE_FALLOCATE -DHAVE_MINIDUMP 1 row in set obclient> show proxycongestion; *************************** 1. row *************************** cluster_name: ob1.jianhua.sjh zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13201 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 *************************** 2. row *************************** cluster_name: ob1.jianhua.sjh zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13203 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 2 rows in set obclient> show proxycongestion all\G *************************** 1. row *************************** cluster_name: MetaDataBase zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13201 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 *************************** 2. row *************************** cluster_name: MetaDataBase zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13203 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 *************************** 3. row *************************** cluster_name: ob1.jianhua.sjh zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13201 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 *************************** 4. row *************************** cluster_name: ob1.jianhua.sjh zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13203 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 4 rows in set obclient> show proxycongestion all "ob1.jianhua.sjh"\G *************************** 1. row *************************** cluster_name: ob1.jianhua.sjh zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13201 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 *************************** 2. row *************************** cluster_name: ob1.jianhua.sjh zone_name: zone1 zone_state: ACTIVE server_ip: xx.xx.xx.xx:13203 server_state: ACTIVE alive_congested: 0 last_alive_congested: 0 dead_congested: 0 last_dead_congested: 0 stat_alive_failures: 0 stat_conn_failures: 0 conn_last_fail_time: 0 conn_failure_events: 0 alive_last_fail_time: 0 alive_failure_events: 0 ref_count: 1 2 rows in set
Note
When
dead_congestedoralive_congestedis set to0, the OBServer node has been added to the blocklist.clusternamedoes not need to be enclosed in single or double quotation marks in versions earlier than ODP V1.1.2. It must be enclosed in single or double quotation marks in ODP V1.1.2 and later.