arbitration_node_connection_with_observer_error Unable to connect observer on the arbitration node|V4.4.2|OceanBase Cloud Platform|OCP docs|Distributed Database

arbitration_node_connection_with_observer_error Unable to connect observer on the arbitration node

Last Updated：2026-06-12 02:01:01 Updated

Alert description

This alert indicates that the arbitration service node cannot connect to its corresponding observer.

Alert principle

The following table lists the key parameters involved in the monitoring logic of this alert.

Parameter	Value
Monitoring Metrics	N/A
Monitoring Expression	`{LOG_LEVEL=EDIAG,KWORD=arb active keepalive probe_connectivity,DURATION=0 SECONDS,ALERT_LEVEL=DOWN}`
Metric Collection	arbitration_log
Metric Source	Arbitration service logs, for example,`/home/admin/oceanbase/log/observer.log`.
Detection Cycle (ms)	500

Rule information

Monitoring Metrics	Default Threshold	Duration	Detection Cycle	Elimination Cycle
N/A	N/A	0 Seconds	The agent reads the file every 500 ms, and OCP generates an alert within one minute.	5 Minutes

Alert information

Alert Trigger Method	Alert Level	Scope
Log Matching Rules	Downtime	All Arbitration Services

Alert template

Alert Overview
- Template: ${alarm_target} ${alarm_name}
- Example: alarm_template_id=0:address=ip:port:host=ip:server_type:arbitration_observer:error_code=-1:keyword=arbactive keepalive probe_connectivity The Arbitration node cannot connect to the observer.
Alert details
- Template: Alert:[${alarm_name}] Arbitration service instance: ${address}, host: ${host}, log type: ${server_type}, log file: ${filename}, log level: ${log_level}, keyword=${keyword}, error code=${error_code}, log details=${error_message}.
- Example: Alert: [Arbitration Node Cannot Connect Observer] Arbitration Service Instance: xx.xx.xx.xx:2889, Host: xx.xx.xx.xx, Log Type: arbitration_observer, Log File: /home/admin/oceanbase/log/observer.log, Log Level: EDIAG, Keywords=arb active keepalive probe_connectivity, Error Code=4124, Log Details=[2026-04-24 11:05:15.681142] EDIAG [CLOG] probe_once_ (ob_log_active_keep_alive.cpp:276) [117128][TimerWK0_ArbServerTimer][C0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4124] active keepalive probe_connectivity blacklisted for new connections(tmp_ret=-4124, tmp_ret="OB_CONNECT_ERROR", dst="xx.xx.xx.xx:2889", in_blacklist=true) BACKTRACE:0x346b97db 0x3448c8a2 0x106b51b7 0x106b4c19 0x106b49f5 0x106b47e6 0x13717054 0x13714007 0x32d3517e 0x33d4110c 0x1379a3f0 0x3469f36f 0x346b37a6 0x150907d9d3fb 0x150907bf3e83
Alert recovery
- Template: Alert:[${alarm_name}] Arbitration service instance: ${address}, host: ${host}, log type: ${server_type}, log file: ${filename}, log level: ${log_level}, keyword=${keyword}, error code=${error_code}, log details=${error_message}.
- Example: Alert: [Arbitration Node Cannot Connect Observer] Arbitration Service Instance: xx.xx.xx.xx:2889, Host: xx.xx.xx.xx, Log Type: arbitration_observer, Log File: /home/admin/oceanbase/log/observer.log, Log Level: EDIAG, Keywords=arb active keepalive probe_connectivity, Error Code=4124, Log Details=[2026-04-24 11:05:15.681142] EDIAG [CLOG] probe_once_ (ob_log_active_keep_alive.cpp:276) [117128][TimerWK0_ArbServerTimer][C0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4124] active keepalive probe_connectivity blacklisted for new connections(tmp_ret=-4124, tmp_ret="OB_CONNECT_ERROR", dst="xx.xx.xx.xx:2889", in_blacklist=true) BACKTRACE:0x346b97db 0x3448c8a2 0x106b51b7 0x106b4c19 0x106b49f5 0x106b47e6 0x13717054 0x13714007 0x32d3517e 0x33d4110c 0x1379a3f0 0x3469f36f 0x346b37a6 0x150907d9d3fb 0x150907bf3e83

Impact on the system

The network isolation between the arbitration replica and followers may prevent election when the leader fails, compromising the high availability of 2F1A/4F1A configurations. A failure will directly impact system services.

Possible causes

Hardware failure.
Configuration faults of firewalls, iptables, and so on.
Router or gateway configuration is incorrect.

Solution

Identify the faulty object and the failed target. Based on the address, host, error_code in the alert details, and the dst="ip:port" in the logs, identify the specific arbitration service instance with a fault and the observer node that cannot be connected. If in_blacklist=true appears in the logs, it indicates that this observer has been marked by the arbitration side as a target for "new connection failure". You need to focus your troubleshooting on this target node and its link.
```
grep -n "arb active keepalive probe_connectivity" /home/admin/oceanbase/log/observer.log*
```
Check the target observer process and listening port. Log in to the target observer node specified in the alert, confirm that the observer process is running normally, and verify that the target port is in a listening state in the logs. If the observer has exited, is stuck, or the port is not listening, restore the observer service first.
```
ps -ef | grep observer
ss -lntp | grep <target port in the alert log>
```
Verify the network connectivity between the arbitration node and the observer node. On the arbitration node, perform a connectivity test on the dst address and port specified in the log to identify the cause of the connection failure, such as host unreachability, port unreachability, or link jitter. If ping is successful but TCP connection fails, it usually indicates issues with the firewall, security group, port listening, or routing policy.
```
ping <observer_ip>
nc -vz <observer_ip> <observer_port>
traceroute <observer_ip>
```
Check the host and network configurations. Focus on verifying the firewall, iptables/nftables, switch ACLs, security groups, routing, and gateway configurations between the arbitration node and the observer node to confirm whether there is any interception when the arbitration node accesses the observer's RPC port. Also, check whether the bound address or RPC port configuration of the target observer node has changed, to avoid situations where the service is started but does not listen on the target address or port as indicated in the alert.
Verify that the alert is resolved after recovery. After recovery, re-execute the TCP connectivity test on the arbitration node and continuously monitor the arbitration service logs to confirm that the keywords "arb active keepalive probe_connectivity failed" or "blacklisted for new connections" no longer appear. If the log level permits, successful logs such as "probe connect ok" should be visible after the alert resolves.

If the error persists after 1 to 2 minutes, continue collecting the observer.log files from both nodes, network policy configurations, and link probing results for further analysis.
```
grep -E "probe connect ok|probe_connectivity" /home/admin/oceanbase/log/observer.log*
```
Seek further assistance if necessary. If the observer service is normal, port listening is normal, and TCP probing from the arbitration node to the observer node still fails, it is recommended to collect the observer.log files, system logs, network policies, and link probing results from both ends at the time of the failure. Contact the network or system administrator to further investigate potential issues such as hardware faults, switch/gateway abnormalities, or cross-subnet policy blocking.

OceanBase

Customer Stories

Documentation