Description
This alert is triggered when the arbitration service is unavailable.
Principle
| Parameter | Value |
|---|---|
| Metric | arbitration_service_available |
| Source | OCP-Agent checks whether the local observer process is started and whether the arbitration service port is occupied. |
| Collected metric | arbitration_service_available |
| Metric expression | sum(arbitration_service_available{@LABELS}) by (@GBLABELS) |
| Collection cycle | 30 seconds |
Alert rule
| Metric expression | Default threshold | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| arbitration_service_available | 0 | 0 seconds | 30 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the metric | Critical | Arbitration service |
Alert templates
- Overview
- Template: ${alarm_target} ${alarm_name}
- Example: svr_ip=xxx.xxx.xxx.xxx:svr_port=2882 Arbitration service unavailable
- Details
- Template: Cluster: The arbitration service ${svr_ip}:${svr_port} is unavailable.
- Example: Cluster: The arbitration service xxx.xxx.xxx.xxx:2882 is unavailable.
Impact on the system
If the arbitration service is unavailable, service reliability and high availability of the tenant are affected. If a tenant replica fails, the tenant cannot normally provide the arbitration service.
Possible causes
- The server hosting the arbitration service is down.
- The port of the arbitration service cannot be accessed externally because the firewall is enabled.
Solutions
If the observer process of the server hosting the arbitration service unexpectedly exits, reload the observer process. You can execute the alert clearance plan to handle the alerted issue. For more information, see Execute the alert clearance plan.
Notice
You can reload the observer process only once within 12 hours since the event occurs.
Check whether the server hosting the arbitration service runs normally. If the server fails, restart the server and restart the arbitration service in the OceanBase Cloud Platform (OCP) console.
If the server is running normally, check whether the arbitration service process exists, whether the arbitration service port is listening normally, and whether the firewall prohibits access from other IP addresses.
