Alert description
In multi-VPC mode, OCP needs to manage OceanBase clusters in multiple VPCs. OCP accesses the services in these VPCs through a cross-VPC secure channel established by IC-Server and IC-Agent. IC-Server provides a VIP for OCP and proxies its access to other VPCs.
This alert is triggered when IC-Server cannot be connected.
Alert principle
The following table describes the key parameters involved in the monitoring logic.
| Parameter | Value |
|---|---|
| Monitoring metric | ic_server_health The value of this metric indicates the result of the health check interface. A value of 0 indicates that the connection is unavailable and the alert is triggered. |
| Metric source | The URL of the health check interface for accessing IC-Server is http:// **IP** :8088/ok, where IP is the IP address of the host monitored by the alert. |
| Metric collection | ic_server_health |
| Monitoring expression | min(ic_server_health{@LABELS}) by (@GBLABELS) |
| Collection interval | 1 minute |
Rule Information
| Monitoring metric | Default threshold | Duration | Alert cycle | Elimination cycle |
|---|---|---|---|---|
| ic_server_health | 0 | 0 seconds | 15 seconds | 5 minutes |
Alert Information
| Alert trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the monitoring metric | Severe | Server |
Alert Template
Alert Overview
- Template: The OCP node ${ocp_address} cannot connect to the IC-Server ${ic_server_address}.
- Example: The OCP node xxx.xxx.xxx.1 cannot connect to the IC-Server xxx.xxx.xxx.2.
Alert Details
- Template: The OCP node ${ocp_address} cannot connect to the IC-Server ${ic_server_address}.
- Example: The OCP node xxx.xxx.xxx.1 cannot connect to the IC-Server xxx.xxx.xxx.2.
Alert Recovery
- Template: Alert: ${alarm_name}, Is the Inter Connector Server healthy (can be normally accessed): ${recover_value}.
- Example: Alert: Server connection failed, Is the Inter Connector Server healthy (can be normally accessed): 1.
Impact on the System
The OCP system cannot operate or manage OceanBase clusters and related resources in other VPCs.
Possible Causes
The host that installed the IC-Server failed.
Network issues preventing OCP from establishing connections with some IC-Servers.
Solution
Log in to the host where IC-Server is installed and check if the IC-Server process is normal and if there are any error logs.
Log in to the IC-Server host. If login to the host fails:
Check the host failure reason and make sure the host can be logged in normally.
If login succeeds, perform the next step.
Run the following command to check whether the ic-server process is normal.
ps -ef|grep ic-serverIf the process is not started, run the
supervisord restart ic-servercommand to restart the process.If the process is already started, proceed to the next step.
Run the following command to check whether there are any error logs in
~/logs/ic-server/server.log.grep -r "error" ~/logs/ic-server/server.log~/logs/ic-server/server.log"~" needs to be replaced with the actual path based on the deployment status of IC-Server.If there are error logs, perform troubleshooting based on the error logs.
If there are no error logs, proceed to the next step.
Confirm whether the network issue is preventing OCP from connecting with some IC-Servers. For more information, see Network troubleshooting.
If the above solution fails to resolve the issue, contact Technical Support.