Description
This alert is triggered when the number of exporters on the host is insufficient.
OceanBase Cloud Platform (OCP) V3.1.1 provides six types of exporters, and they return different types of data. Three types of exporters are used to return the monitoring data of OBServers, two types of exporters are used to return the monitoring data of OBProxy servers, and one type of exporters is used to return the monitoring data of the hosts. If both OBServer and OBProxy are deployed on the same host, six exporters are expected to exist on the host. When it is detected that less than six exporters are active, the alert is triggered.
Principle
OCP-Server checks the exporter status once every minute. When the number of active exporters does not meet the expectation and the state of insufficient exporters persists for more than 3 minutes, the alert is triggered.
Alert rule
| Metric | Default threshold | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| None | None | 0 minutes | 60 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Timed task of OCP | Stopped | Server |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: Host: ${host}, Alert: ${alarm_name}. During the deployment of [${host_services}], the number of required exporters is ${expected_exporter_count}, but that of actual exporters is ${actual_exporter_count}.
Overview example: service=OCP:svr_ip=xxx.xxx.xxx.xxx. The number of exporters is insufficient.
Details example: Host: xxx.xxx.xxx.xxx, Alert: The number of exporters is insufficient. During the deployment of [OB,ObProxy], the number of required exporters is 6, but that of actual exporters is 3.
Impact on the system
OCP cannot collect metrics of a faulty host. The following issues may occur:
The data is fully collected for a metric.
The data is partially collected for a metric.
No data is collected for a metric.
Some metrics are used for alert detection. If alert-related metrics are unavailable, the corresponding alert feature is unavailable.
Possible cause
OCP-Agent failed to start.
Suggested solutions
Run the
select * from ocp_exporter_addresscommand in the faulty MetaDB of OCP, and view hosts that do not have sufficient exporters.Find the record in which the value of the status field is inactive. The IP address of the instance field is the IP address of the target host.
Log on to that host and restart OCP-Agent.
If the restart does not solve the issue, perform the following steps:
Go to the host details page in the OCP console. Click Download Logs to download the OCP-Agent log and identify the cause.
Alternatively, find the target host in the OCP console and click its IP address. On the OCP-Agent tab, check the related processes such as pos_proxy, node_exporter, and ocp_agentd for exceptions. If an exception is found, click Download Logs to identify the cause.
