Alert description
This alert indicates that the RPC Request P99 Latency of the LogService node has exceeded the threshold, posing a risk of RPC request timeouts.
Alert principle
The following table lists the key parameters involved in the monitoring logic of this alert.
Parameter |
Value |
|---|---|
| Monitoring Metrics | logservice_rpc_requestP99_duration_ms: The P99 duration of RPC requests for the corresponding node of the logservice in milliseconds. |
| Monitoring Expression | 1000 * histogram_quantile(0.99, sum(rate(rpc_request_duration_seconds_bucket{@LABELS}[@INTERVAL])) by (@GBLABELS,le)) |
| Metric Collection | rpc_request_duration_seconds_bucket |
| Metric Source | OCP monagent collects metrics from http://127.0.0.1:${monagent.logservice.exporter.port}/metrics using prometheusInput. |
| Collection Cycle | 60 Seconds |
Rule information
Monitoring Metrics |
Default Threshold |
Duration |
Detection Cycle |
Elimination Cycle |
|---|---|---|---|---|
| logservice_rpc_requestP99_duration_ms | 1000 (ms) | 60 Seconds | 30 Seconds | 5 Minutes |
Alert expression: logservice_rpc_requestP99_duration_ms >= 1000
Alert information
Alert Trigger Method |
Alert Level |
Scope |
|---|---|---|
| Based on monitoring metric expression | Warning | Host (LogService) |
Alert template
Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: logservice_cluster=my_ls_cluster:host=11.124.9.47 oblogservice corresponding node RPC request timed out
Alert details
- Template: Cluster: ${logservice_cluster}, Host: ${host}, Corresponding Node RPC Request Timeout: ${value_shown} ms, Exceeds Threshold ${alarm_threshold} ms
- Example: Cluster: my_ls_cluster, Host: 11.124.9.47, Corresponding Node RPC Request Timeout: 1500 milliseconds, Exceeds Threshold of 1000 milliseconds
Alert recovery
- After the RPC P99 latency drops below the threshold, it is automatically removed after the elimination period (5 minutes) is met.
Impact on the system
Excessively high RPC request latency can slow down operations such as log reading/writing and replica synchronization, leading to client request backlogs and, in severe cases, timeout failures.
Possible causes
- Node CPU, disk I/O, or network resource bottlenecks
- Thread pool task backlog, RPC processing queue
- Increased disk pwrite/pread latency
- Network jitter between nodes in the cluster
Solution
- View the node's CPU, memory, disk I/O, RPC latency, and other related metrics in OCP Monitoring.
- Check whether the values of logservice_rpc_inflight_requests and thread pool queue depth are abnormal.
- Troubleshoot disk and network bottlenecks
- If only a single node is abnormal, consider migrating traffic or restarting that node.