Alert description
This alert indicates that the end SCN of committed logs in LogService is delayed by more than the threshold, indicating a lag in replica log synchronization.
Alert principle
The following table lists the key parameters involved in the monitoring logic of this alert.
Parameter |
Value |
|---|---|
| Monitoring Metrics | logservice_replica_committed_end_scn_delay: The delay (in milliseconds) between the end SCN of committed logs and the current time in the log service. |
| Monitoring Expression | avg(replica_committed_end_scn_delay{@LABELS}) by (@GBLABELS) / 1000 |
| Metric Collection | replica_committed_end_scn_delay |
| Metric Source | OCP monagent collects metrics from http://127.0.0.1:${monagent.logservice.exporter.port}/metrics using prometheusInput. |
| Collection Cycle | 60 Seconds |
Rule information
Monitoring Metrics |
Default Threshold |
Duration |
Detection Cycle |
Elimination Cycle |
Alert Level |
|---|---|---|---|---|---|
| logservice_replica_committed_end_scn_delay | 5000 (ms) | 60 Seconds | 30 Seconds | 5 Minutes | Notice |
| logservice_replica_committed_end_scn_delay | 10000 (ms) | 60 Seconds | 30 Seconds | 5 Minutes | Warning |
Alert expression:
logservice_replica_committed_end_scn_delay >= 5000(Note)logservice_replica_committed_end_scn_delay >= 10000(Warning)
Alert information
Alert Trigger Method |
Alert Level |
Scope |
|---|---|---|
| Based on monitoring metric expression | Note, Warning | Host (LogService) |
Alert template
Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: logservice_cluster=my_ls_cluster:host=11.124.9.47:stream_id=1 oblogservice Delay in End SCN of Committed Logs in Log Service
Alert Details
- Template: Cluster: ${logservice_cluster}, Host: ${host}, Log Stream ID: ${stream_id}, Committed Log End SCN Latency: ${value_shown} ms, Exceeds Threshold ${alarm_threshold} ms
- Example: Cluster: my_ls_cluster, Host: 11.124.9.47, Log stream ID: 1, Committed log end SCN delay: 12000 milliseconds, Exceeds threshold of 10000 milliseconds
Alert Recovery
- After the end SCN latency drops below the threshold, it is automatically eliminated after the elimination cycle (5 minutes) is met.
Impact on the system
Log replica synchronization lag can lead to downstream consumption delays, increased RPO, and in severe cases, affect the high availability and data consistency of the log service.
Possible causes
- Insufficient write/sync performance at the source or replica end
- Network bandwidth or latency issues
- Log spilling or slow log pull due to disk I/O bottleneck
- Replica synchronization interrupted due to single-node failure
Solution
- View the latency trend by stream_id in OCP to locate the lagging log streams.
- Analyze bottlenecks by combining metrics such as logservice_receive_, logservice_fetch_, and logservice_pwrite_*
- Check the health status and network connectivity of replica nodes
- Restart or replace lagging nodes if necessary.