Alert description
This alert indicates that the oblogservice process has restarted or stopped in the LogService cluster.
Alert principle
The following table lists the key parameters involved in the monitoring logic of this alert.
Parameter |
Value |
|---|---|
| Monitoring Metrics | logservice_boot_time_delta_seconds: The change in the startup timestamp of the OBLOBSERVICE process over a period, indicating whether the process has restarted. |
| Monitoring Expression | max(delta(process_boot_time_seconds{name="oblogservice",@LABELS}[@INTERVAL])) by (@GBLABELS) |
| Metric Collection | process_boot_time_seconds{name="oblogservice"} |
| Metric Source | OCP MonAgent collects OBLOBSERVICE process information through the processInput plugin. |
| Collection Cycle | 60 Seconds |
Rule information
Monitoring Metrics |
Default Threshold |
Duration |
Detection Cycle |
Elimination Cycle |
|---|---|---|---|---|
| logservice_boot_time_delta_seconds | 10 (seconds) | 0 Seconds | 20 Seconds | 5 Minutes |
Alert expression: logservice_boot_time_delta_seconds > 10
Alert information
Alert Trigger Method |
Alert Level |
Scope |
|---|---|---|
| Based on monitoring metric expression | Warning | Host (LogService) |
Alert template
Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: logservice_cluster=my_ls_cluster:host=11.124.9.47 oblogservice process stopped
Alert Details
- Template: Cluster: ${logservice_cluster}, Host: ${host}, Alert: ${alarm_name}.
- Example: Cluster: my_ls_cluster, Host: 11.124.9.47, Alert: oblogservice process stopped.
Alert recovery
- After the process startup timestamp returns to normal, it will automatically expire after the clearance period (5 minutes).
Impact on the system
During the stop or restart of the oblogservice process, the corresponding node cannot provide services such as log reception, storage, and synchronization normally. This may lead to log write delays and lag in replica synchronization, affecting the upper-layer business that depends on this LogService cluster.
Possible causes
- Abnormal exit of the oblogservice process (such as OOM, core dump, or being killed)
- Process restart triggered by O&M operations (upgrade, configuration change, manual restart)
- The process cannot be started due to insufficient host resources or disk failure.
Solution
- Log in to the alert host and check whether the oblogservice process is running:
ps -ef | grep oblogservice - Check the oblogservice logs to identify the cause of the exit or restart.
- If the process exits abnormally, fix the root cause and then restart the process manually or through OCP.
- For scheduled maintenance operations, the alert can be dismissed after confirming that the cluster status has returned to normal.