monagent_process_stop

Description

This alert is triggered when the ocp_monagent process stops.

The ocp_agent process monitors the uptime of the processes of components in OceanBase Database. When the uptime of a process is 0, the process is stopped. This alert indicates that the process has stopped at the moment rather than for a long time. In the latter case, other status alerts are triggered.

Relevant alerts:

• observer process: observer_process_stop
• obproxy process: obproxy_process_stop
• obproxyd.sh process: obproxyd_process_stop
• ocp_agentd process: agentd_process_stop
• ocp_mgragent process: mgragent_process_stop
• ocp_monagent process: monagent_process_stop

Principle

Alert rule

Alert information

Alert templates

• Alert Overview

  • Template: ${alarm_target} ${alarm_name}
  • Example: alarm_template_id=0:ob_cluster=AdminMETA-12:host=xxx.xxx.xxx.xxx ocp_monagent process stopped

• Alert Details

  • Template: Cluster: ${ob_cluster_name}, Host: ${host}, Alert: ${alarm_name}.
  • Example: Cluster: AdminMETA, Host: xxx.xxx.xxx.xxx, Alert: ocp_monagent process stopped.

• Alert Recovery

  • Template: Alert: ${alarm_name}, ocp_monagent process start timestamp change: ${recover_value}
  • Example: Alert: ocp_monagent process stopped, ocp_monagent process start timestamp change: 0 seconds

Impact on the system

OCP-Agent uses the ocp_monagent process to collect monitoring data.

1. If the ocp_monagent process is restarted by the ocp_agentd process soon after it stops, the impact on the OCP service is negligible. The ocp_agentd process can try at most three times to start the ocp_monagent process. Monitoring data for a few seconds may be lost, and the monitoring data cached by the process will be lost. You can rebuild the cache by querying internal tables of OceanBase Database. If the process restarts frequently, the cache implementation will lead to cache penetration, which increases the workload of OBServer nodes. Usually, the monitoring data is collected at a rate less than 20 QPS.

2. If the ocp_monagent process is not restarted after it stops, a large amount of monitoring data will be lost, and alerts that depend on the monitoring data become invalid. However, alerts that do not depend on the monitoring data are still active. In this case, OBServer nodes cannot retain the monitoring history.

Possible causes

• If the ocp_monagent process consumes more than 2 GiB of memory, it is restarted by the ocp_agentd process. This may be caused by the following reasons:

  • Monitoring data is collected at a high rate, which occupies large memory space.
  • The process memory leaks.

• A process bug causes an execution error and the process unexpectedly exits.

• A monitoring configuration error, such as a YAML syntax error of a custom configuration in the module_config file stored in the /home/admin/ocp_agent/conf/ directory.

• An expected O&M action. In this case, you can restart the OCP-Agent in the OCP console.

Solutions

1. Log in to the OBServer server as the administrator or a user with the sudo privilege.
2. Run the /home/admin/ocp_agent/bin/ocp_agentctl config -u monagent.limit.memory.quota=xxx command to modify the memory limit for the ocp_monagent process.
3. Restart ocp_agent.

A specific example is as follows:

-- Modify the memory limit for the ocp_monagent process.
/home/admin/ocp_agent/bin/ocp_agentctl config -u monagent.limit.memory.quota=10240MB
{"successful":true,"message":"success","error":""}

-- Restart the ocp_agent process.
systemctl restart ocp_agent

-- You can run the following command to check whether the modification takes effect.
cat /sys/fs/cgroup/memory/ocp_agent/ocp_monagent/memory.limit_in_bytes
10737418240

cat /home/admin/ocp_agent/conf/config_properties/common_meta.yaml | grep -A4 monagent.limit.memory.quota
    - key: monagent.limit.memory.quota
      value: 10240MB
      valueType: string
      encrypted: false
      configVersion: "2025-09-24T16:51:33.747+08:00"