Description
In OceanBase Database V3.x, if the memory usage of the STORAGE_SHORT_TERM_META_CTX_ID module reaches 50% of the system memory, no more memory can be further allocated to the sys500 tenant. The size of SSTables are not fixed. Therefore, the calculation results based on quantity are inaccurate. Therefore, OCP queries the all_virtual_memory_info view to monitor the memory usage of the sys500 tenant. This alert is triggered when the memory usage of the sys500 tenant exceeds 30%.
Principle
| Parameter | Value |
|---|---|
| Metric | tenant500_storage_short_meta_hold_percentage,tenant500_storage_short_meta_hold_gb |
| Source | OCP monitors the minidump directory in the operation directory of the obproxy process, collects the latest core dump file, and uses the difference between the core dump file creation time and the current time as the metric. |
| Collected metric | obproxy_coredump_time_seconds |
| Metric expression | max(obproxy_coredump_time_seconds{}) by (obproxy_cluster,svr_ip) |
| Collection cycle | 5 seconds |
Alert rule
| Metric expression | Metric description | Default threshold | Detection cycle | Time before clearance |
|---|---|---|---|---|
| obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 | The time elapsed since the latest core dump of the obproxy process | 0 | 10 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Metric expression | Critical | Host |
Alert templates
Overview
- Template: ${alarm_target} ${alarm_name}
- Example: alarm_template_id=2:obproxy_cluster=TEST-2000005:host=xxx.xxx.xxx.xxx. An OBProxy server core dump occurred.
Details
- Template: Cluster: ${obproxy_cluster}. Host: ${host}. Alert: A core dump occurred on the OBProxy server ${value_shown} ago.
- Example: Cluster: TEST. Host: xxx.xxx.xxx.xxx. Alert: A core dump occurred on the OBProxy server 19 minutes 50.54 seconds ago. Impact on the system
Impact on the system
A core dump file is generated when an unexpected error occurs to a process. The file stores the environment information when the error occurs for later analysis. To protect the system, the process restarts after a core dump occurs. The impact on the system needs to be analyzed on a case by case basis.
Possible causes
- An unexpected error occurred in the program.
- If the OBProxy version is outdated, a core dump can be triggered due to design restrictions. For example, a mysql packet greater than 16 MB can trigger a core dump of OBProxy of an outdated version.
Suggested solutions
- Obtain the core dump file and contact OceanBase Technical Support for analysis. By default, the core dump files are stored in the
/home/admin/logs/obproxy/minidumpdirectory. The suffix of a core dump file name is.dmp. - The default memory size for OBProxy is 2 GB. If you allocate a memory size that is far greater than default size, the core dump process may take a long time. In an emergency, the obproxy process is not restarted. This is to ignore the core dump and prevent the escalation of the issue. You can upgrade OBProxy to a later version. In later versions, memory management is optimized.
