Description
This alert is triggered when the memory usage by the OceanBase sys tenant (ID = 500) exceeds the threshold. This sys tenant is used for statistical purpose only and does not refer to a particular tenant.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
| Parameter | Value |
|---|---|
| Metric | ob_tenant500_mem_hold_percent
|
| Source | select sum(hold), sum(used) from __all_virtual_memory_info where tenant_id = 500 and svr_ip = @svr_ip and svr_port = rpc_port();
|
| Collected metrics (unit: GB) | mem_hold and mem_total |
| Metric expression | 100 * sum(mem_hold{metric_group="tenant500_memory_info",@LABELS}) by (@GBLABELS) / sum(mem_total{metric_group="all_virtual_server_stat",@LABELS}) by (@GBLABELS) |
| Collection cycle | 60 seconds |
Alert rule
| Metric | Default threshold (unit: %) | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| ob_tenant500_mem_hold_percent | 95 | 0 seconds | 60 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Metric expression | Critical | Server |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: ${alarm_target} ${alarm_name}. The memory usage is ${value}%, exceeding the threshold of ${alarm_threshold}%.
Overview example: ob_cluster=C1-1000. The memory usage of the OceanBase sys tenant (ID = 500) exceeds the threshold.
Details example: ob_cluster=C1-1000. The memory usage of the OceanBase sys tenant (ID = 500) is 96.0%, exceeding the threshold of 95.0%.
Impact on the system
If the OceanBase sys tenant (ID = 500) consumes too many memory resources, the OBServer memory will be depleted, causing tenants in the OBServer to malfunction.
Possible cause
Some known defects cause the memory usage of the OceanBase sys tenant (ID = 500) to increase continuously in specific scenarios.
Suggested solutions
Go to the
If the memory usage is expected, increase the alert threshold to avoid the alert from being frequently generated.
In the left-side navigation pane, choose System Management > Alerts . In the Alert Item Configuration tab, find the target alert, and then click
Edit to modify the alert threshold.If the memory usage exceeds the expectation, restart the observer process to prevent the OBServer from malfunctioning due to insufficient memory.
Find the target OBServer in the
OBServers list on theOverview page of the OceanBase cluster, and then clickRestart to restart the observer process.