Description
This alert is triggered when the memory usage by the OceanBase sys tenant (ID = 500) exceeds the threshold. This sys tenant is used for statistical purpose only and does not refer to a particular tenant.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
Parameter |
Value |
|---|---|
| Metric | ob_tenant500_mem_hold_percent
Note |
| Source | select sum(hold), sum(used) from __all_virtual_memory_info where tenant_id = 500 and svr_ip = @svr_ip and svr_port = rpc_port();
Note |
| Collected metrics (unit: GB) | mem_hold and mem_total |
| Metric expression | 100 * sum(mem_hold{metric_group="tenant500_memory_info",@LABELS}) by (@GBLABELS) / sum(mem_total{metric_group="all_virtual_server_stat",@LABELS}) by (@GBLABELS) |
| Collection cycle | 60 seconds |
Alert rule
Metric |
Default threshold (unit: %) |
Duration |
Detection cycle |
Time before clearance |
|---|---|---|---|---|
| ob_tenant500_mem_hold_percent | 95 | 0 seconds | 60 seconds | 5 minutes |
Alert information
Trigger method |
Alert level |
Scope |
|---|---|---|
| Metric expression | Critical | Server |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: ${alarm_target} ${alarm_name}. The memory usage is ${value}%, exceeding the threshold of ${alarm_threshold}%.
Overview example: ob_cluster=C1-1000. The memory usage of the OceanBase sys tenant (ID = 500) exceeds the threshold.
Details example: ob_cluster=C1-1000. The memory usage of the OceanBase sys tenant (ID = 500) is 96.0%, exceeding the threshold of 95.0%.
Impact on the system
If the OceanBase sys tenant (ID = 500) consumes too many memory resources, the OBServer memory will be depleted, causing tenants in the OBServer to malfunction.
Possible cause
Some known defects cause the memory usage of the OceanBase sys tenant (ID = 500) to increase continuously in specific scenarios.
Suggested solutions
Go to the Resource Management page of the sys tenant of the OceanBase cluster, and view the Memory curve chart in the Resource Usage Tendency section to check for any unexpected memory usage.
If the memory usage is expected, increase the alert threshold to avoid the alert from being frequently generated.
In the left-side navigation pane, choose System Management > Alerts . In the Alert Item Configuration tab, find the target alert, and then click Edit to modify the alert threshold.
If the memory usage exceeds the expectation, restart the observer process to prevent the OBServer from malfunctioning due to insufficient memory.
Find the target OBServer in the OBServers list on the Overview page of the OceanBase cluster, and then click Restart to restart the observer process.