Alert description
This alert is triggered when the memory usage of OceanBase tenants exceeds the threshold.
Alert principle
The following table describes the key parameters involved in the monitoring logic of this alert.
| Parameter | Value |
|---|---|
| Monitoring metric | ob_tenant500_mem_hold_gigabyte The amount of memory used by OceanBase tenants. This alert is triggered when the memory usage of OceanBase tenants exceeds the threshold (100 GB by default). |
| Metric source | select /*+ MONITOR_AGENT READ_CONSISTENCY(WEAK) */ sum(hold) as hold, sum(used) as used from __all_virtual_memory_info where tenant_id = 500 and svr_ip = ? and svr_port = ? and mod_name <> 'OB_KVSTORE_CACHE_MB'select /* MONITOR_AGENT */ sum(hold) as hold, sum(used) as used from V$OB_MEMORY where tenant_id = 500 and svr_ip = ? and svr_port = ? and MOD_NAME <> 'KvstorCacheMb' |
| Sample metric (GB) | tenant500_mem_hold |
| Monitoring expression | sum(ob_tenant500_memory_hold_bytes{@LABELS}) by (@GBLABELS) / 1073741824 |
| Sampling interval | 60 seconds |
Rule information
| Monitoring metric | Default threshold (GB) | Duration | Detection interval | Elimination interval |
|---|---|---|---|---|
| ob_tenant500_mem_hold_gigabyte | 100 | 0 seconds | 60 seconds | 5 minutes |
Alert information
| Alert trigger method | Alert level | Scope |
|---|---|---|
| Based on the monitoring metric expression | Critical | Server |
Alert template
Alert summary
- Template: ${alarm_target} ${alarm_name}
- Example: ob_cluster=obcluster-1:svr_ip=xxx.xxx.xxx.xxx The memory usage of OceanBase 500 tenants exceeds the threshold.
Alert details
- Template: Cluster: ${ob_cluster_name}, Host: ${host}, Alert: ${alarm_name}. The memory usage ${value_shown} exceeds ${alarm_threshold} GiB.
- Example: Cluster: obcluster-1, Host: xxx.xxx.xxx.xxx, Alert: The memory usage of OceanBase 500 tenants exceeds the threshold. The memory usage 101.0 GiB exceeds 100.0 GiB.
Alert recovery
- Template: Alert: ${alarm_name}, The memory usage of OceanBase 500 tenants: ${value_shown}
- Example: Alert: The memory usage of OceanBase 500 tenants exceeds the threshold. The memory usage of OceanBase 500 tenants: 95.0 GiB
Impact on the system
Excessive memory usage of OceanBase tenants can lead to the exhaustion of memory resources on the server. OceanBase tenants on the OBServer node may not work properly due to insufficient memory.
Possible causes
Some known defects can cause the memory usage of OceanBase tenants to increase continuously in specific scenarios.
Procedure
Log in to the OCP console and go to the Resource Management page of the OceanBase cluster sys tenant. View the Resource Usage Tendency section and check the Memory line chart to determine whether the memory usage meets expectations.
If the memory usage meets expectations, you can adjust the alert threshold to avoid continuous alert triggering.
On the Edit page of the Alert Item Configuration tab in Systems Management > Alerts, you can find the corresponding alert and modify the alert threshold.
If the memory usage does not meet expectations, you need to restart the observer process to avoid the OBServer node from being unable to work due to insufficient memory.
You can click Restart in the OBServers section of the Overview page of the OceanBase cluster to restart the observer process.