Description
This alert is triggered when the memory usage of a tenant exceeds the threshold.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
| Parameter | Value |
|---|---|
| Metric | ob_tenant_host_memstore_percent |
| Source | SQL: select /*+read_consistency(weak)*/ tenant_name, tenant_id, stat_id, value from v$sysstat, __all_tenant where stat_id IN (130001, 130004) and (con_id > 1000 or con_id = 1) and__all_tenant.tenant_id = v$sysstat.con_id;The value of the value field is used as the value of sysstat_value, and the values of other fields are used as labels. |
| Collected metric | sysstat_value |
| Metric expression | 100 * sum(sysstat_value{metric_group="sysstat",stat_id="130001",@LABELS}) by (@GBLABELS) / sum(sysstat_value{metric_group="sysstat",stat_id="130004",@LABELS}) by (@GBLABELS) |
| Collection cycle | 1 second |
Note
- When the statistical event ID (
stat_id) is130001, the total used memory space of the tenant is collected. - When
stat_idis130004, the maximum available memory space of the tenant is collected.
In the metric expression, the ratio of the sum of the values of the value field when stat_id is 130001 to the sum of the values of the value field when stat_id is 130004 is used as the memory usage of a tenant.
The value of the ob_tenant_host_memstore_percent metric indicates the memory usage of a tenant in the OceanBase cluster. When the memory usage exceeds the threshold, this alert is triggered. The default threshold is 85%.
Alert rule
| Metric | Default threshold (unit: %) | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| ob_tenant_host_memstore_percent | 85 | 0 seconds | 60 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the metric | Warning | Tenant |
Alert templates
Overview template: ${alarm_target} ${alarm_name}
Details template: Cluster: ${ob_cluster_name}. Alert: ${alarm_name}. The MemStore usage reaches ${value_shown}, which exceeds the threshold ${alarm_threshold}%.
Overview example: ob_cluster=obcluster-1:tenant_name=tenant-1:svr_ip=xxx.xxx.xxx.xxx. The memory usage of the OceanBase Database tenant exceeds the threshold.
Details example: Cluster: obcluster-1. Alert: The memory usage of the OceanBase Database tenant exceeds the threshold. The MemStore usage reaches 86.0%, which exceeds the threshold 85.0%.
Impact on the system
If the tenant memory is insufficient, SQL statement execution may be slowed down, and writes may fail.
Possible causes
This problem is commonly found in cases where write requests sharply increase but only limited resources are available in the tenant.
Solutions
Scale up the tenant memory. You can execute the alert clearance plan to handle the alerted issue. For more information, see Execute the alert clearance plan.
Throttle the concurrency of SQL statements or increase the memory size in the unit specification of the tenant based on the check result.
Go to the Performance Monitoring page of this tenant in the OceanBase Cloud Platform (OCP) console, and then click the Performance & SQL tab.
Select Last 7 Days for Select Time.
View the curve chart of MEMStore usage percent.
Check whether the memory usage at the time when the alert was generated abnormally increases.
If the memory usage abnormally increases or regularly increases, accompanied by an alert indicating insufficient resources, bind a hint to write-type SQL statements of this tenant to limit the concurrency.
For more information, see Apply throttling to an OceanBase cluster.
If the memory usage normally fluctuates or regularly increases and resources are sufficient, increase the memory size in the unit specification of the tenant.
For more information, see Manage unit specifications.