Description
This alert is triggered when the memory usage of a tenant exceeds the threshold.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
| Parameter | Value |
|---|---|
| Metric | ob_tenant_host_memstore_percent |
| Source | SQL: unknow select /*+read_consistency(weak)*/ tenant_name, tenant_id, stat_id, value from v$sysstat, __all_tenant where stat_id IN (130001, 130004) and (con_id > 1000 or con_id = 1) and __all_tenant.tenant_id = v$sysstat.con_id; Note The value of the value field is used as the value of sysstat_value, and the values of other fields are used as labels. |
| Collected metric | sysstat_value |
| Metric expression | 100 * sum(sysstat_value{metric_group="sysstat",stat_id="130001",@LABELS}) by (@GBLABELS) / sum(sysstat_value{metric_group="sysstat",stat_id="130004",@LABELS}) by (@GBLABELS) |
| Collection cycle | 1 second |
Note
When the statistical event ID (stat_id) is 130001, the sum of the used memory space of the tenant is collected.
When the stat_id is 130004, the maximum available memory space of the tenant is collected.
In the metric expression, the ratio of the sum of the values of the value field when stat_id is 130001 to the sum of the values of the value field when stat_id is 130004 is used as the memory usage of a tenant.
The value of the metric ob_tenant_host_memstore_percent indicates the memory usage of a tenant in the OceanBase cluster. When the memory usage exceeds the threshold, this alert is triggered. The default threshold is 85%.
Alert rule
| Metric | Default threshold (unit: %) | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| ob_tenant_host_memstore_percent | 85 | 0 seconds | 60 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Metric expression | Warning | Tenant |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: ${alarm_target} ${alarm_name}. The memory usage is ${value}%, exceeding the threshold of ${alarm_threshold}%.
Overview example: ob_cluster=C1-1000:tenant_name=tenant-1:svr_ip=192.168.1.1. The memory usage of an OceanBase tenant exceeds the threshold.
Details example: ob_cluster=C1-1000:tenant_name=tenant-1:svr_ip=192.168.1.1. The memory usage of the OceanBase tenant reaches 86.0%, exceeding the threshold of 85.0%.
Impact on the system
If tenant memory is insufficient, SQL statement execution may be slowed down, or writes may fail.
Possible cause
This problem is commonly found in cases where write requests sharply increase but only limited resources are available in the tenant.
Suggested solution
Click Performance Monitoring of this tenant in the OCP console, and then click the Performance & SQL tab.
Select 1 Week for the time.
View the curve chart of MEMStore Usage Percentage .
Check whether the memory usage at the time of the alert abnormally increases.
If the memory usage abnormally increases or it regularly increases, accompanied with an alert indicating insufficient resources, bind a hint to write-type SQL statements of this tenant to limit concurrences.
For more information, see Apply throttling to an OceanBase cluster.
If the memory usage normally fluctuates or regularly increases and resources are sufficient, increase the memory of this tenant in the unit type configuration
For more information, see Manage unit specifications.