Alert description
Note
This alert takes effect only for OceanBase clusters of version V4.2.0.0 or later.This alert detects the number of tablets in an OceanBase tenant whose number of compacted SSTables exceeds the limit.
Alert principle
In OceanBase storage, an upper limit is set for the number of SSTables per partition. When the number of SSTables exceeds the high watermark (which is 56 by default), no new SSTable can be added, potentially preventing MiniCompaction. A minor compaction compacts eligible minor compaction SSTables, merging multiple SSTables into one to reduce the number of SSTables. Therefore, when the number of minor compaction SSTables exceeds the alert threshold, you should monitor the execution of minor compactions.
The following table lists the key parameters involved in the monitoring logic of this alert.
Parameter |
Value |
|---|---|
| Monitoring Metrics | ob_tenant_mini_merge_tablet_count: The number of tablets whose number of minor compactions exceeds the threshold. An alert is triggered when this value exceeds the threshold. |
| Monitoring Expression | sum(ob_tenant_mini_merge_tablet_count{@LABELS}) by (@GBLABELS) |
| Metric Collection | ob_tenant_mini_merge_tablet_count |
| Data Source | SQL collection: select /*+ MONITOR_AGENT READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(20000000) */x.tenant_id,x.svr_ip,x.svr_port,count(*) as cnt from (select tenant_id,svr_ip,svr_port,tablet_id,count(*) as cnt from GV$OB_SSTABLES where table_type = 'MINOR' or table_type = 'MINI' group by tenant_id,svr_ip,svr_port,tablet_id having cnt >=30) as x where x.cnt >0 group by x.tenant_id,x.svr_ip,x.svr_portThis SQL statement is executed in the sys tenant of each cluster. It is counted when the number of minor compaction SSTables per tablet in a tenant is greater than or equal to 30. |
| Collection Cycle | 5 Seconds |
Rule information
Monitoring Metrics |
Default Threshold (Unit: Count) |
Duration |
Detection Cycle |
Elimination Cycle |
|---|---|---|---|---|
| ob_tenant_mini_merge_tablet_count | 0 | 0 Seconds | 60 Seconds | 5 Minutes |
Alert information
Alert Trigger Method |
Alert Level |
Scope |
|---|---|---|
| Based on monitoring metric expressions | Critical | Tenant |
Alert template
Alert Overview
- Template: ${alarm_target} ${alarm_name}
- Example: alarm_template_id=0:ob_cluster_name=obcluster:ob_cluster_id=4:tenant_name=mysql:host=xx.xx.xx.xx Number of tablets exceeding the maximum number of SSTables allowed for minor compaction in the OceanBase tenant
Alert details
- Template: Cluster: ${ob_cluster_name}, Tenant: ${tenant_name}, Alert: ${alarm_name}. The number of tablets with excessive minor compactions, ${value_shown}, exceeds the threshold of ${alarm_threshold}.
- Example: cluster: obcluster, tenant: mysql, alert: Number of tablets exceeding the limit for OceanBase tenant minor compaction SSTables. The number of tablets exceeding the limit for minor compaction SSTables is 10, which exceeds 0.
Alert Recovery
- Template: Alert: ${alarm_name}, Number of tablets exceeding the limit for OceanBase tenant minor compaction SSTables: ${value_shown}
- Example: Alert: Number of tablets exceeding the limit for minor compaction of SSTables in the OceanBase tenant. Number of tablets exceeding the limit for minor compaction of SSTables in the OceanBase tenant: 10
Impact on the system
When the number of minor compactions exceeds the high watermark (which is 56 by default), no new SSTables can be added, and minor compaction cannot proceed. This may subsequently affect the minor compaction of the memtable and the release of memory. If the memory usage exceeds the limit, write operations will be throttled or even halted.
Possible causes
- The Minor Compaction task queue is accumulating, as tasks are being consumed slower than they are added.
- Minor compactions cannot be scheduled due to certain internal limitations.
Solution
If the cause is a backlog in the minor compaction task queue, you can refer to the following command to increase the number of threads in the tenant's minor compaction queue:
alter system set compaction_mid_thread_score = xxx tenant xxx;For other reasons, you can refer to the Troubleshoot excessive SSTables topic for troubleshooting and resolution.