Description
This alert is triggered if the length of a general request queue of a tenant exceeds the threshold and the request processing time increases. An OceanBase Database tenant supports the following request queues with different priorities:
- queue0: the queue of transaction-related remote procedure calls (RPCs).
- queue1: the queue of retries due to row lock contention.
- queue2: the queue of general RPCs and RPCs triggered by SQL requests.
- queue3: the queue of lock retry SQL requests,
OB_TASKrequests, andOB_GTS_TASKrequests. - queue4: the queue of general SQL requests and
OB_SQL_TASKrequests. - queue5: the queue of warmup requests.
The foregoing queues are listed in descending order of priority. You can view the status of the tenant queues by searching for the dump tenant info keyword in the database or querying the __all_virtual_dump_tenant_info view.
Principle
| Parameter | Value |
|---|---|
| Metric | ob_tenant_request_queue_size,request_queue_time |
| Source | ob_tenant_request_queue_size: the size of the general request queue. The value is collected from the req_queue_total_size field of the __all_virtual_dump_tenant_info view. request_queue_time: The value is collected from the v$sysstat view. |
| Collected metric | ob_tenant_request_queue_total_size,ob_sysstat |
| Metric expression | ob_tenant_request_queue_total_size: sum(rate(ob_tenant_request_queue_total_size{@LABELS}[@INTERVAL])) by (@GBLABELS) request_queue_time: sum(rate(ob_sysstat{stat_id="20002",@LABELS}[@INTERVAL])) by (@GBLABELS) / sum(rate(ob_sysstat{stat_id="20001",@LABELS}[@INTERVAL])) by (@GBLABELS) |
| Collection cycle | 5 seconds |
Alert rule
| Metric expression | Metric description | Default threshold | Detection cycle | Time before clearance |
|---|---|---|---|---|
| ob_tenant_request_queue_size >10 and request_queue_time > 1000 | ob_tenant_request_queue_size specifies the length of the request queue of the tenant. request_queue_time specifies the time, in microseconds, spent on processing a request in the queue. |
|
10 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the metric | Critical | Host |
Alert templates
Overview
- Template: ${alarm_target} ${alarm_name}
- Example: alarm_template_id=0:ob_cluster=TEST-1:tenant_name=test:host=xxx.xxx.xxx.xxx. The length of the request queue of the OceanBase Database tenant exceeds the threshold.
Details
- Template: Cluster: ${ob_cluster_name}. Tenant: ${tenant_name}. Alert: ${alarm_name}. The length of the request queue of the tenant reaches ${ob_tenant_request_queue_size_value}, which exceeds the threshold ${ob_tenant_request_queue_size_alarm_threshold}.
- Example: Cluster: TEST. Tenant: test. Alert: The length of the request queue of the OceanBase Database tenant exceeds the threshold. The length of the request queue of the tenant reaches 100, which exceeds the threshold 10.
Impact on the system
The backlog of request queues of a tenant has a significant impact on the system:
- The memory usage exceeds the threshold.
- The service response slows down.
Possible causes
- Transaction processing takes a long time and results in a backlog of requests.
- The resources of the OBServer node are insufficient to process excessive requests.
Solutions
- Execute the alert clearance plan to fix the alerted issue. For more information, see Execute the alert clearance plan.
- Retry the alert clearance plan. Make sure that the plan is executed to solve the original issue.
- Roll back the alert clearance plan. You can contact OceanBase Technical Support for assistance.