Description
This alert is triggered when it takes over three hours to execute a task in an OBServer. The following task type is supported:
DELETE_SERVER (task_type): a task to delete a server.
Principle
| Parameter | Value |
|---|---|
| Metric | ob_server_task_max_duration_seconds |
| Source | Internal views. SQL statements for metric collection: SELECT job_type as task_type, TIMESTAMPDIFF(SECOND, START_TIME, CURRENT_TIMESTAMP) as max_sys_task_duration_seconds, svr_ip FROM DBA_OB_SERVER_JOBS WHERE job_status='INPROGRESS' |
| Collected metric | ob_server_task_max_duration_seconds |
| Metric expression | max(ob_server_task_max_duration_seconds{@LABELS}) by (@GBLABELS) |
| Collection cycle | 1 minute |
Alert rule
| Metric expression | Metric description | Default threshold (unit: s) | Detection cycle | Time before clearance |
|---|---|---|---|---|
| ob_server_task_max_duration_seconds > 10800 | The timeout duration for the execution of a task in an OBServer. | 10800 (3 hours) | 60 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the metric | Critical | Tenant |
Alert templates
- Alert overview: ${alarm_target} ${alarm_name}
- Alert details: Cluster: ${ob_cluster_name}. Host: ${host} (zone: ${obzone}). Alert: A task in the OBServer has timed out. Task Type: ${task_type}. Execution Time: ${value_shown}, which exceeds the threshold of ${alarm_threshold} seconds.
- Overview example: ob_cluster=C1-1000:tenant_name=T1. A task in the OBServer timed out.
- Details example: Cluster: C1. Host: xxx.xxx.xxx.xxx (zone: Z1). Alert: A task in the OBServer has timed out. Task Type: DELETE_SERVER. Execution Time: 4 hours, which exceeds the threshold of 10800 seconds.
Impact on the system
- A prolonged task execution period affects task scheduling in OceanBase Database, which reduces the task processing throughput.
- Prolonged occupation of resources affects the system stability.
Possible causes
None.
Solutions
Check the observer.log.wf file for ERROR logs and submit the identified ERROR logs to OceanBase Technical Support for troubleshooting.
