Description
This alert indicates that log stream downgrade occurs in a tenant. A tenant for which the arbitration service is enabled must have two or four full-featured replicas. When a tenant with two full-featured replicas has one failed replica, or a tenant with four full-featured replicas has two failed replicas, log stream degradation is triggered for the tenant. In that case, the tenant can still provide services even if the number of available replicas does not meet the majority requirement. OCP will generate alerts to indicate the tenants that have downgraded log streams.
Principle
| Parameter | Value |
|---|---|
| Metric | ob_tenant_log_stream_degraded_count |
| Source | select /*+ MONITOR_AGENT*/ t2.tenant_id, t2.tenant_name, sum(t1.degraded_list <> '') as degraded_count from V$OB_LOG_STAT t1 inner join DBA_OB_TENANTS t2 on t1.tenant_id = t2.tenant_id and t2.tenant_type != 'META' where svr_ip= ? and svr_port= ? group by t1.tenant_id |
| Collected metric | ob_tenant_log_stream_degraded_count |
| Metric expression | sum(log_stream_degraded_count{@LABELS}) by (@GBLABELS) |
| Collection cycle | 60 seconds |
The log_stream_degraded_count metric indicates the number of downgraded log streams in the tenant. If the value is greater than 0, this alert is triggered.
Alert rule
| Metric expression | Default threshold | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| log_stream_degraded_count | 0 | 0 seconds | 60 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the metric | Critical | Tenant |
Alert templates
- Overview
- Template: ${alarm_target} ${alarm_name}
- Example: ob_cluster_name=obcluster:ob_cluster_id=4:tenant_name=mysql The OceanBase tenant has downgraded log streams.
- Details
- Template: Cluster: ${ob_cluster_name}; Tenant: ${tenant_name}; Alert: ${alarm_name} Log streams in the ${tenant_name} tenant are downgraded by the arbitration service. The tenant has ${value_shown} downgraded log streams.
- Example: Cluster: obcluster; Tenant: mysql; Alert: The OceanBase tenant has downgraded log streams. Log streams in the mysql tenant are downgraded by the arbitration service. The tenant has 3 downgraded log streams.
Impact on the system
When log stream downgrade occurs, the tenant is unavailable for a few seconds, which may slightly affect the business system.
Possible causes
- The IDC where a full-featured replica of the tenant is located is faulty.
- The OBServer node where the log stream of a full-featured replica is located fails.
Solutions
Connect to the OceanBase cluster as the
root@sysuser and run the following command to query the downgraded log streams and the OBServer nodes where the log streams are located:select t2.tenant_id, t2.tenant_name, t1.degraded_list from V$OB_LOG_STAT t1 inner join DBA_OB_TENANTS t2 on t1.tenant_id = t2.tenant_id and t2.tenant_type != 'META';View the running status of the OBServer nodes obtained in
Step 1 .If an OBServer node is down, we recommend that you restart the OBServer node. After the OBServer node is started, the alert is cleared.
