Alert description
This alert indicates that OCP failed to synchronize tenant information from the OceanBase cluster. When an exception occurs during the background "Synchronize Tenant" task, OCP marks the system tenant (sys) of that cluster as unavailable and sends this alert to notify the administrator of the synchronization failure and its cause.
Alert principle
Parameter |
Value |
|---|---|
| Monitoring Metrics | N/A |
| Monitoring Expression | N/A |
| Metric Collection | N/A |
| Metric Source | OCP Backend Cluster/Tenant Synchronization Task |
| Collection Cycle | N/A |
Rule information
Monitoring Metrics |
Default Threshold |
Duration |
Detection Cycle |
Elimination Cycle |
|---|---|---|---|---|
| N/A | N/A | N/A | N/A | 5 Minutes |
Alert information
Alert Trigger Method |
Alert Level |
Scope |
|---|---|---|
| Maintenance Event Trigger | Critical | Tenant |
Alert template
Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: ob_cluster=obcluster1:tenant_name=sys:ob_tenant_id=1 OceanBase tenant information synchronization failed
Alert Details
- Template: Cluster: ${ob_cluster_name}, Tenant: ${tenant_name}:${ob_tenant_id}, Alert: ${alarm_name}. Synchronization of information for tenant ${tenant_name}:${ob_tenant_id} failed with error: ${error_message}.
- Example: Cluster: obcluster1, Tenant: sys:1, Alert: OceanBase tenant information synchronization failed. Tenant sys:1 information synchronization failed, error: Connection refused.
Impact on the system
Tenant information synchronization failure may result in:
The cluster's system tenant is displayed as unavailable: The system tenant is marked as UNAVAILABLE in OCP until the synchronization is successful and its status is restored.
Inaccurate tenant views: The list of tenants, their status, specifications, and other details in the cluster view may be outdated or incorrect, leading to discrepancies between the console display and the actual information in OceanBase Database.
Operations dependent on tenant information are restricted: Operations that rely on the latest tenant information, such as creating/deleting tenants, modifying allowlists, and scaling, may fail or be executed based on incorrect data.
Possible causes
Cluster unreachable: OCP cannot connect to the OceanBase cluster due to operations such as cluster upgrade or restart.
OCP or MetaDB exception: The synchronization task fails due to an exception in the OCP service or MetaDB.
Solution
Confirm the alert information: Locate the cluster and the cause of the failure based on the cluster name (ob_cluster_name), tenant name (tenant_name, usually sys), ob_tenant_id, and error_message in the alert.
Troubleshoot based on the error_message:
Connection errors (Connection refused / timeout): Check the network connectivity between OCP and OceanBase Database, the process and port of OceanBase Database, and the allowlist.
Timeout: Check the OceanBase load, slow SQL, and network latency. Increase the timeout period if necessary.
Permission or syntax errors: Check the permissions of the OCP connection account and the compatibility of the OceanBase version.
Fix the issue based on the error type specified in
error_messageand observe the result.