Description
This alert is triggered when the query for the backup storage directory capacity times out or the thread is interrupted due to issues such as a network failure between OceanBase Cloud Platform (OCP) and the backup server.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
| Parameter | Value |
|---|---|
| Metric | storage_capacity_monitor_error_code |
| Source | OCP-Server monitors the backup storage directory capacity. In one of the following scenarios, an error code is returned and assigned to the collected metric. * 1001: The number of backup storage directory capacity query attempts exceeds the threshold. * 1002: The backup storage directory capacity query times out or the thread is interrupted. |
| Collected metric | storage_capacity_monitor_error_code |
| Metric expression | max(storage_capacity_monitor_error_code{@LABELS}) by (@GBLABELS) |
| Collection cycle | 1 second |
Note
The metric source of this alert is special. For more information, see the description in the Source row of the preceding table.
The value of the metric is the error code collected by OCP-Server when OCP-Server monitors the backup storage directory capacity. When the value of the metric is 1002, this alert is triggered.
Alert rule
| Metric | Default threshold | Duration | Detection cycle | Time before clearance |
|---|---|---|---|---|
| storage_capacity_monitor_error_code | None | 0 seconds | 30 minutes | 40 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Reminder of OCP | Stopped | Service |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: ${alarm_target} ${alarm_name}
Overview example: storage_url=file:///obbackup/yc225_214/inc_data/. The number of the backup storage directory capacity query attempts exceeds the threshold. Error code 1001.0.
Details example: storage_url=file:///obbackup/yc225_214/inc_data/. The number of the backup storage directory capacity query attempts exceeds the threshold. Error code 1001.0.
${alarm_target} follows the storage_url=file:///obbackup//yc225_214/inc_data/ format.
Impact on the system
The trend data of storage capacity on the Backup & Recovery page is displayed at an interval of more than half an hour or is not displayed.
Possible causes
This problem is commonly found in the following scenarios:
A network error occurred between OCP and the backup server.
The backup storage is inaccessible or is not functional.
Suggested solutions
Check the network connection between the OCP-Server and the backup storage server.
Perform the following steps to verify whether a network error exists:
# Ping the backup storage server from the OCP-Server. The IP address of the backup storage server is 192.168.0.1. ping 192.168.0.1 # Ping the OCP-Server from the backup storage server. The IP address of the OCP-Server is 192.168.0.2. ping 192.168.0.2If the system consecutively returns data transmission messages, the network connection is normal and the alert may have been triggered by other issues.
Otherwise, the network connection is disconnected.
Contact your network administrator for troubleshooting or troubleshoot and fix the network failure on your own. For more information, see Network troubleshooting.
Check the accessibility of the backup storage directory.
Log on to the backup storage server.
Run the following commands to check whether the administrator has the permission to read the backup storage directory.
# The default backup storage directory is /obbackup. cd / && ll | grep obbackup # The following content is returned. drwxrwxrwx 16 root root 4096 Aug 10 14:34 obbackuprwxrwxrwx indicates that all users can read and write the directory.
These two "root" indicate the user of the backup storage directory and the group of the user, respectively.
Without this permission, the administrator cannot access the directory. You may learn about Linux permissions by using Google Search and analyze the reason for the access rejection.
You can run the following command to modify the permissions for accessing the backup storage directory.
chmod -R 777 /obbackup