backup_storage_capacity_timeout_or_interrupted Backup restore directory capacity retrieval timeout or thread interrupted

Alert description

When restoring data, the backup storage capacity is obtained. If the network between OCP and the backup server is abnormal, the operation will be retried. If the operation takes too long or the thread is interrupted, this alert is triggered.

Alert principle

The following table describes the key parameters involved in the alert monitoring logic.

The value of the monitoring metric is the error code collected when OCP-Server monitors the backup and restore storage capacity. This alert is triggered when the monitoring metric value is 1002.

Rule Information

Alert Information

Alert Template

• Alert Overview

  • Template: ${alarm_target} ${alarm_name}
  • Example: storage_url=file:///obbackup/yc225_214/inc_data/ Backup restore directory capacity timeout or thread interrupted

• Alert Details

  • Template: Cluster: ${ob_cluster_name}, Alert: ${storage_url} ${alarm_name}. Capacity timeout or interrupted, error code: ${value}.
  • Example: Cluster: obcluster-1, Alert: file:///obbackup/yc225_214/inc_data/ Backup restore directory capacity timeout or thread interrupted. Capacity timeout or interrupted, error code: 1002.

• Clear Alert

  • Template: Alert: ${alarm_name}, Storage capacity collection error code: ${value}
  • Example: Alert: Backup restore directory capacity timeout or thread interrupted, Storage capacity collection error code: -

where ${alarm_target} is in the format of storage_url=file:///obbackup//yc225_214/inc_data/

Impact on the system

The storage capacity trend data on the backup restore page is unavailable or displayed with more than 30 minutes interval.

Possible causes

This issue is commonly caused by one of the following reasons:

• Network anomalies between OCP and the backup server, or between the OceanBase cluster and the backup storage server.

• The backup storage server is unavailable or abnormal.

Solution

1. Check whether the network between the OCP console and the backup storage server is faulty.

   • When the storage type is File or COS, check whether the network between the OceanBase cluster and the backup storage server is faulty. Run the following commands to check whether the network is faulty.

   Run the following commands to check whether the network is faulty.

   # Ping the backup storage server from the OCP console server. xxx.xxx.xxx.1 is the IP address of the backup storage host.
   ping xxx.xxx.xxx.1
   # Ping the OCP console server from the backup storage server. xxx.xxx.xxx.2 is the IP address of the OCP host.
   ping xxx.xxx.xxx.2
   

   • When the storage type is OSS, check whether the network between the OCP console and the backup storage server is faulty.

     Run the following commands to check whether the network is faulty.

     # Ping the backup storage server from the OCP console server. xxx.xxx.xxx.1 is the IP address of the backup storage host.
     ping xxx.xxx.xxx.1
     # Ping the OCP console server from the backup storage server. xxx.xxx.xxx.2 is the IP address of the OCP host.
     ping xxx.xxx.xxx.2
     

     • If the network is normal, the system returns continuous data sending information. In this case, the alert is caused by other reasons.

     • If the network is faulty, the system does not return continuous data sending information. Contact the network administrator to troubleshoot the network issue. If no network administrator is available, troubleshoot and repair the network issue by referring to Network troubleshooting.

2. Check whether the backup storage directory is accessible.

   1. Log in to the backup storage server.

   2. Run the following commands to check whether the admin user has the read permission on the backup storage directory.

   # /obbackup is the default backup storage directory.
   cd / && ll | grep obbackup
   
   # The following information is returned.
   drwxrwxrwx   16 root root 4096 Aug 10 14:34 obbackup
   

   • rwxrwxrwx indicates that all users can read, write, and execute the backup storage directory.

   • The two root values indicate the user and group to which the backup file storage directory belongs.

   If the permission is not rwxrwxrwx, the admin user may not be able to access the backup storage directory. To learn more about Linux permissions, visit www.baidu.com.

   3. If the permission is insufficient, run the following command to modify the permission of the backup storage directory to ensure that it is accessible.

   chmod -R 777 /obbackup