inc_backup_delay

Description

This alert is triggered when a log backup is delayed.

Principle

OCP-Server runs a timed task to check the log backup tasks every minute. When the log backup latency exceeds the threshold, this alert is triggered. The log backup tasks include:

1. Logical log backup

2. Physical log backup

Alert rule

Alert information

Alert templates

• Overview: ${alarm_target} ${alarm_name}

• Details: Cluster: ${ob_cluster_name}[${ob_cluster_id}], Host: ${host}, Alert: A log backup is delayed by ${log_backup_latency} seconds, exceeding the threshold of ${threshold} seconds, Checkpoint: ${checkpoint}, Current time: ${current_time}, Tenant list: ${tenants}.

• Overview example: ob_cluster=C1-1000. A log backup is delayed.

• Details example: Cluster: obcluster-1[10006], Host: xxx.xxx.xxx.xxx, Alert: A log backup is delayed by 30 seconds, exceeding the threshold of 5 seconds, Checkpoint: 2020-01-05 02:00:00, Current time: 2020-01-15 01:00:00, Tenant list: tenant1,tenant2.

Impact on the system

If the log backup is significantly delayed, the backup data is not up-to-date and the system cannot restore to the latest state or perform new physical data backup tasks.

Possible causes

This problem is commonly found in the following scenarios:

• An error occurred while performing a log backup task.

• Something is wrong with the backup media.

Suggested solutions

• If the alert is triggered during a logical log backup task, check the agentserver.log of the backup service for troubleshooting.

• If the alert is triggered during a physical log backup task, check the OBServer logs to find the specific cause. Perform the following steps to find the logs:

  1. Log on to the OCP console. On the Clusters page, find the target cluster in the Clusters list and click its name. Then, the Overview page of the cluster appears.

  2. Click the More icon in the upper-right corner and select Download Logs .

     • Specify the time range, which must contain the alert time.

     • For the log type, select OBServer Logs .

  3. Click Download .

     Wait for the OBServer logs to be downloaded.

  4. Open the observer.log file and search for WARN- and ERROR-level logs.