log_backup_storage_space_over_threshold Log Backup Storage Space Exceeds Threshold

Alert description

This alert monitors the usage capacity of log backup storage and is triggered when the log backup storage space reaches the usage threshold.

Alert principle

OCP-Server is configured with a check task that runs every minute to trigger an alert when the log backup storage usage exceeds the threshold.

Rule information

Alert information

Alert template

• Alert Overview

  • Template: ${alarm_target} ${alarm_name}
  • Example: alarm_template_id=0:ob_cluster=obcluster-10006:tenant_name=tenant_A Log backup storage space exceeds the limit

• Alert Details

  • Template: Cluster: ${ob_cluster_name}[${ob_cluster_id}], Tenant: ${tenant_name}. The storage capacity of the log backup storage address ${storage_url} has exceeded the limit. The used storage capacity ratio is ${value}, which exceeds the threshold ${threshold}. Please confirm!
  • Example: Cluster: obcluster[10006], Tenant: tenant_A, Log backup storage address file:///obbackup/log/ Storage capacity exceeded the limit. Used storage capacity: 80%. Threshold: 70%. Please confirm!

Impact on the system

If the log backup storage directory runs out of disk space, new log backups may fail to be written, which can affect recovery timeliness and data integrity.

Possible causes

• The mount disk for the log backup storage directory is configured with insufficient capacity.
• The retention period configured for backup cleanup upon expiration is set too low, causing log backup files to consume excessive storage space.
• Excessively frequent log backups generate a large number of backup files.

Solution

• Expand the volume where the log backup storage directory is mounted.

• Reduce the number of days for expired cleanup in the backup strategy.

  • You can reduce the value of the OCP system parameter ocp.backup.alarm.backup-data-retention-days to modify the retention period for global backup data.

  • You can also reduce the value of Retention Period of Backup Files in the backup strategy to decrease the retention period for only a specific cluster.

• Check whether the log backup frequency is appropriate, and adjust the scheduling configuration as needed to reduce the number of log backup files.