ob_host_disk_percent_over_threshold

Description

This alert is triggered when disk usage in the directory of the /data/1, /data/log1, or /home/admin mount point exceeds the threshold.

Alert information

Alert rule

Alert templates

• Overview: ${alarm_target} ${alarm_name}

• Details: ${alarm_target} ${alarm_name} The disk usage is ${value}%, exceeding the threshold of ${alarm_threshold}%.

• Overview example: ob_cluster=C1-1000:svr_ip=xxx.xxx.xxx.xxx Excessive disk usage of the OBServer

• Details example: ob_cluster=C1-1000:svr_ip=xxx.xxx.xxx.xxx Excessive disk usage of the OBServer The disk usage of the OBServer reaches 98.0%, exceeding the threshold of 97.0%.

Impact on the system

When the OceanBase log disk space is insufficient, the OBServer may malfunction. When the /home disk is full, the OBServer cannot write to the log file.

Possible causes

In normal cases, the data disk and log disk will not be full because the data disk is pre-configured and the log disk is recyclable. The /home disk may be full when applications generate too many files. OceanBase Cloud Platform (OCP) automatically clears log files of the OBServer.

The OceanBase data disk space can be configured with special alert items. For more information, see the following alert items:

• ob_server_sstable_percent_over_threshold

• ob_zone_sstable_percent_over_threshold

Suggested solutions

1. Identify the faulty disk.

   # Run the df command to view the disk usage.
   df
   Filesystem             1K-blocks       Used Available Use% Mounted on
   /dev/sda3               51343996   17639360  31066812  37% /
   devtmpfs                24617024          0  24617024   0% /dev
   /dev/sda5              405722588   95356892 289685816  25% /home
   /dev/mapper/vgob-log   603736168   13550516 559447836   3% /data/log1
   /dev/mapper/vgob-data 4506040888 4022511972 256367216  95% /data/1
   

   In the preceding command:

   • /data/1 is the data disk of OceanBase.

   • /data/log1 is the log disk of OceanBase.

   • /home is used to print log files that are stored in the /home/admin/oceanbase/log directory.

2. Run the following commands to find the directories and files that occupy the most space:

   # Find the five directories that occupy the most space.
   du -a /home/admin | sort -n -r | head -n 5
   
   # Find the five largest files.
   cd /home/admin && find -type f -exec du -Sh {} + | sort -rh | head -n 5
   

3. Delete unwanted files.