node_file_inode_usage High inode usage on the server|V4.3.6| docs|Distributed Database

node_file_inode_usage High inode usage on the server

Last Updated：2025-09-08 08:15:43 Updated

Description

In a Linux system, an inode stores the metadata of a file, including the file size, document owner ID, read/write permissions, modification timestamp, and block location.

An inode is an index of a file, and it also occupies disk space. If too many small files are created, it may occupy excessive inodes. This alert is triggered when the inode usage exceeds 80%.

Principle

Parameter	Value
Monitoring metric	file_inode_usage
Source of the metric	Host basic monitoring, collected by node_exporter. Inode usage is similar to IUsed / Inodes in `df -i`.
Metrics to be collected	node_filesystem_files_free, node_filesystem_files
Monitoring expression	round(100 - 100 * sum(node_filesystem_files_free{@LABELS}) by (@GBLABELS) / sum(node_filesystem_files{@LABELS}) by (@GBLABELS))
Collection interval	1 second

Rule information

Monitoring expression	Meaning of the monitoring metric	Default threshold (percent)	Detection interval	Elimination interval
file_inode_usage > 80	Inode usage	80%	10 seconds	5 minutes

Alert information

Alert trigger method	Alert level	Scope
Monitoring expression	Severe	Server

Alert template

Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: svr_ip=xxx.xxx.xxx.xxx High inode usage on the server
Alert details
- Template: Cluster: ${ob_cluster_name}, Host: ${host}, Alert: ${alarm_name}. Inode usage ${value_shown} exceeds ${alarm_threshold} %.
- Example: Cluster: obcluster-1, Host: xxx.xxx.xxx.xxx, Alert: High inode usage on the server. Inode usage 95 % exceeds 80 %.
Alert recovery
- Template: Alert: ${alarm_name}, Server inode usage: ${value_shown}
- Example: Alert: High inode usage on the server, Server inode usage: 60 %

Impact on the system

After all inodes are used, the system cannot create files. This is fatal for database writes, leading to transaction commit failures, log synchronization issues, and OBServer unavailability.

Possible causes

Too many small files occupy a large number of inodes.

Procedure

View the directory that uses the most inodes.
```
df -i
```

View the directory that contains the most files.

for i in /target-path/* ; do echo $i; find $i | wc -l | sort -nr; done

Delete garbage files.

Notice

The system may contain hidden files, soft links, and hard links, which all occupy inodes. After you confirm that the files are garbage, delete them.
If there are too many files, an error may be returned when you run the rm -rf * command. In this case, you can run the batch deletion command: ls -a | xargs -n 1000 rm -rf.