node_file_inode_usage|V4.2.2| docs|Distributed Database

node_file_inode_usage

Last Updated：2024-03-22 09:34:34 Updated

Description

In Linux, index nodes (inodes) store the following metadata of files: file size, file owner ID, read/write permissions, change timestamp, and block position.

An inode is an index of a file. The storage of inodes also occupy disk space. If many small files are created, many inodes may be occupied. This alert is triggered when the usage of inodes exceeds 80%.

Principle

Parameter	Value
Metric	file_inode_usage
Source	This metric is a basic host monitoring metric collected by node_exporter. The inode usage is equivalent to the ratio of IUsed to Inodes in the `df -i` command output.
Collected metric	node_filesystem_files_free, node_filesystem_files
Metric expression	round(100 - 100 * sum(node_filesystem_files_free{@LABELS}) by (@GBLABELS) / sum(node_filesystem_files{@LABELS}) by (@GBLABELS))
Collection cycle	1 second

Alert rule

Metric expression	Metric description	Default threshold (unit: percentage)	Detection cycle	Elimination cycle
file_inode_usage > 80	The inode usage.	80%	10 seconds	5 minutes

Alert information

Trigger method	Alert level	Scope
Based on the expression of the metric	Critical	Server

Alert templates

Overview
- Template: ${alarm_target} ${alarm_name}
- Example: svr_ip=xxx.xxx.xxx.xxx node_file_inode_usage
Details
- Template: Cluster: ${ob_cluster_name}, host: ${svr_ip}, alert: ${alarm_name}, inode usage ${value_shown} exceeds ${alarm_threshold}%.
- Example: Cluster: obcluster, host: xxx.xxx.xxx.xxx, alert: node_file_inode_usage, inode usage 2% exceeds 1%.

Impact on the system

After all inodes are occupied, no more files can be created in the system. This is fatal to database write operations and will cause transaction commit failures, log synchronization failures, unavailability of OBServers, and other consequences.

Possible causes

There are too many small files in the system that occupy a large number of inodes.

Suggested solutions

View the directory that uses the most inodes.
```
df -i
```

Find the directory with the most files.

for i in /target-path/* ; do echo $i; find $i | wc -l | sort -nr; done

Delete junk files.
Notes
- There may be hidden files in the system. Both soft links and hard links will occupy inodes. Confirm and delete the junk files.
- If there are too many files, an error may be reported when you run the rm -rf * command. In this case, you can run the batch-delete command ls -a | xargs -n 1000 rm -rf.