Hitting the limits (inode-max)

Recently one of my qmail servers (at work) stopped to deliver mails. A quick look at the log files revealed that qmail was unable to create the mail files in the Maildir directories:

mail qmail: 1158303878.354871 delivery 5050: failure:
can_not_open_new_email_file_errno=28_file=/dir/to/maildir/folder

Trying to manually create a file in the folder gave me a no free space on device error but df command said the hard disk had plenty of free space. After some more research the problem resulted to be that the partition had hit the inode-max limit. This value is the max number of files that can be stored in the file system.

To check out this you can use the command "df -i" or look in /proc/sys/fs/inode-nr.

This seems to be a common problem in mail servers that create a lot of small files (mail messages and mail folders) since each file uses a inode.

In linux kernels previous to the 2.4 series there was a inode-max parameter that could be changed (i.e. increased) to fix this problem. From kernels 2.4 onwards this parameter is managed automatically by the kernel but it still has a max value and to increase that maximum we must increase the file-max parameter of the kernel.

After some more research I understand that inode-max is usually three time the file-max value and that file-max value must be set according to the available ram in the server.

In this online book they recomend setting 256 files per 4MB of ram to file-max, for example if we have 128MB ram then

256 * (128 / 4) = 8192

and to set it in a running server simple run

# echo "8192" >/proc/sys/fs/file-max

where value is the number obtained from the calculation. Then inode-max will be roughly 3*value that is, three times file-max.

In RedHat we can set this parameter on the /etc/sysctl.conf file so this setting is kept at every reboot of the machine.

# Improve the number of open files
fs.file-max = 8192

So if you are setting up a server that requires to store lots of files (mail, http) then you may need to increase the value of file-max.

There are questions I still cannot answer myself:

- What happens if we keep increasing file-max ignoring the RAM amount??
- Maybe using a File System that handles small files more efficiently we can
reduce the number of inodes required?? for example ReiserFS instead of Ext3?

Any tips on these is appreciated....


Here are some links with more information about server tunning and file-max explanation:

System Tunning
file-max discussion at kernel level