File System Benchmarks for Postfix Mail Server

After some research I decided to use XFS file system to host my next mail server (Postfix). This decision was based only on comments and suggestions I read in forums and some benchmarks published on the Internet about linux file systems [1].

I installed my new mail server with Postfix as MTA [2] and Courier as POP [3] and leaved a free 60GB partition to make some benchmark testing. This way I was able to extract some hard numbers to validate my desicion. The results show XFS is indeed the best option when running a mail server, at least for Postfix.

Test Environment

The test setup is very simple: I have one server with Ubuntu Server 6.06 LTS installed with the most recent updates and a second client machine with Kubuntu Feisty running Postal/Rabid [4] software used to load the server with SMTP/POP requests.

The machines specifications are:

	Server Machine	Client Machine
OS	Ubuntu Server 6.06 LTS	Kubuntu Feisty Fawn
CPU	Intel Xeon 2.40GHz	Intel Pentium III 800MHz
RAM	512MB	256MB
Disk	SEAGATE ST373307LC 73408MB	Maxtor 6Y120L0 SCSI 122942 MB

In the server machine I had Ubuntu Server installed with Postfix and Courier-POP configured and added 500 user accounts with Maildir boxes.

The client side is my desktop PC running Kubuntu Feisty where I simply run postal and rabid to load the server with a lot of SMTP and POP requests. I apply no limits on the number of requests per minute as we are testing the disk input/output throughput rather than the mail service itself.

Testing Procedure

On the server side I followed these steps:

• Create test partition with one of the file systems under consideration. Especial care was taken to use the creation options that are known to increase each particular file system performance as stated in various benchmarks and forums.
  
  
  
  • mkfs.ext3 -J size=100 -m 1 -O dir_index,filetype,has_journal
  
  
  • mkfs.reiserfs -b 4096 -s 16386
  
  
  • mkfs.xfs -f -l size=64m -d agcount=16
  
  
  • mkfs.jfs -s 64
  
  
  
  


• Mounted the newly created partition to a testing mount point. Again I tried my best to use the recommended mount options in order to improve each file system performance. In the case of ext3 and reiserfs we tested all three journaling methods they support.
  
  
  
  • mount -t ext3 -o noatime,nodiratime,data=journal
  
  
  • mount -t ext3 -o noatime,nodiratime,data=ordered
  
  
  • mount -t ext3 -o noatime,nodiratime,data=writeback
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=journal
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=ordered
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=writeback
  
  
  • mount -t xfs -o noatime,nodiratime,logbufs=8
  
  
  • mount -t jfs
  
  
  
  


• Copied the necessary files from the root partition to the testing partition to create a fully functional chroot environment and chrooted it:
  


• Once inside the chroot environment I start all services. First deleting all the syslogs and then starting sysklogd, postfix, courier-authdaemon and courier-pop in that order. This way all mailboxes, logs and configuration files are read/write in the file system under test.

On the client side I simply start postal and rabid to send unlimited SMTP and POP requests to the server:

postal -m 5 -r 24000 -t 5 -c 3 -s 0 server_ip userlist.txt
rabid  -p 5 -c 5 -r 24000 -s 0 server_ip userlist.txt

The userlist.txt contains the list of all 500 user accounts with password that were created on the server after installation. The postal command above will establish five SMTP connections to the server and send as much messages as it can with sizes between 0 and 5kbytes to all 500 users. On the other side rabid will establish as much POP connections as it cans to the sever from five different processes and download a maximum of five messages per connection.

Both postal and rabid were executed at the same time on the client machine to send concurrent SMTP/POP requests to the server. After 10 minutes we start "vmstat" on the server to obtain CPU and RAM load statistics:

sudo vmstat 60 30 > /var/log/mail.vmstat

This command will output CPU, Disk and RAM load statistics of the server for 30 minutes in one minute intervals and save them on the mail.vmstat file.

Once vmstat finishes we take the mail.log and mail.vmstat files and analyzed them using some custom ruby scripts.

Results


This graph shows the average number of successful SMTP transactions between Postfix (server) and postal (client) per minute. Each connection represents up to 3 messages with sizes between 0 and 5kbytes. No rocket science is needed to see that XFS outperforms by a large margin all other file systems. I managed to get a maximum of 1700 messages delivered per minute using XFS while JFS follows with about 900 messages per minute. ReiserFS and Ext3 in all jornaling modes never passed the 700 messages per minute rank.


This graph shows the average number of successful POP transactions between Courier-POP (server) and rabid (client) per minute. Each connection established by rabid will download and delete 5 messages from the server.

We can see here that contrary to SMTP we have ReiserFS and Ext3 being the best performers with very high averages. XFS and JFS do not perform that well but with 1800 and 1300 transactions per minute respectively they are not that bad.

The bad performance XFS and JFS get here agrees with the fact that they perform badly when deleting large amounts of small files. If we do not delete the messages from the server after download (or use IMAP instead of POP) we may get better performance statistics.


This graph shows the same as the first graph but instead of connections per minute we show the actual number of bytes that were transmitted per minute. This shows that we actually read/write messages from the disk.

Because we ran postal and rabid at the same time we do not have valid statistics for the number of bytes per minute downloaded via POP. This is because the POP client download rate is limited by the SMTP upload rate. In other words the POP client cannot download from the server more than what the SMTP client sends to the server.

This is easy to verify on the mail log files were you will see a lot of POP connections that retrieved zero bytes. Also you can verify this by graphing the number of bytes per minute for POP and you will see that all bars will be equal or slightly lower but never larger than the values obtained for SMTP.



We have here the CPU utilization (real) obtained by running "vmstat" on the server while it was receiving SMTP/POP requests. We see here that XFS is less CPU intensive that all other file systems except for JFS that outperforms all others by a large margin.

This result is surprising as all benchmarks state that XFS is the more CPU hungry of all the file systems. I took this CPU value from the vmstat output data by adding the user and system CPU utilization values. If this is not a good CPU utilization measurement please let me know.


I am not sure what this graph represents but from the "vmstat" man page this value represents: "wa: Time spent waiting for IO". Since we are interested here in file system performance I assume this value is important so I show it here for those that know what it is. To my understanding this represents average CPU time per minute wasted waiting for IO so high values are bad while low values are good. Comments on this graph are appreciated.

Conclusions

XFS simply rocks as a file system for SMTP and POP servers, period. The results I got showed that XFS is the best for delivery of large amounts of mails, it is good for POP and does not consume all CPU.

After Thoughts

These results show clearly that the file system used can greatly affect the applications performance (mail in this case). It can be interesting to test other applications that make heavy use of disk like File Servers and/or Database Servers or even test other SMTP (qmail) or POP (Cyrus, Dovecot) servers and compare their performance over different file systems and Postfix/Courier.

This is the first benchmark test so I am not expert and may have made mistakes. Any comments to improve or correct my tests are welcome.

References

[1] Best File System for Server Usage
[2] Basic SMTP Server Setup In Kubuntu
[3] Basic POP Server Setup In Kubuntu
[4] Postal - SMTP and POP benchmark program.