One of my mail servers failed when the ext3 partition that has the users maildirs ran out of I-Nodes. After this it was clear that ext3 is not scalable enough to handle large number of users, each one with large amounts of files/directories. This is a typical setting for a IMAP/SMTP service that allows users to organize their mails on the server rather than downloading them.
After a lot of research (Google) I found a lot of benchmarks and articles that made one thing clear: there is really no real silver bullet in the journaled file systems camp. For example ReiserFS is famous for being the best handling small files and it is true it does but it has really bad performance when handling big files. On the other hand XFS can handle massively large files really fast but has other problems as high CPU consumption and low performance deleting large amounts of files.
There are a lot more examples like these were one file system shines in one test but leaves a lot to desire on others.
After a week of deliberation I have decided to give XFS a try and use it from now on on all my future server installations.
Why XFS?
After looking at some benchmarks
here and here it is easy to see that XFS performs well or better in almost all tests. For those tests it is not the best it usually falls behind the best by a small margin.
The only benchmark it fails by a large margin is for deleting large number of files. This is one of the most criticized problems of XFS but there are tweaks to improve it (see below).
Another problem often criticized is that XFS seems to be CPU intensive when compared to other file systems. Looking at the benchmarks we can see this is true but we can also see that the high CPU usage is not the highest in all tests.
When evaluating journaled file systems there are a lot of features that must be accounted for. A very nice review of what too look can be found here and here.
The decision between XFS and JFS seems strange as JFS appears to be as good as XFS in all benchmarks using less CPU than XFS. The decision to go with XFS usually comes from the fact that it is more mature and has more features than JFS or other journaling file system in fact.
Some of the features that are important to servers that XFS supports are;
- Extended Inode Attributes: This allow use to add custom attributes to inodes giving use more granular control over the file systems. For example we could add tags to inodes and search files based on the tags.
- Online Defragmentation: Modern file systems are resistant to certain point to fragmentation of file but as the available free space is reduced the amount of fragmentation can increase to an alarming rate. The ability to defragment the disk without unmounting it is desirable for a high availability server
- Online backup utitlities: Again for high availability servers the capacity to backup and restore complete partitions is a plus. XFS has utilities that allow backup and restore of entire partitions online.
- Dynamic I-Node Allocation: This allows us to create large number of files in the file system without worries of running out of I-Nodes. All modern file systems, except for ext3, support this feature.
- Support For Quotas: Limiting the disk usage capacity in a multi user environment is very important to be able to offer services with varying levels of quality of service.
- Allocation Groups: As far as I know this is an XFS only feature that allows processes running concurrently to allocate space in the file system concurrently without interfering with each other. With this XFS enables free space and inode management operations to proceed in parallel throughout the file system increasing performance on large partitions. Current dual-core/multi-processor systems can take advantage of this feature to further increase overall system performance.
Some Remarks Against XFS
- XFS is very slow deleting large amounts of small files!!.
I haven't found yet a situation were I needed to actually delete a lot of small files from a mail server. On the other hand I do have found situations were I need to delete large files (i.e. 4GB database tables and 2GB log files). Also after reading this post it seems that it is possible to tweak XFS to perform even better than ext3 when deleting large number of files. - XFS only logs meta data on the journal so you will loose data on a crash
The only File System that is known to be robust to system crashes and sudden power outages is ext3 and only if it is in ordered or journal mode. All other file systems prefer to log only meta data to improve performance. You must decide between a robust but slower file system (i.e. ext3, ReiserFS) or a faster but less robust file system (i.e. XFS, JFS).
For a server the trend goes to speed performance rather than robustness to crashes and system failures. The Philosophy behind this trend is that if you are intend to install a server you surely have also a disaster recovery strategy. Having a nice backup/restore plan and more than one independent power source are also part of installing a high availability, high performance server. - If you need maildir storage then ReiserFS is the way to go!
Maybe but the benchmarks show that XFS is not that far behind ReiserFS when handling small files and at the same time is the best handling large files. ReiserFS may be well optimized to handle small files but it has really bad performance problems handling medium to large file sizes.
Also to note is that the incoming Reiser 4 is the worst performer in all speed tests and is even more CPU hungry than XFS. - What about JFS?? I heard it performs equal to or even better than XFS with less CPU load and more robust to sudden crashes!![/B]
Yes I heard it too and some benchmarks show that JFS can be as good or better than XFS in read/write performance. But as mentioned above JFS is relative new (in the linux camp) and is not as mature nor has as many features as XFS. For example it has some problems with user quotas.
So XFS is the all purpose File System
Sorry to say but XFS is no silver bullet here. As I said my conclusions are based on server application needs. I would not recommend using XFS or any other meta-data only journaling file system on a desktop computer, specially on a laptop that has more probability of power outages. The amount of complaints about all KDE configuration files being zero-length or filled with zeros after a crash talk by themselves (link).
XFS is not robust to crashes and power outages as ext3 can be so I will stay with ext3 for my desktop installations and my laptop. Ext3 is the recommended default by Ubuntu and with some tweaks it is possible to improve it's performance further (see here and here).
Future Work
All my conclusions are based on others hard worked results and opinions of others. To validate my conclusions I have collected a one week worth of qmail log files I am planning to use them to obtain mail usage statistics statistics and create some more realistic disk usage patterns. Current benchmarks are based on synthetic tests that may not reflect real load patterns in real mail or database applications,
Once I obtain some useful statistics from my logs I plan to perform some benchmarks over all file systems mentioned here with their respective performance tweaks. Once I get some results they will be posted here in this blog.
General Resources
http://linuxgazette.net/issue55/florido.html
http://bulma.net/impresion.phtml?nIdNoticia=1154
http://linux.inet.hr/first_benchmarks_of_the_ext4_file_system.html
http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://linuxgazette.net/122/piszcz.html
http://www.debian-administration.org/articles/388
http://adria.fesb.hr/~jsosic/mojbench.html
XFS Resources
http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://en.wikipedia.org/wiki/XFS
http://everything2.com/index.pl?node_id=1479435
https://launchpad.net/ubuntu/+bug/37435
http://www-128.ibm.com/developerworks/linux/library/l-fs10.html
http://www.ubuntuforums.org/showthread.php?t=246969
http://forums.gentoo.org/viewtopic-t-488215.html
https://ols2006.108.redhat.com/reprints/chinner-reprint.pdf
http://everything2.com/index.pl?node_id=1479435
ReiserFS Resources
http://www-128.ibm.com/developerworks/library/l-fs.html
http://www.moon-soft.com/program/bbs/readelite666510.htm
http://en.wikipedia.org/wiki/ReiserFS
http://en.wikipedia.org/wiki/Reiser4
Ext3 Resources
http://www-128.ibm.com/developerworks/linux/library/l-fs8.html
http://www.suseforums.net/index.php?showtopic=29773
http://forums.gentoo.org/viewtopic-t-488215.html
http://en.wikipedia.org/wiki/Ext3
http://en.wikipedia.org/wiki/Ext4
JFS Resources
http://www-128.ibm.com/developerworks/library/l-jfs.html
No comments:
Post a Comment