File System Benchmarks for Postfix Mail Server

After some research I decided to use XFS file system to host my next mail server (Postfix). This decision was based only on comments and suggestions I read in forums and some benchmarks published on the Internet about linux file systems [1].

I installed my new mail server with Postfix as MTA [2] and Courier as POP [3] and leaved a free 60GB partition to make some benchmark testing. This way I was able to extract some hard numbers to validate my desicion. The results show XFS is indeed the best option when running a mail server, at least for Postfix.

Test Environment

The test setup is very simple: I have one server with Ubuntu Server 6.06 LTS installed with the most recent updates and a second client machine with Kubuntu Feisty running Postal/Rabid [4] software used to load the server with SMTP/POP requests.

The machines specifications are:

	Server Machine	Client Machine
OS	Ubuntu Server 6.06 LTS	Kubuntu Feisty Fawn
CPU	Intel Xeon 2.40GHz	Intel Pentium III 800MHz
RAM	512MB	256MB
Disk	SEAGATE ST373307LC 73408MB	Maxtor 6Y120L0 SCSI 122942 MB

In the server machine I had Ubuntu Server installed with Postfix and Courier-POP configured and added 500 user accounts with Maildir boxes.

The client side is my desktop PC running Kubuntu Feisty where I simply run postal and rabid to load the server with a lot of SMTP and POP requests. I apply no limits on the number of requests per minute as we are testing the disk input/output throughput rather than the mail service itself.

Testing Procedure

On the server side I followed these steps:

• Create test partition with one of the file systems under consideration. Especial care was taken to use the creation options that are known to increase each particular file system performance as stated in various benchmarks and forums.
  
  
  
  • mkfs.ext3 -J size=100 -m 1 -O dir_index,filetype,has_journal
  
  
  • mkfs.reiserfs -b 4096 -s 16386
  
  
  • mkfs.xfs -f -l size=64m -d agcount=16
  
  
  • mkfs.jfs -s 64
  
  
  
  


• Mounted the newly created partition to a testing mount point. Again I tried my best to use the recommended mount options in order to improve each file system performance. In the case of ext3 and reiserfs we tested all three journaling methods they support.
  
  
  
  • mount -t ext3 -o noatime,nodiratime,data=journal
  
  
  • mount -t ext3 -o noatime,nodiratime,data=ordered
  
  
  • mount -t ext3 -o noatime,nodiratime,data=writeback
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=journal
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=ordered
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=writeback
  
  
  • mount -t xfs -o noatime,nodiratime,logbufs=8
  
  
  • mount -t jfs
  
  
  
  


• Copied the necessary files from the root partition to the testing partition to create a fully functional chroot environment and chrooted it:
  


• Once inside the chroot environment I start all services. First deleting all the syslogs and then starting sysklogd, postfix, courier-authdaemon and courier-pop in that order. This way all mailboxes, logs and configuration files are read/write in the file system under test.

On the client side I simply start postal and rabid to send unlimited SMTP and POP requests to the server:

postal -m 5 -r 24000 -t 5 -c 3 -s 0 server_ip userlist.txt
rabid  -p 5 -c 5 -r 24000 -s 0 server_ip userlist.txt

The userlist.txt contains the list of all 500 user accounts with password that were created on the server after installation. The postal command above will establish five SMTP connections to the server and send as much messages as it can with sizes between 0 and 5kbytes to all 500 users. On the other side rabid will establish as much POP connections as it cans to the sever from five different processes and download a maximum of five messages per connection.

Both postal and rabid were executed at the same time on the client machine to send concurrent SMTP/POP requests to the server. After 10 minutes we start "vmstat" on the server to obtain CPU and RAM load statistics:

sudo vmstat 60 30 > /var/log/mail.vmstat

This command will output CPU, Disk and RAM load statistics of the server for 30 minutes in one minute intervals and save them on the mail.vmstat file.

Once vmstat finishes we take the mail.log and mail.vmstat files and analyzed them using some custom ruby scripts.

Results


This graph shows the average number of successful SMTP transactions between Postfix (server) and postal (client) per minute. Each connection represents up to 3 messages with sizes between 0 and 5kbytes. No rocket science is needed to see that XFS outperforms by a large margin all other file systems. I managed to get a maximum of 1700 messages delivered per minute using XFS while JFS follows with about 900 messages per minute. ReiserFS and Ext3 in all jornaling modes never passed the 700 messages per minute rank.


This graph shows the average number of successful POP transactions between Courier-POP (server) and rabid (client) per minute. Each connection established by rabid will download and delete 5 messages from the server.

We can see here that contrary to SMTP we have ReiserFS and Ext3 being the best performers with very high averages. XFS and JFS do not perform that well but with 1800 and 1300 transactions per minute respectively they are not that bad.

The bad performance XFS and JFS get here agrees with the fact that they perform badly when deleting large amounts of small files. If we do not delete the messages from the server after download (or use IMAP instead of POP) we may get better performance statistics.


This graph shows the same as the first graph but instead of connections per minute we show the actual number of bytes that were transmitted per minute. This shows that we actually read/write messages from the disk.

Because we ran postal and rabid at the same time we do not have valid statistics for the number of bytes per minute downloaded via POP. This is because the POP client download rate is limited by the SMTP upload rate. In other words the POP client cannot download from the server more than what the SMTP client sends to the server.

This is easy to verify on the mail log files were you will see a lot of POP connections that retrieved zero bytes. Also you can verify this by graphing the number of bytes per minute for POP and you will see that all bars will be equal or slightly lower but never larger than the values obtained for SMTP.



We have here the CPU utilization (real) obtained by running "vmstat" on the server while it was receiving SMTP/POP requests. We see here that XFS is less CPU intensive that all other file systems except for JFS that outperforms all others by a large margin.

This result is surprising as all benchmarks state that XFS is the more CPU hungry of all the file systems. I took this CPU value from the vmstat output data by adding the user and system CPU utilization values. If this is not a good CPU utilization measurement please let me know.


I am not sure what this graph represents but from the "vmstat" man page this value represents: "wa: Time spent waiting for IO". Since we are interested here in file system performance I assume this value is important so I show it here for those that know what it is. To my understanding this represents average CPU time per minute wasted waiting for IO so high values are bad while low values are good. Comments on this graph are appreciated.

Conclusions

XFS simply rocks as a file system for SMTP and POP servers, period. The results I got showed that XFS is the best for delivery of large amounts of mails, it is good for POP and does not consume all CPU.

After Thoughts

These results show clearly that the file system used can greatly affect the applications performance (mail in this case). It can be interesting to test other applications that make heavy use of disk like File Servers and/or Database Servers or even test other SMTP (qmail) or POP (Cyrus, Dovecot) servers and compare their performance over different file systems and Postfix/Courier.

This is the first benchmark test so I am not expert and may have made mistakes. Any comments to improve or correct my tests are welcome.

References

[1] Best File System for Server Usage
[2] Basic SMTP Server Setup In Kubuntu
[3] Basic POP Server Setup In Kubuntu
[4] Postal - SMTP and POP benchmark program.

Use time command to get CPU usage too.

The linux time command is a nice little utility that can tell you how much time (cpu and real time) a command or program takes to finish. For example to find out how much time it takes in your computer to find certain file on you home directory you can use:

time find /home -name "filename"

real    0m16.365s
user    0m0.208s
sys     0m0.492s

The time command simply receives whatever program you pass it along with all the parameters you give it and gives you back some statistics about the program execution.

If you read the manual page you can see that time can tell you a lot more than just excecution time of program/commands.

In BASH the way to make time give you other values in other formats it to set up the "TIMEFORMAT" environment variable before calling time. For example to get the above time values plus the CPU percentage we use:

export TIMEFORMAT="%E real,%U user,%S sys, %P cpu"

time find /home -name "*.txt"  

7.184 real,0.292 user,0.504 sys, 11.08 cpu

Note that the man page method (i.e. -f switch) does not work with the version of Bash that comes with Ubuntu Server and Kubuntu Edgy/Feisty.

Linux Mail Server Setup

Foreword

As system administrator I have had the privilege of working with all sorts of linux servers and services in a variety of distributions. Here I present two configurations I have used in most of my installations.

The first one is based on qmail SMTP with virtual domain support with VPopmail/MySQL on RedHat ES. This configuration I used mostly as a system admin on a cable modem company that handled thousands of residential users and hundreds of commercial users (i.e. with their own email domains). I can say this setup is scalable in terms of users but may present some problems with large queues caused by large traffic bursts. Also due to the maildir format I had some problems with inode exhausting but this is a file system problem (i.e. using XFS instead of ext3 could have avoided this problem).

The second configuration is more updated with current standards. It uses the more easy to configure and secure Postfix for SMTP and Courier-POP for POP. It could have used other POP servers like Dovecot. I have tested this configuration on a small laboratory with 30 users so no idea on how scalable it may be.

I do not support virtual domains in this configuration because I consider current solutions to email virtual domains confusing. With the virtualization hype these days I believe SMTP servers should not handle virtual mail domains. We should install several virtual machines in a single hardware and manage each mail domain with a fully functional SMTP server in each of the virtual machines.

Redhat Eterprise Server (ES) Mail setup

Pros:

• Fully functional qmail SMTP and POP services.


• Supports virtual domains with VPopmail/MySQL.


• Maildir storage format.

Cons:

• Installation from source code and requires some source modifications.


• Rather complicated manual configurations.


• No support for encrypted communications. This limits the server to local network or locahost connections (i.e. Webmail client).


• Roaming support uses the outdated POP-Before-SMTP.

Instructions:

• Setup qmail server
  First we install the qmail SMTP server on RedHat ES 4 from source code. Due to some incompatibilities with the version of glibc that comes with RedHat we are forced to modify the source code a little to get it to compile.


• Setting TCPSERVE
  We use tcpserve to handle all incoming connections to the qmail SMTP port (25). With tcpserve we can control access to the port and allow/deny relaying of emails to users depending on IP address. This access control is needed to avoid spammers from using our server for free.


• VPopmail
  We end our setup by installing Vpopmail to support virtual domains managed with MySQL backend. We configure Vpopmail to support maildirs and to manage roaming. Vpopmail modifies the tcpserve tcp rules files to allow POP authenticated users to relay mail (POP-Before-SMTP).
  

Ubuntu Mail Server Setup (Kubuntu Feisty Fawn/Ubuntu Server)

Pros:

• Fully functional Postfix SMTP and Courier-POP services.


• Installation and configuration is a breeze (soooo easy).


• Maildir storage format.


• Roaming using SMTP-AUTH.


• Encrypted transmission for SMTP (TLS) and POP (SSL).

Cons:

• No support for virtual domains.

Instructions:

• Installing the SMTP server
  First we install and configure a fully functional Postfix server and enable SMTP-AUTH for roaming with SASL authentication.
  
• Installing the POP sever
  Next we install the Courier-POP server with Maildir storage support.


• Enable Encrypted Communications
  Now we enable secure transmission by encryption of SMTP (TLS) and POP (SSL) traffic. We create self signed certificates and enable Postfix and Courier-POP to use them.


• Testing
  Finally some pointers on how to test SMTP/POP services using a console window via telnet/openssl commands.

Sony NW-A808 PMP Linux review

I recently got this little piece of hardware and I really like it. It supports video, audio and images in a very small and stylish pack. I was deciding between the NW-A808 and the iRiver X20 but finally decided to go with the NW-A808 mostly for the cool factor. The iRiver X20 is bigger and has a plastic case that makes it feel like a toy. The NW-A808 has a very slim and metallic case (iPod like) that feels just right on my hand. Also the NW-A808 has a nice variety of colours to choose while the X20 comes only with a black version.

The NW-A808 GUI supports various languages so I can set it up in English or Spanish if I want instead of the default Japanese. It supports album covers, video thumbnails and has a lot of play list features like search, bookmarks, and much more.

The good and the bad

The good thing is that this device supports MSC (aka UMS) that is very well supported by Linux. Plugging the player to my laptop running Kubuntu Feisty I get a pop-up window asking me if I want to open it in a new window.

Opening the device (/media/WALKMAN) in Konqueror shows me the content of the device and I can easily copy/delete files into it. It is possible to add images and videos to the device by directly copying them into the flash memory.

The bad thing is that first: it does not support Ogg and second: that to add music files to the device we are forced to use a crappy piece of proprietary software called Sonic Stage on Windows. I haven't found yet any Linux alternatives to replace Sonic Stage but rest assured that someone (maybe me), someday (far in the future) will take the task to make this device Linux friendly.

Uploading Images

To upload your images to the NW-A808 flash memory you MUST create sub folders inside the DCIM directory that is in the flash memory and copy your images inside those sub folders. You can copy them directly inside the DCIM folder but they will not get listed on the device library browser.

From the instruction manual the images format supported is:

• File Format: DCF 2.0/Exif 2.21


• Extension: .jpg


• Codec: JPEG(Baseline)


• Max Resolution: 4,000x4,000 (16 Megapixels)


• Quantity: Maximum 10,000 files

Uploading Videos

As with the photos we only need to copy our video files in mp4 format inside the VIDEO folder that is inside the device flash memory. Pay attention to the supported video format as any file not supported may be listed in the video library but the player will refuse to play it.

• File Format: MP4


• Extension: .mp4, .m4v


• Video Codecs:
  
  
  
  • MPEG-4
    
    
    
    • Profile: Simple Profile
    
    
    • Max bitrate: 2,500 kbps
    
    
    • Max fps: 30 fps
    
    
    • Max resolution : 320x240(QVGA)
    
    
    
    
  
  
  • AVC (H.264/AVC)
    
    
    
    • Profile: Baseline Profile level 1.2 and 1.3
    
    
    • Max bitrate: 768 kbps
    
    
    • Max fps: 30 fps
    
    
    • Max resolution : 320x240(QVGA)
    
    
    
    
  
  
  
  


• Audio Codecs: AAC-LC
  
  
  
  • Number of Channels: 2
  
  
  • Samplig rates: 24, 32, 44.1 and 48 kHz
  
  
  • Bitrate: 288kbps per channel
  
  
  
  

Based on the video/audio specifications above we can use the all mighty ffmpeg to convert almost any video to a format that can be played on the device:
Since this is a device created by Sony I concluded it supports the same video format the PSP does . A simple Google search for "mencoder PSP" led me to this post with the following command:

ffmpeg -y -i $infile -vcodec libx264 -b 768k -bt 768k -level 13 \
-r 30 -croptop auto -cropbottom auto -s 320x240 -acodec libfaac \
-ar 48000 -ab 100k -ac 2 -f mp4 $outfile.mp4

With this command I was able to convert most of my mpg and avi files to mp4 and play them in the NW-A808. Once you transcode you videos copy them to the Videos directory inside the device flash and they will be listed on the video library.

There is a little functionality drawback if we copy files directly to the flash. The NW-A808 displays the video file name and a nice THM thumbnail in the video browser when the files are copied using Sony's proprietary software. I have tried in several ways to create the thumbnail file manually without success so far.

Filenames and Folders



It is possible to copy images and videos in any folder inside the device flash. It is also possible to create sub folders in any place and put files inside those sub folders. But only the images and videos that are inside the DCIM and VIDEO folders respectively will get listed on the device browser. See the image above to get an idea of the folder layout.

You can copy all video files directly inside the VIDEO folder in flash or create sub folders (one level deep only) and put the video files on those sub folders.

For images you cannot directly put the files inside the DCIM directory. You must create sub folders (one level deep) inside the DCIM folder and copy the images inside those sub folders.

Also make sure the extensions are jpg for images and mp4/m4v for video or the player won't list them.

Uploading Audio Files

Now the biggest drawback of this and all other Network Walkman devices produced by Sony. To upload music files we must use Sonic Stage on windows because simply copying the music files on the device won't list them on the album browser.

So far the closest I have found to make this device work on Linux is the libnw library created by Waider. Too bad this library does not work on my device but Waider has shown interest in modifying his library to support the NW-A808. Once libnw supports this device it won't be difficult to write an Amarok MediaDevice interface to manage it.

Sony uses GPL software?

Not really sure why but inside the NW-A808 box came a copy of the GPLv2 licence. It means (maybe?) that the software provided by Sony to handle the deviceuses some GPLed software/libraries.

I am no lawyer but as I understand if this is true then isn't Sony supposed to release the source code of such applications?. With a source code it would be easier to implement a Linux alternative to make these devices more Linux friendly.

Best File System for Server Usage

Problem Review

One of my mail servers failed when the ext3 partition that has the users maildirs ran out of I-Nodes. After this it was clear that ext3 is not scalable enough to handle large number of users, each one with large amounts of files/directories. This is a typical setting for a IMAP/SMTP service that allows users to organize their mails on the server rather than downloading them.

After a lot of research (Google) I found a lot of benchmarks and articles that made one thing clear: there is really no real silver bullet in the journaled file systems camp. For example ReiserFS is famous for being the best handling small files and it is true it does but it has really bad performance when handling big files. On the other hand XFS can handle massively large files really fast but has other problems as high CPU consumption and low performance deleting large amounts of files.

There are a lot more examples like these were one file system shines in one test but leaves a lot to desire on others.

After a week of deliberation I have decided to give XFS a try and use it from now on on all my future server installations.

Why XFS?

After looking at some benchmarks
here and here it is easy to see that XFS performs well or better in almost all tests. For those tests it is not the best it usually falls behind the best by a small margin.

The only benchmark it fails by a large margin is for deleting large number of files. This is one of the most criticized problems of XFS but there are tweaks to improve it (see below).

Another problem often criticized is that XFS seems to be CPU intensive when compared to other file systems. Looking at the benchmarks we can see this is true but we can also see that the high CPU usage is not the highest in all tests.

When evaluating journaled file systems there are a lot of features that must be accounted for. A very nice review of what too look can be found here and here.

The decision between XFS and JFS seems strange as JFS appears to be as good as XFS in all benchmarks using less CPU than XFS. The decision to go with XFS usually comes from the fact that it is more mature and has more features than JFS or other journaling file system in fact.

Some of the features that are important to servers that XFS supports are;

• Extended Inode Attributes: This allow use to add custom attributes to inodes giving use more granular control over the file systems. For example we could add tags to inodes and search files based on the tags.


• Online Defragmentation: Modern file systems are resistant to certain point to fragmentation of file but as the available free space is reduced the amount of fragmentation can increase to an alarming rate. The ability to defragment the disk without unmounting it is desirable for a high availability server


• Online backup utitlities: Again for high availability servers the capacity to backup and restore complete partitions is a plus. XFS has utilities that allow backup and restore of entire partitions online.


• Dynamic I-Node Allocation: This allows us to create large number of files in the file system without worries of running out of I-Nodes. All modern file systems, except for ext3, support this feature.


• Support For Quotas: Limiting the disk usage capacity in a multi user environment is very important to be able to offer services with varying levels of quality of service.


• Allocation Groups: As far as I know this is an XFS only feature that allows processes running concurrently to allocate space in the file system concurrently without interfering with each other. With this XFS enables free space and inode management operations to proceed in parallel throughout the file system increasing performance on large partitions. Current dual-core/multi-processor systems can take advantage of this feature to further increase overall system performance.
  

Some Remarks Against XFS

• XFS is very slow deleting large amounts of small files!!.
  
  
  I haven't found yet a situation were I needed to actually delete a lot of small files from a mail server. On the other hand I do have found situations were I need to delete large files (i.e. 4GB database tables and 2GB log files). Also after reading this post it seems that it is possible to tweak XFS to perform even better than ext3 when deleting large number of files.
  



• XFS only logs meta data on the journal so you will loose data on a crash
  
  
  The only File System that is known to be robust to system crashes and sudden power outages is ext3 and only if it is in ordered or journal mode. All other file systems prefer to log only meta data to improve performance. You must decide between a robust but slower file system (i.e. ext3, ReiserFS) or a faster but less robust file system (i.e. XFS, JFS).
  
  For a server the trend goes to speed performance rather than robustness to crashes and system failures. The Philosophy behind this trend is that if you are intend to install a server you surely have also a disaster recovery strategy. Having a nice backup/restore plan and more than one independent power source are also part of installing a high availability, high performance server.
  



• If you need maildir storage then ReiserFS is the way to go!
  
  
  Maybe but the benchmarks show that XFS is not that far behind ReiserFS when handling small files and at the same time is the best handling large files. ReiserFS may be well optimized to handle small files but it has really bad performance problems handling medium to large file sizes.
  
  Also to note is that the incoming Reiser 4 is the worst performer in all speed tests and is even more CPU hungry than XFS.
  



• What about JFS?? I heard it performs equal to or even better than XFS with less CPU load and more robust to sudden crashes!![/B]
  
  Yes I heard it too and some benchmarks show that JFS can be as good or better than XFS in read/write performance. But as mentioned above JFS is relative new (in the linux camp) and is not as mature nor has as many features as XFS. For example it has some problems with user quotas.
  

So XFS is the all purpose File System

Sorry to say but XFS is no silver bullet here. As I said my conclusions are based on server application needs. I would not recommend using XFS or any other meta-data only journaling file system on a desktop computer, specially on a laptop that has more probability of power outages. The amount of complaints about all KDE configuration files being zero-length or filled with zeros after a crash talk by themselves (link).

XFS is not robust to crashes and power outages as ext3 can be so I will stay with ext3 for my desktop installations and my laptop. Ext3 is the recommended default by Ubuntu and with some tweaks it is possible to improve it's performance further (see here and here).

Future Work

All my conclusions are based on others hard worked results and opinions of others. To validate my conclusions I have collected a one week worth of qmail log files I am planning to use them to obtain mail usage statistics statistics and create some more realistic disk usage patterns. Current benchmarks are based on synthetic tests that may not reflect real load patterns in real mail or database applications,

Once I obtain some useful statistics from my logs I plan to perform some benchmarks over all file systems mentioned here with their respective performance tweaks. Once I get some results they will be posted here in this blog.

General Resources

http://linuxgazette.net/issue55/florido.html
http://bulma.net/impresion.phtml?nIdNoticia=1154
http://linux.inet.hr/first_benchmarks_of_the_ext4_file_system.html
http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://linuxgazette.net/122/piszcz.html
http://www.debian-administration.org/articles/388
http://adria.fesb.hr/~jsosic/mojbench.html

XFS Resources

http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://en.wikipedia.org/wiki/XFS
http://everything2.com/index.pl?node_id=1479435
https://launchpad.net/ubuntu/+bug/37435
http://www-128.ibm.com/developerworks/linux/library/l-fs10.html
http://www.ubuntuforums.org/showthread.php?t=246969
http://forums.gentoo.org/viewtopic-t-488215.html
https://ols2006.108.redhat.com/reprints/chinner-reprint.pdf
http://everything2.com/index.pl?node_id=1479435

ReiserFS Resources

http://www-128.ibm.com/developerworks/library/l-fs.html
http://www.moon-soft.com/program/bbs/readelite666510.htm
http://en.wikipedia.org/wiki/ReiserFS
http://en.wikipedia.org/wiki/Reiser4

Ext3 Resources

http://www-128.ibm.com/developerworks/linux/library/l-fs8.html
http://www.suseforums.net/index.php?showtopic=29773
http://forums.gentoo.org/viewtopic-t-488215.html
http://en.wikipedia.org/wiki/Ext3
http://en.wikipedia.org/wiki/Ext4

JFS Resources

http://www-128.ibm.com/developerworks/library/l-jfs.html

Testing SMTP and POP services from a console

Test a non-secure SMTP server

To test a non-secure SMTP server we simply connect to the port 25 of that machine using telnet:

telnet smtp_ip_address 25
Trying smtp_ip_address...
Connected to smtp_ip_address.
Escape character is '^]'.
220 Welcome to Postfix ESMTP Server
ehlo localhost
250-testmail
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from:auser@internaldomain.org
250 2.1.0 Ok
rcpt to:noone@externaldomain.org
554 5.7.1 : Relay access denied
rcpt to:noone@internaldomain.com
550 5.1.1 : Recipient address rejected: User unknown in local recipient
rcpt to:realuser@internaldomain.com
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
Hello There!!
.
250 2.0.0 Ok: queued as 7ECF05100C6
quit
221 2.0.0 Bye
Connection closed by foreign host.

The green text are the commands you send to the server while the blue text indicates the server responses. The example above is for a simple Postfix SMTP server.

Testing the AUTH command

From the previous example we can see that the Postfix SMTP server supports the AUTH command that allows roaming users to use the server as relay. To test this command we must first encode our authentication credentials in base64 format:

perl -MMIME::Base64 -e 'print encode_base64("\000username\@domain.net\000password")'

You must have the MIME::Base64 perl module in your system (is installed by default in Kununtu). Also make sure you replace the username, domain.net and password in the command to reflect your own valid values.

After the command finishes it will output your authentication credentials as a base64 encoded string that looks like:

AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=

Copy that string somewhere and proceed to connect to the server as in the previous example:

telnet smtp_ip_address 25
Trying smtp_ip_address...
Connected to smtp_ip_address.
Escape character is '^]'.
220 Welcome to Postfix ESMTP Server
ehlo localhost
250-testmail
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
auth plain AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=
235 2.0.0 Authentication successful
mail from:auser@internaldomain.org
250 2.1.0 Ok
rcpt to:noone@externaldomain.org
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
hahahaha
.
250 2.0.0 Ok: queued as 7B6265100C6
quit
221 2.0.0 Bye
Connection closed by foreign host.

In this example we used the "auth plain" command to authenticate ourselves against the SMTP server. Once authenticated we are allowed to relay mails to any domain (not only internal domains) from anywhere (roaming).

A closer look at the the previous example will reveal that when we send the "rcpt to" command to an external domain (noone@externaldomain.org) the server response was a "554 relay access denied". In this example we are authenticated with the server (via the AUTH command) so now the "rcpt to" command was accepted.

Test a non-secure POP server

Similar to SMTP we can test a POP server by connecting to it using telnet:

telnet pop_ip_address 110
Trying pop_ip_address...
Connected to pop_ip_address.
Escape character is '^]'.
+OK Hello there.
user username
+OK Password required.
pass password
+OK logged in.
list
+OK POP3 clients that break here, they violate STD53.
1 520
.
quit
+OK Bye-bye.
Connection closed by foreign host.

Again green are your commands and blue are the server responses. In this example we simply connect to the server, authenticate and get the list of mails in our inbox. We can also see the contents of the mails with the "retr num" command replacing num with the mail number as displayed by the "list" command.

Test secure SMTP/POP servers

If the SMTP/POP server supports encryption (TLS/SSL) we won't be able to test it using simple text telnet connections. In this case we must use the openssl command utility that will take care of all TLS/SSL negotiations, connect to the server and allow us to send plain text commands over the encrypted connection.

For SMTP we can check if the server support TLS encryption by looking at the server response to the "ehlo" command. If we get a "250-STARTTLS" within all the responses then the server supports TLS.

To connect to the SMTP server over a TLS encrypted connection we can use the following command:

openssl s_client -starttls smtp -crlf -connect smtp_ip_address:25

You will get a lot of messages from this command that show the SSL negotiations. You really don't need to care about all those messages. When openssl finishes setting up the encrypted connection you will see the same welcome message we got on the first example:

220 Welcome to Postfix ESMTP Server

From here on you can proceed with plain text commands as in the first two examples.

In the case of POP the server it is a little more difficult to check if it supports TLS/SSL connections. POP uses a different port (995 instead of 110) for SSL encrypted connections. This means that as long as the server uses the default ports if the server has the port 995 open we can assume it supports POP over SSL.

To connect to the POP server we must use the openssl utility command to take care of all the SSL negotiation procedures. The following command will do the trick:

openssl s_client  -crlf -connect pop_ip_address:995

After all the SSL certificates are exchanged and the encrypted connection is established we are greeted by the POP server hello message as in the previous POP example:

+OK Hello there.

From this point on we can proceed issuing normal text commands and getting server responses. All the encryption part will be handled automatically by openssl.

References

I learned about testing the AUTH command and SMTP over TLS from the following page: http://qmail.jms1.net/test-auth.shtml

Add encryption to SMTP and POP (TLS/SSL)

In previous posts we configured a fully functional SMTP and POP servers for a single domain with local system users and with roaming capabilities using SASL authentication.

In this post we further improve our setup to allow encrypted communications in SMTP and POP. This way we add a security layer to protect users from mishap from third persons and protect sensitive information.

Creating the SSL certificate for SMTP

The recommended way to do this is to create a certificate request and get it signed by a Certificate Authority but for demonstration purposes I will create a self signed certificate.

For postfix we must create a cert/key pair using the openssl command. In your home directory or a temporary directory you can create the SSL cert/key pair like this:

openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 \
        -nodes -keyout smtpd.key -keyform PEM -days 1095 -x509

You will be asked a set of questions that you can fill any way you feel like. This is only to create an identity that must be unique among certificates. Note that the above is a one line command split in two due to space constraints.

Next we create some directories to store the SSL certificates. I like to store them inside the postfix configuration directory.

sudo mkdir /etc/postfix/ssl
sudo mkdir /etc/postfix/ssl/cert
sudo mkdir /etc/postfix/ssl/private

sudo chmod 755 /etc/postfix/ssl/cert
sudo chmod 710 /etc/postfix/ssl/private

Then we move the certificate pair to the newly created directories:

sudo mv smtpd.cert /etc/postfix/ssl/cert
sudo mv smtpd.key /etc/postfix/ssl/private

Enabling TLS in Postfix

Simply edit the /etc/postfix/main.cf file and add the following:

###
# TLS parameters

smtpd_tls_cert_file=/etc/postfix/ssl/cert/smtpd.cert
smtpd_tls_key_file=/etc/postfix/ssl/private/smtpd.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache
smtpd_tls_auth_only = no

###
# END

The "smtpd_tls_auth_only" option can be used for added security. It forces the communication the be encrypted before any credentials (user and passwords) are transmitted. Unfortunately some client's may fail to authenticate with this setting so it defaults to "no".

Finally restart postfix:

sudo /etc/init.d/postfix restart

Enable POP over SSL

If you followed my previous post then you already have installed the required courier-pop-ssl package. If not simply install it with this command:

sudo aptitude install courier-pop-ssl

Generating the SSL certificate for POP

By default Kubuntu uses an auto generated SSL certificate (pop3d.pem) that you can find inside the "/etc/courier" directory. The pop3d.pem file is actually a cert/key pair combined into a single file. For our POP server we will use the same cert/key pair we created for the SMTP server to generate our pop3d.pem file.

In your home or a temporary directory:

sudo cat /etc/postfix/ssl/cert/smtpd.cert > pop3d.pem
sudo openssl gendh >> pop3d.pem
sudo cat /etc/postfix/ssl/private/smtpd.key >> pop3d.pem
sudo chmod 600 pop3d.pem

Now backup the auto generated certificate and replace it with the new one:

sudo cp /etc/courier/pop3d.pem /etc/courier/pop3d.pem.orig
sudo rm /etc/courier/pop3d.pem
sudo cp pop3d.pem /etc/courier

You can put the generated pop3d.pem file anywhere you want as long as you change the path in the courier configuration file (i.e. /etc/courier/pop3d-ssl). For security reasons it may be better to keep it in the default place.

Finally restart the service but this time the SSL one.

sudo /etc/init.d/courier-pop-ssl restart

To force your users to use always SSL simply disable the normal POP server and leave only the one with SSL.

sudo /etc/init.d/courier-pop stop
sudo /etc/init.d/courier-pop-ssl restart

Also to stop courier-pop from starting at boot time we must remove the start up script from the rcX.d directories with the following command:

sudo update-rc.d -f courier-pop remove

This way when the server reboots only the SSL enable POP service will be started. If for some reason you require to start the normal POP service at boot time again you can do it with the command:

sudo update-rc.d  courier-pop defaults

Maildir

Do not forget to modify the "/etc/courier/pop3d-ssl" configuration file in Kubuntu Feisty to use ".maildir" instead of the "Maildir" for storage of mails. As with normal POP simply edit the file and replace the value of the MAILDIRPATH variable.

In the case of Ubuntu Server Edition all these services (POP, POP-SSL, IMAP, IMAP-SSL) use the configuration inside "/etc/default/courier" to determine the maildir's directory. Simply make sure you modify the file and replace the MAILDIR value with ".maildir"

Testing

In previous posts we were able to test our configurations by connecting via telnet to the corresponding ports. Now that we have enabled encryption we cannot test with simple text commands via telnet.

It is still possible to test the SMTP and POP services over encrypted connections in a console using the openssl utility but I will explain how to do this in a future post.

Basic POP Server Setup in Kubuntu (Courier)

In a previous post I explained how to set up a very basic SMTP server to deliver mails to users local maildir's and to relay mails to external domains (only for authorized users).

Now to allow users to download/view their emails in their own PC's we must enable a POP or IMAP service. I will only deal with POP here but since the configuration procedure for POP and IMAP in Kubuntu is very similar everything said in this post for POP applies with minor changes to IMAP.

Installing the Software

In Kubuntu installing the courier POP servers cannot be easier:

sudo aptitude install courier-pop courier-pop-ssl

Note that we are also installing the courier-pop-ssl package that allows POP over SSL connections. I will talk more about SSL in a future blog post. During the installation you will be asked if you want directory based configuration files, chose not to and then press ok when asked.

Creating Maildirs

The maildir vs mbox formats to store emails is an old debate and I will not deal with it here. I like maildir storage simply because I had a lot of lock and corrupted file problems in the past due to the "single file" mbox format and migrating to maildir format kept those problems away.

By default Courier-POP in Kubuntu uses the maildir format to store users mail. The only problem I have with the default settings is that it uses a folder called "Maildir" inside each user home directory. I personally prefer to hide the mail's directory from the users to avoid accidents (i.e. total deletion).

Now to enable each user's maildir we must first create the maildir directory:

sudo maildirmake /home/username/.maildir
sudo chown -R username:usergroup /home/username/.maildir
sudo chmod -R 700 /home/username/.maildir

You can replace .maildir with whatever you like. The dot "," at the begining makes that directory hidden so it would not be listed normally in the file manager (i.e. Konqueror) or by issuing "ls" on the console.

Creating the .maildir directory must be done for all current users in the system. For new users the maildir creation can be automated by creating a maildir directory inside the "/etc/skel" template directory:

sudo maildirmake /etc/skel/.maildir

Now every time a new user is created the .maildir folder will be automatically created in that user's home directory.

Configuring Courier POP

The default settings that come with Kubuntu are enough to get the services running. The only change we have is to replace the "Maildir" directory with ".maildir". This is done easily by editing the "/etc/courier/pop3d" configuration file:

sudo kate /etc/courier/pop3d   

    and

replace MAILDIRPATH=Maildir with MAILDIRPATH=.maildir

In the case of Ubuntu Server the above configuration won't work. Instead of modifying the /etc/courier/pop3d file you must edit the /etc/default/courier file:

sudo kate /etc/default/courier   

    and

replace MAILDIR=Maildir with MAILDIR=.maildir

Restart and Test

Finally we restart the POP/IMAP service:

sudo /etc/init.d/courier-pop restart

And to test it we connect to the port 110 using telnet:

telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK Hello there.
user username
+OK Password required.
pass userpass
+OK logged in.
list
+OK POP3 clients that break here, they violate STD53.
.
quit
+OK Bye-bye.
Connection closed by foreign host.

As in the previous post the blue text are server response messages and the green text are your input commands. To authenticate you simply use the "user" and "pass" commands. Once authenticated you can list your emails with the "list" command and exit with the "quit" command.

Coming Next

In the old days our server is now ready to go online but these days this configuration would be considered insecure. Current state of the art in servers must provide security mechanisms like encrypted communications to avoid leakage of sensitive/personal information.

In my next post I will explain how to encrypt SMTP and POP using TLS/SSL to have a more secure server ready for the current Internet.

Basic SMTP Server Setup in Kubuntu (Postfix)

Kubuntu is mostly a desktop oriented linux distribution, but this does not mean that it cannot be used as a server. Several guides exist on the Ubuntu Wiki that teach you how to setup different kinds of servers.

This small tutorial will show you how to set up a local SMTP server (Postfix) for a small office or school with a single domain.

This configuration was tested on Kubuntu Feisty Fawn and in Ubuntu Server distributions. This configuration may work also with Dapper and with Edgy but I have not tested it.

Installing Software

In a fresh installation of Kubuntu simply install the postfix and support packages:

sudo aptitude install postfix libsasl2 sasl2-bin libsasl2-modules libdb3-util

Note that the above command installs SASL too that will be used to enable roaming in our server. During the installation of postfix the package manager will give you the opportunity to configure the most basic options of postfix. I simply select not to configure it at the time and latter modify the configurations files with a text editor. This way I get more control on the look of the file and can format it in a easy to read layout.

Configuring the server

The main configuration file main.cf of postfix can be found inside the "/etc/postfix" directory. One of the selling points of Postfix when compared to other SMTP servers like qmail and sendmail is the simplicity of it's configuration file as we will see.

Edit the file with your favorite editor:

sudo vi /etc/postfix/main.cf

   or

sudo kate /etc/postfix/main.cf

   or (if you prefer Gnome)

sudo gedit /etc/postfix/main.cf

Simply delete everything in there and copy/paste the following configuration. Make sure to read the comments and replace setting values to reflect your domain name, IP addresses, etc.

###
# Search hostnames in the hosts file before querying DNS.
# This option is very when trying a new domain that has not 
# been published by your DNS server or when testing a replacement
# server.

smtp_host_lookup = native,dns

###
# Add a nice banner to say Hi to the mail clients.
# Add a message that reflects your company or school

smtpd_banner = Welcome to SchoolName ESMTP Server.

###
# Some default configuration settings 
#

append_dot_mydomain = no     # appending .domain is the MUA's job.
biff = no

###
# Alias maps and databases (Can be configured to use a Databases)
#

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases

###
# Your host name (FQDN)
# This is the name of the machine your server is being installed.
# Normally this name must be a FQDN (Fully Qualified Domain Name)
# that you need to buy. For testing you can add the name to the
# /etc/hosts file and enable postfix to query that file before querying
# a DNS server. Refer to the first option on this setup file

myhostname = you_host_name
mydomain = your_domain.com
myorigin = $mydomain

###
# Domains this server is the final destination. Any mail address
# whose domain is listed here will be delivered locally and no further 
# forwarding will occur.

mydestination = $mydomain, $myhostname, localhost.$mydomain, localhost

###
# List of IP subnets allowed to use this server as mail relay. Only 
# machines with these IP addresses will be able to send email to addresses
# other than your own domains defined in the "mydestination" option.

relayhost =
mynetworks_style = subnet
mynetworks = 127.0.0.0/8 192.168.1.1/28 169.168.2.1/28

###
# Allows to add comments to mail recipients as specified on RFC 822. 
# (i.e. user+comment@your_domain.com)

recipient_delimiter = +

###
# Allow connections to all network interfaces in all IP protocol versions 
# 

inet_interfaces = all
inet_protocols = all

###
# Enable support for maildir format. The trailing / is important!
# The trailing / tells postfix it must use maildirs instead of mbox format.
#

home_mailbox = .maildir/

###
# Set user mail quota. (0 -> unlimited)
#

mailbox_size_limit = 0

###
# END
#

Some important notes you must remember:

myhostname/mydomain: Replace your_host_name with a FQDN hostname you have registered and your_domain.com with the domain part of the FQDN.

mydestination: This is the list (space separated) of all the domains for which this server is the final destination. This server will deliver all emails addressed to those domains locally (i.e. to your users maildir directories). As you can see we use the values from the mydomain and myhostname options.

mynetworks: Only the IP addresses (individual or subnets) that you specify here will be able to use the server to relay email to other domains. This means that if you want to send an email to another domain (not included in mydestination option) you must do so from a machine with one of the IP addresses specified here or you will get a relay denied error message.

home_mailbox: The default setting that comes with Kubuntu is set to "Maildir". I like to replace it with ".maildir" so the mail directory is hidden and normal users won't see it on Konqueror. Many times novice users mess with the Maildir folder and then I have to do magic to restore it.

Creating the Aliases database

We must create an alias database to receive mails for standard users (i.e. postmaster) and redirect those mails to a real user. To do this we simply edit the "/etc/aliases" file like:

alias1:      root
alias2:      root
postmaster:  root
root:        username

You can add as much aliases as you want. Make sure to create a root alias to a real user (i.e. username) that is the actual administrator of the system.

Once you finish adding aliases simply save the file and regenerate the aliases database file with the following command:

sudo newaliases

This will create the aliases.db file used by postfix to lookup aliases. If this file is not found then Postfix will fail to start.

Testing the server

Restart the postfix service to get your server running:

sudo /etc/init.d/postfix restart

To test if the service is running use the telnet command in console:

telnet your_host_name 25
Trying 127.0.1.1...
Connected to your_host_name.
Escape character is '^]'.
220 Welcome to SchoolName ESMTP Server.
quit
221 2.0.0 Bye
Connection closed by foreign host.

The text in blue is the server responses and the text in green are your input commands. The first response you get from the server is a 220 message with the text you set on the smtpd_banner option on the configuration file. With the quit command you simply disconnect from the server.

Some notes on mail relay

Most of the headaches people get setting mail servers are mostly due to the relay settings. To explain how postfix allows or denies email relay we must first understand what is a local domain and an external domain.

Local domains are those domains your mail server is responsible for and that usually are hosted in the server itself (i.e. the users mailbox are in the server hard disk). This domains are specified with the mydestination option in the configuration file.

For example if mydestination contains the domain "mydomain.edu" then any email the server receives with a destination address "juan@mydomain.edu" will be delivered to the user "juan" on the local machine. If the user "juan" does not exist in the local machine then the email will be bounced back to the sender with an user not found error.

An external domain is domain not hosted in your mail server. This includes all domains in existence except those configured in the mydestination option. If your email server receives an email with destination address "juan@otherdomain.com" then the server will reject or relay the email depending on your relay settings.

The default relay configuration is to reject all emails whose destination is not a local domain (i.e. not listed in mydestination option). This is a good default to avoid spammers to use your server to send spam but this will also restrict your server to receive and send emails locally only.

To allow your users to relay emails to external domains you must use the mynetwork option and specify the IP addresses or IP subnets of your users. If the server receives an email with destination address "juan@otherdomain.com" and the sender IP address is in the list of mynetwork option then the server will relay the email to the otherdomain.com mail server.

The drawback of using the mynetwork option is that the list of IP addresses you need to specify usually is not fixed. Some users may have dynamic IP addresses at home or school and sometimes your boss/professor may need to access his email from abroad (i.e. international meeting) making it almost impossible to maintain the list of addresses.

The solution to this problem is to enable roaming in the server. Roaming was implemented with a pop-before-send method in which users had to retrieve their emails using POP service before being able to relay emails with the SMTP server. Latter the SMTP protocol implemented the AUTH command that allows the server to challenge for a username and password before relaying any emails and this method is what we will tackle now.

Enabling Roaming

Postfix implements the AUTH command that allows users to authenticate themselves with the server before sending emails. This approach is more flexible than modifying the IP addresses list in the postfix main.cf configuration file.

For authentication we will use SASL that is very flexible. We could authenticate users using the system usernames/passwords or create a separate database to allow relaying to a selected groups of users only. In Kubuntu because Postfix runs chrooted in /var/spool/postfix we have change a couple paths to make SASL live in the false root. (ie. /var/run/saslauthd becomes /var/spool/postfix/var/run/saslauthd):

sudo mkdir -p /var/spool/postfix/var/run/saslauthd

Next we edit the SASL init configuration defaults "/etc/default/saslauthd" to look like:

START=yes
MECH_OPTIONS=""
THREADS=5
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
MECHANISMS="pam"

For Ubuntu Server the init defaults are a little different. If you are using Ubuntu Server (recomended) instead of Feisty/Edgy then use these defaults:

START=yes
NAME="saslauthd"
DAEMON="/usr/sbin/${NAME}"
DESC="SASL Authentication Daemon"
PWDIR=/var/spool/postfix/var/run/${NAME}
PIDFILE="${PWDIR}/${NAME}.pid"
MECH_OPTIONS=""
THREADS=5
PARAMS="-c -m ${PWDIR}"
MECHANISMS="pam"

We must also create the SASL's configuration file that tells postfix what authentication daemon it should connect for authentication. Create a file "/etc/postfix/sasl/smtp.conf" and insert the following options:

#pwcheck_method: authdaemond
#pwcheck_method: pwcheck
pwcheck_method: saslauthd
log_level: 3
mech_list: plain login

You can choose between the three authentication daemons: authdaemond, saslauthd and pwcheck. The most flexibe is saslauthd that can be configured to use several authentication methods like shadow, sasldb, pam, etc.

Now we must configure Postfix to enable SASL authentication by adding the following to the /etc/postfix/main.cf file:

###
# Postfix SASL Authentication configuration. To allow relay of mails
# from roaming users.
#

smtpd_sasl_local_domain =
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = /etc/postfix/sasl/smtpd
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks,  reject_unauth_destination

###
# END

The "permit_mynetworks" makes SASL honor the "mynetwork" configuration option inside the main.cf file. This way users that are inside those IP networks can still relay emails even without authentication.

Ubuntu Server little hack

After configuring the SASL daemon I found that the init script "/etc/init.d/saslauthd" was failing to start the service. After some debugging I found that the directory checking code was failing causing the script to end with no warnings nor errors.

To fix this problem we must comment two small lines on the init script "/etc/init.d/saslauthd":

#dir=`dpkg-statoverride --list $PWDIR`
#test -z "$dir" || createdir $dir

Simply edit the script and search for the two lines above and make sure they have a "#" sign in front. Save and we will be ready for testing.

Restart and Test

We must restart both the saslauthd and postfix services for the new configurations to take effect.

sudo /etc/init.d/saslauthd restart
sudo /etc/init.d/postfix restart

And to test the new configuration we proceed as before but with an additional check for the AUTH command:

telnet your_host_name 25
Trying 127.0.1.1...
Connected to your_host_name.
Escape character is '^]'.
220 Welcome to SchoolName ESMTP Server.
ehlo localhost
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
quit
221 2.0.0 Bye
Connection closed by foreign host.

In this test you have to issue the "ehlo localhost" command and the server will respond with a lot of 250- messages. Each message refers to some feature the server supports. To see if the server supports AUTH simply look for the messages:

250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN

If you see this two messages in the list of responses you get then the server is configured to handle authentication.

Lost your Akregator feed list? Get it back from the archived data

While using my laptop via ssh (with X forwarding) to read my Akregator's feeds the Internet connection failed and Akregator was forced to close. Once back home I fired up Akregator to finish reading my feeds and to my surprise my feed list was gone!.

The feeds.opml file that contains the list was empty and the feeds.opml~ backup file had only the default entries.

After the shock and some failed recovery attempts I found that the archived feed data was intact (i.e. ~/.kde/share/apps/akregator/Archive) and a closer look at the contents of that directory revealed that the archive filenames are in fact the feed's URL addresses with some minor changes (i.e. "/"'s replaced with "_"'s).

With some bash magic I was able to recover the feed list using the following script:

#!/bin/sh

list=`ls http*mk4`

for i in $list; do

    feed=`echo $i | sed -e 's/http___/http:\/\//' \
               | sed -e 's/_/\//g' | sed -e 's/.mk4//'`
    echo "Adding $feed"
    `akregator -a $feed`
done

Copy the script inside the archives directory and make sure you are inside that directory before running it. Once running you will see how Akregator is fired up and the feed list starts filling automatically.

Make sure to make the script executable (i.e. chmod +x script.sh) or simply run it using the bash interpreter (i.e. bash script.sh).

On the down side I was unable to recover the group information and all the feed's were added to the Imported group inside Akregator.

As an additional note I must mention that in the archive directory there are two binary files that seem to contain all the feed list information (including groups). One of the files is called "feedlistbackup.mk4" and it's contents is almost identical to that of the feeds.opml file with some additional binary elements added to it. Maybe someone that knows the binary format can create a simple program to recover the feed list from that file including group information.

Booting the Thinkpad Rescue and Recovery with GRUB

The IBM Thinkpad laptops T40, T42, T42p, T43 and T43p that we use at our lab come with a 5GB Rescue and Recovery partition that can be booted up to restore the laptops OS to the pre-installed state.

In our lab the first thing our students do when they get one of these laptops is to wipe out the pre-installed Windows XP and install their favorite flavor of Linux. This is all good until I get the laptop back and have to restore it to the original state (school rules).

Our lab has the recovery cd's for the T40 and T42 series but not for the T43 series. And out of care I installed Kubuntu on a T43p laptop without creating the recovery CD's first. Even thought I left the rescue and recovery partition intact, Kubuntu by default writes Grub to the MBR effectively rendering it un-bootable.

Under the circumstances I was ready to ask budget to buy the recovery cd's from IBM (8,000~ Yen) but some research lead me to a better (free) solution using Grub itself.

Lucky me this thread on the Thinkwiki shows how to make Grub boot the rescue and recovery partition by editing it's entry on the menu.lst file.

title IBM Rescue and Recovery
root (hd0,1)
parttype (hd0,1) 0x0b
unhide (hd0,1)
chainloader +1

In the case of Kubuntu the rescue and recovery partition was automatically added to the menu.lst file during the installation and all you need to do is add the two lines in bold. After this when you boot the PC press ESC to get the Grub boot menu and select the IBM Rescue and Recovery partition.

With this I was able to create a recovery DVD's and save 8,000 Yen to the laboratory.

MATLAB xcb_xlib_lock Assertion in Feisty

Important Update: With the last update of Feisty Fawn (Herd 5 and up) using this work around will make your KDE unusable!. KDM will start but suddenly disappear leaving a black screen with a cursor blinking at the top left. From the console startx will run ok but with no means to start KDE from within it.

The good news is that with the new updates Matlab works out of the box without need of this work around at all. If you made a copy of the libxcb-xlib.so.0.0.0 file simply restore it and your system will be back on shape. Make sure to install the newest version of libxcb-xlib and Matlab should work without problems... It does on my system.

After I installed Feisty on my machine I installed and tried Matlab and for my unpleasant surprise Matlab started crashing every time I opened a new window. For example the command "plot" that opens a new window would crash Matlab with the following assertion:

MATLAB: xcb_xlib.c:41: xcb_xlib_lock: Assertion `!c->xlib.lock' failed.
 Aborted

My only option at the moment was to keep using Edgy until Mathworks fixes the bug in any of their next releases (something that may never happen). But after I tested Feisty I really did not want to give up on it and as a good Linux user I asked the community and as always got a better solution.

It happens that this xcb_xlib_lock assertion bug is common not only in Matlab but in several other applications used in (K)Ubuntu. Reading the Ubuntu forums (see here and here) you can see that the problem has been addressed in two different ways:

One is to download the libx11-6_1.0.3-5_i386.deb package that comes with Edgy and force a downgrade with:

sudo dpkg --force-all --install libx11-6_1.0.3-5_i386.deb

I have not tried this method but some people report it works without problems.

The second method (the one I used) is to download and modify the source code so the Assertion check is bypassed. This sounds difficult but thanks to the Debian package management system it can be done in a few steps.

First download the source code and all build dependencies in a temp directory (i.e. ~/tmp):

apt-get source libxcb-xlib0
sudo apt-get build-dep libxcb-xlib0

Change to the source directory and edit the libxcb-1.0

cd libxcb-1.0
vi src/xcb_xlib.c

Feel free to use your favorite text editor (I like vim) and comment the following two lines:

assert(!c->xlib.lock);   change to:  //assert(!c->xlib.lock);

assert(c->xlib.lock);    change to:  //assert(c->xlib.lock);

This source file has few code lines so finding the lines to comment should be no problem at all. To comment them simple add "//" at the start of the line or you can delete them all together.

Next build the library:

./configure
make

If the configure script fails make sure you have the build-essential packages installed. If the make command finishes without any errors LOG OUT the X session and change to a console terminal. If you do not log out the next step is gonna crash you X very hard. To change to a console terminal press ++ (after you have logged out!) and to return back to the X console press ++.

cp /usr/lib/libxcb-xlib.so.0.0.0 /usr/lib/libxcb-xlib.so.0.0.0.orig
cp ~/tmp/libxcb-1.0/src/.libs/libxcb-xlib.so.0.0.0 /usr/lib

Replace ~/tmp with the temporary directory where you downloaded and compiled the source code. Finally restart your X11 server (reboot the PC to be sure) and now Matlab runs flawlessly on Feisty. You can also delete the tmp directory if you want.

I must mention that this bug is caused by bad programming practices when using the Xlib libraries and not a problem in Xlib itself. Read here and here for more information.

Video Preview Thumbnails in Digikam

For a long time I thought Digikam did not support video thumbnail previews because I never got those on my system (Kubuntu). Of course this was all my fault for being too lazy to check first. A simple scrap over the Digikam's FAQ showed me:

Why don't I have thumbnails for my video files?
Why thumbnail generation stops at the first video file I have ?


First, check if it's the same in konqueror.

Digikam uses KDE to generate thumbnails and KDE rely on aRts to do that.
So, you should check your aRts configuration. ARts can use xine plugin for video files, so try if xine is able to play your video files. You may have to use additional codecs (eg libxine-extracodecs on debian/ubuntu if you already have libarts1-xine) for that.
Try also to play your video files using xine (if libarts1-xine is installed).

Remember that if it doesn't work in konqueror, it won't work in digikam...

Indeed opening the photos folder in Konqueror did not show any video thumbnails leading me to think this is not a missing feature but a missing configuration or library.

Reading carefully the FAQ entry above it hints about libarts1-xine being installed so I quickly entered the following command:

sudo aptitude install libarts1-xine

And in less than a minute I had video thumbnail previews of all my video clips on Digikam and in Konqueror too.

Kirocker Applet on Kubuntu Feisty

After fixing my little problem with the Amarok charset encodings (see previous post) I was able to rebuild my collection and have all my music titles, albums and artists names nicely displayed on the OSD screen and collection browser.

After that I read about this little Kirocker applet and wanted to try it. Unfortunately there are no packages in the official repositories so we need to install it from sources.

To install it in Kubuntu Feisty Herd 4 (may work too in Edgy and Dapper) you need to satisfy a lot of development dependencies that are difficult to figure out. The configure script will fail a lot as you figure out what packages you need to install to allow the configure script to finish. To avoid you the trouble here are the instructions I used to get Kirocker to compile on my system.

First install all dependencies:

     $ sudo aptitude install build-essential
     $ sudo aptitude install kdelibs4-dev

Download the source code from here and compile/install it:

     $ tar xvfz kirocker-2.0.tar.gz
     $ cd kirocker
     $ ./configure --prefix=/usr
     $ make
     $ sudo make install

Once installed you can add it to the kicker as any other applet and enjoy!.

Feisty Herd 4 impressions

Out of boredom I installed the Feisty Herd 4 release of Kubuntu just to kill time. All I can say is that I am somewhat impressed with this release. The installation was incredible fast and the new interface is very polished and professional.

There is this new animation when clicking an icon that looks just great and the logout/halt screen is a lot more aesthetic. Also I can verify a lot more responsiveness in my rather old hardware (Intel Celeron 1.7G with 512MB Ram).

A feature all users of Feisty have celebrated is the KNetworkManager that makes it easier for laptop users to handle several connection settings (i.e. school, home, work) but in my case I use a desktop computer that is always connected to the wall. I haven't tried it yet on my laptop but I can say it does not work well on my setting. Even thought I am browsing the Internet and writing this blog the KNetworkManager tray icon insists that my ethernet cable is disconnected and that I have no Internet.

Also some minus points go to the installer partition manager. The first option it gives you would simply use the whole disk erasing any other partitions you may have. This is no good for novice users. The manual partition option is Ok but for some reason QParted did not fit on my screen resolution forcing me to move the window horizontally several times to be able to see all available partitions.

Another minus points go to the internationalization support. I always select Japanese as the installer language and the installer warns me this will be the default language on the installed system. After I finished installing the system I hoped I would get a fully functional Japanese system but instead I got a fully functional English system. Enabling Japanese is not big deal as this happened to me on Edgy too before but again this is a no-no for novice Japanese users.

Also when selecting the keyboard layout the Japanese (ja) layouts did not work in the testing box. The layout seemed to be a us keyboard. Strange enough after the installation finished the system's keyboard layout did work with my Japanese keyboard.

Apart from the above details Feisty is very stable and fast. The look is very polished and the smooth. I simply love it and as a person that have used almost all other major distributions I recommend it to all.

Update: It seems that if I configure the network on the live CD before starting the installation then the system will be indeed set up with the language you chose during installation. It makes sense since all those localized packages won't fit in a single 700MB CD so the installer seems to download them from the Internet.

Amarok's Only Problem

Amarok is such a great application I have convinced Windows users to switch to Linux simply because they want to use Amarok. Unfortunately for me and a lot of non-english natives, Amarok is having a little problem with the MP3 id3v1 tags since the 1.4.0 release.

In short the problem (bug?) is that Amarok won't handle id3v1 tags with encodings other than ISO-8859-1 because they claim to be following the spec. Other character encodings are supported only with id3v2 tags. You can read the details in this bug report.

First assuming you agree to take on the colossal task of re-tagging all your MP3 collection, that I haven't found an automatic way to do yet, you will find that even changing all the tags to id3v2 does not solve the problem. Read comments #15 and #19 on the bug report.

I personally have found that the only way Amarok would read the tags and display them correctly is if I use the Amarok's own tag editor itself and doing so will render those tags unreadable on any other software/utility. At this point I really don't know who is following the spec, Amarok or the other hundred apps/utils I have used.

My solution was a little drastic: I converted all my MP3 collection to ogg/vorbis using mp32ogg. I found that the ogg/vorbis tags are UTF-8 by default and all applications I have used so far can read and display my tags correctly.

This solution may not work to everyone since some require other encodings for compatibility with some hardware players but for me is good enough. And yes, I took on the colossal task of converting my collection and re-tag them all. Fortunately mp32ogg works on whole directories and not only on individual files.

For all others that reported this problem let's hope Amarok's developers hear your voice and add the encoding option back to the next great Amarok release. For my part I will keep my max of votes on this bug report.

A dual screen hack: X2VNC and Win2VNC

These two little gems: X2VNC and Win2VNC from Fredrik Hubinette have become one of my favorite utilities to use at work.

Imagine a dual head system where your left screen has KDE, your right screen has WinXP and your mouse moves freely between the two. Using X2VNC and/or Win2VNC you can do it.

My setting is like that described above: two PC's one with Kubuntu and the other running Windows XP. I have RealVNC server installed on the Windows box and use X2VNC in the Linux box to connect to the Windows one and presto! now I can use a single mouse and keyboard to control both boxes.

It is possible to do the connection the other way around by installing the X11VNC server on the Linux box and then connect to it from the Windows box using Win2VNC. The last time I tried this configuration the mouse in Win2VNC wasn't as smooth as in X2VNC.

You cannot use the RealVNC server in Linux in this case because this VNC server creates a new X11 session for each VNC connection, With x11vnc you get control over the current X11 session.

Windows Setup for dual screen hack:

Download and install the free RealVNC. Start the VNC server in user-mode or as a system service. For testing purposes configure "No Authentication" on the Authentication tab of the server options and make sure the Linux box IP address is configured on the Connections tab of the server configuration.

To open the configuration dialog of the VNC server simply select "Options" in the context menu (right click) of the VNC icon on the system tray.

Linux Setup for dual screen hack:

Install the x2vnc package of your distribution. In Kubuntu edgy/feisty is as simple as:

sudo aptitude install x2vnc

Then run the following command to enable the dual screen hack:

x2vnc -east windows_ip_address:0

Now if you move your mouse to the far right of you screen (Linux Screen) it should appear on your Windows screen and now everything you type on your keyboard would appear also on your Windows screen.

If your Windows screen is to the left of the Linux screen then you must replace the "east" option with "west". You can also use "north" and "south" in case you have the Windows screen above or below the Linux one.

Security Setup
To avoid unauthorized access to your system it is better to enable access control to your VNC server. To do this simply enable password authentication on the RealVNC server and restrict the IP addresses that can connect to the server in the connection settings.

On the Linux side we must create a passwd file so that X2VNC can authenticate with the server. For this we need the vnppasswd command utility that comes with the RealVNC server package:

sudo aptitude install vnc4server

After installation is finished we issue the next command to create a password file:

vncpasswd

This command will prompt you for a password that must be the same you configured on the Windows RealVNC server. Once you enter the password a passwd file will be created inside the .vnc/passwd directory inside your home directory. To make X2VNC use this file for authentication when connecting issue the command:

x2vnc -east -passwdfile ~/.vnc/passwd windows_ip_address:0 

I have created a desktop shortcut where I configure this command to avoid having to type it every time I boot the machine. Now with a simple click I enable the dual screen hack.

Testing POP over SSL the old way

In the old days it was common to test SMTP/POP servers using a simple telnet connection to the corresponding port (25/110) and interacting directly with the server via text commands. This was simple, clean and easy to do without need of configuring any mail clients just for testing.

These days any basic SMTP/POP server is configured with TLS or SSL to improve security. This is a necessity rather than a luxury these days as more commerce and business are dependent of Internet services like Mail.

For a time I thought that the old school of direct text interaction with these servers was impossible until I found the openssl command utility. From the man page it is easy to see that this openssl utility has several uses and one of them is. taken from the man page itself: SSL/TLS Client and Server Tests!!.

For example to connect to a POP3 server that uses SSL we call the following command:

openssl s_client -connect localhost:995

Make sure you replace localhost for the server address and if you use a different port (995) change it as well. This command will connect to the POP3 server and handle all the SSL stuff for you. You will see a lot of messages about certificates and SSL handshakes etc... if you are not interested in SSL simply ignore them.

After all the SSL stuff has passed you will be greeted by the POP3 server message (+OK Hello There for courier-pop) and then you can start interacting with the server as you would using telnet.

If you don't know how to interact with a POP server using text commands check my previous post about the topic.

Faking POP

As part of a networking class at my lab we teach students protocols by asking them to implement a simple POP3 client in Java. The client must connect to the POP3 server and interact with the server as a normal POP client.

As soon as the class started we realized the university POP server uses SSL to encrypt the connection (for obvious security reasons) and asking the students to implement the SSL part of the client was not an option.

We needed a POP server set up quickly before the students could present their programs or at least something that behaves like a POP server. A quick search resulted in FakePOP and the great Ubuntu has it in their repositories.

I simply issued the following command:

sudo aptitude install xinetd fakepop

Then create a xinetd configuration file "/etc/xinet.d/fakepop" that contains the following:

service pop3
{
server = /usr/sbin/in.fakepop
disable = no
socket_type = stream
protocol = tcp
user = nobody
wait = no
}

and restart the xinetd daemon:

sudo /etc/init.d/xinetd restart

and in less than a minute I had a POP3 server (or at least something that behaves like one) on my machine and the class was saved.

Also you can create a custom message to be delivered to anyone connecting to the fakepop server. You can create as many messages as you want by putting them as text files inside the "/etc/fakepop" directory. Some examples can be found in the "/usr/share/doc/fakepop/examples" directory like the one below:

From: System Administrator
Subject: example message - text only
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

This is a simple message.
It contains only text.

Long Vacations in Panama

I was in a long vacation (over two months) in a tropical paradise known as Panama. Most people (specially in Japan) do not know Panama is actually a country and only know about it because of the Panama Canal.

If you like beaches you would like to go to Isla Grande, Isla Colon and San Blas were you will find beautiful caribbean beaches or travel the Via Panamericana visiting all the resorts at the Pacific Sea. If you like mountain hiking you can go to El Valle, Boquete or Volcan to find a rich tropical fauna and flora only seen in Panama.


City people can simply stay in Panama City and visit the malls (Multicentro, Multiplaza and Albrook Malls) and all the night clubs around the city (The Gallery, Oz, Mistyk) and of course the Coast Way to eat on the finest restaurants in the country.

Panama has it all in a relative small country so make sure you visit it in your next vacation. Here are some links with information about Panama:




Chiriqui Chatter: a blog by James Dean where he talks about his like in Panama. Here you will find a lot about the culture and cousine and also how to retire in Panama if interested.

Blogs Panama The name says it all. You will find here a lot of blogs about Panama and it's people.

Tavel Blogs from Panama Here you can get the view of Panama from foreign people in Panama. People that have traveled to Panama and wanted to share their experiences, photos and even videos with the world.

Dealante.com Finally if you want to know about the night life in Panama (Wild on E! style stuff) then check this page.

After this long vacations under the sun I got a little rusty in the technology camp. This is the principal reason I haven't learn anything new or interesting to blog about. I will try to resume my studies/work activities and start blogging soon.