Tuesday, March 27, 2007

Linux Mail Server Setup

Foreword

As system administrator I have had the privilege of working with all sorts of linux servers and services in a variety of distributions. Here I present two configurations I have used in most of my installations.

The first one is based on qmail SMTP with virtual domain support with VPopmail/MySQL on RedHat ES. This configuration I used mostly as a system admin on a cable modem company that handled thousands of residential users and hundreds of commercial users (i.e. with their own email domains). I can say this setup is scalable in terms of users but may present some problems with large queues caused by large traffic bursts. Also due to the maildir format I had some problems with inode exhausting but this is a file system problem (i.e. using XFS instead of ext3 could have avoided this problem).

The second configuration is more updated with current standards. It uses the more easy to configure and secure Postfix for SMTP and Courier-POP for POP. It could have used other POP servers like Dovecot. I have tested this configuration on a small laboratory with 30 users so no idea on how scalable it may be.

I do not support virtual domains in this configuration because I consider current solutions to email virtual domains confusing. With the virtualization hype these days I believe SMTP servers should not handle virtual mail domains. We should install several virtual machines in a single hardware and manage each mail domain with a fully functional SMTP server in each of the virtual machines.

Redhat Eterprise Server (ES) Mail setup

Pros:

  • Fully functional qmail SMTP and POP services.

  • Supports virtual domains with VPopmail/MySQL.

  • Maildir storage format.



Cons:

  • Installation from source code and requires some source modifications.

  • Rather complicated manual configurations.

  • No support for encrypted communications. This limits the server to local network or locahost connections (i.e. Webmail client).

  • Roaming support uses the outdated POP-Before-SMTP.




Instructions:

  • Setup qmail server
    First we install the qmail SMTP server on RedHat ES 4 from source code. Due to some incompatibilities with the version of glibc that comes with RedHat we are forced to modify the source code a little to get it to compile.

  • Setting TCPSERVE
    We use tcpserve to handle all incoming connections to the qmail SMTP port (25). With tcpserve we can control access to the port and allow/deny relaying of emails to users depending on IP address. This access control is needed to avoid spammers from using our server for free.

  • VPopmail
    We end our setup by installing Vpopmail to support virtual domains managed with MySQL backend. We configure Vpopmail to support maildirs and to manage roaming. Vpopmail modifies the tcpserve tcp rules files to allow POP authenticated users to relay mail (POP-Before-SMTP).




Ubuntu Mail Server Setup (Kubuntu Feisty Fawn/Ubuntu Server)

Pros:

  • Fully functional Postfix SMTP and Courier-POP services.

  • Installation and configuration is a breeze (soooo easy).

  • Maildir storage format.

  • Roaming using SMTP-AUTH.

  • Encrypted transmission for SMTP (TLS) and POP (SSL).



Cons:

  • No support for virtual domains.



Instructions:


  • Installing the SMTP server
    First we install and configure a fully functional Postfix server and enable SMTP-AUTH for roaming with SASL authentication.
  • Installing the POP sever
    Next we install the Courier-POP server with Maildir storage support.

  • Enable Encrypted Communications
    Now we enable secure transmission by encryption of SMTP (TLS) and POP (SSL) traffic. We create self signed certificates and enable Postfix and Courier-POP to use them.

  • Testing
    Finally some pointers on how to test SMTP/POP services using a console window via telnet/openssl commands.

Monday, March 26, 2007

Sony NW-A808 PMP Linux review



I recently got this little piece of hardware and I really like it. It supports video, audio and images in a very small and stylish pack. I was deciding between the NW-A808 and the iRiver X20 but finally decided to go with the NW-A808 mostly for the cool factor. The iRiver X20 is bigger and has a plastic case that makes it feel like a toy. The NW-A808 has a very slim and metallic case (iPod like) that feels just right on my hand. Also the NW-A808 has a nice variety of colours to choose while the X20 comes only with a black version.

The NW-A808 GUI supports various languages so I can set it up in English or Spanish if I want instead of the default Japanese. It supports album covers, video thumbnails and has a lot of play list features like search, bookmarks, and much more.

The good and the bad

The good thing is that this device supports MSC (aka UMS) that is very well supported by Linux. Plugging the player to my laptop running Kubuntu Feisty I get a pop-up window asking me if I want to open it in a new window.

Opening the device (/media/WALKMAN) in Konqueror shows me the content of the device and I can easily copy/delete files into it. It is possible to add images and videos to the device by directly copying them into the flash memory.

The bad thing is that first: it does not support Ogg and second: that to add music files to the device we are forced to use a crappy piece of proprietary software called Sonic Stage on Windows. I haven't found yet any Linux alternatives to replace Sonic Stage but rest assured that someone (maybe me), someday (far in the future) will take the task to make this device Linux friendly.

Uploading Images

To upload your images to the NW-A808 flash memory you MUST create sub folders inside the DCIM directory that is in the flash memory and copy your images inside those sub folders. You can copy them directly inside the DCIM folder but they will not get listed on the device library browser.

From the instruction manual the images format supported is:

  • File Format: DCF 2.0/Exif 2.21

  • Extension: .jpg

  • Codec: JPEG(Baseline)

  • Max Resolution: 4,000x4,000 (16 Megapixels)

  • Quantity: Maximum 10,000 files



Uploading Videos

As with the photos we only need to copy our video files in mp4 format inside the VIDEO folder that is inside the device flash memory. Pay attention to the supported video format as any file not supported may be listed in the video library but the player will refuse to play it.


  • File Format: MP4

  • Extension: .mp4, .m4v

  • Video Codecs:

    • MPEG-4

      • Profile: Simple Profile

      • Max bitrate: 2,500 kbps

      • Max fps: 30 fps

      • Max resolution : 320x240(QVGA)



    • AVC (H.264/AVC)

      • Profile: Baseline Profile level 1.2 and 1.3

      • Max bitrate: 768 kbps

      • Max fps: 30 fps

      • Max resolution : 320x240(QVGA)





  • Audio Codecs: AAC-LC

    • Number of Channels: 2

    • Samplig rates: 24, 32, 44.1 and 48 kHz

    • Bitrate: 288kbps per channel





Based on the video/audio specifications above we can use the all mighty ffmpeg to convert almost any video to a format that can be played on the device:
Since this is a device created by Sony I concluded it supports the same video format the PSP does . A simple Google search for "mencoder PSP" led me to this post with the following command:

ffmpeg -y -i $infile -vcodec libx264 -b 768k -bt 768k -level 13 \
-r 30 -croptop auto -cropbottom auto -s 320x240 -acodec libfaac \
-ar 48000 -ab 100k -ac 2 -f mp4 $outfile.mp4

With this command I was able to convert most of my mpg and avi files to mp4 and play them in the NW-A808. Once you transcode you videos copy them to the Videos directory inside the device flash and they will be listed on the video library.

There is a little functionality drawback if we copy files directly to the flash. The NW-A808 displays the video file name and a nice THM thumbnail in the video browser when the files are copied using Sony's proprietary software. I have tried in several ways to create the thumbnail file manually without success so far.

Filenames and Folders



It is possible to copy images and videos in any folder inside the device flash. It is also possible to create sub folders in any place and put files inside those sub folders. But only the images and videos that are inside the DCIM and VIDEO folders respectively will get listed on the device browser. See the image above to get an idea of the folder layout.

You can copy all video files directly inside the VIDEO folder in flash or create sub folders (one level deep only) and put the video files on those sub folders.

For images you cannot directly put the files inside the DCIM directory. You must create sub folders (one level deep) inside the DCIM folder and copy the images inside those sub folders.

Also make sure the extensions are jpg for images and mp4/m4v for video or the player won't list them.

Uploading Audio Files

Now the biggest drawback of this and all other Network Walkman devices produced by Sony. To upload music files we must use Sonic Stage on windows because simply copying the music files on the device won't list them on the album browser.

So far the closest I have found to make this device work on Linux is the libnw library created by Waider. Too bad this library does not work on my device but Waider has shown interest in modifying his library to support the NW-A808. Once libnw supports this device it won't be difficult to write an Amarok MediaDevice interface to manage it.

Sony uses GPL software?

Not really sure why but inside the NW-A808 box came a copy of the GPLv2 licence. It means (maybe?) that the software provided by Sony to handle the deviceuses some GPLed software/libraries.

I am no lawyer but as I understand if this is true then isn't Sony supposed to release the source code of such applications?. With a source code it would be easier to implement a Linux alternative to make these devices more Linux friendly.

Saturday, March 17, 2007

Best File System for Server Usage

Problem Review

One of my mail servers failed when the ext3 partition that has the users maildirs ran out of I-Nodes. After this it was clear that ext3 is not scalable enough to handle large number of users, each one with large amounts of files/directories. This is a typical setting for a IMAP/SMTP service that allows users to organize their mails on the server rather than downloading them.

After a lot of research (Google) I found a lot of benchmarks and articles that made one thing clear: there is really no real silver bullet in the journaled file systems camp. For example ReiserFS is famous for being the best handling small files and it is true it does but it has really bad performance when handling big files. On the other hand XFS can handle massively large files really fast but has other problems as high CPU consumption and low performance deleting large amounts of files.

There are a lot more examples like these were one file system shines in one test but leaves a lot to desire on others.

After a week of deliberation I have decided to give XFS a try and use it from now on on all my future server installations.

Why XFS?

After looking at some benchmarks
here
and here it is easy to see that XFS performs well or better in almost all tests. For those tests it is not the best it usually falls behind the best by a small margin.

The only benchmark it fails by a large margin is for deleting large number of files. This is one of the most criticized problems of XFS but there are tweaks to improve it (see below).

Another problem often criticized is that XFS seems to be CPU intensive when compared to other file systems. Looking at the benchmarks we can see this is true but we can also see that the high CPU usage is not the highest in all tests.

When evaluating journaled file systems there are a lot of features that must be accounted for. A very nice review of what too look can be found here and here.

The decision between XFS and JFS seems strange as JFS appears to be as good as XFS in all benchmarks using less CPU than XFS. The decision to go with XFS usually comes from the fact that it is more mature and has more features than JFS or other journaling file system in fact.

Some of the features that are important to servers that XFS supports are;


  • Extended Inode Attributes: This allow use to add custom attributes to inodes giving use more granular control over the file systems. For example we could add tags to inodes and search files based on the tags.

  • Online Defragmentation: Modern file systems are resistant to certain point to fragmentation of file but as the available free space is reduced the amount of fragmentation can increase to an alarming rate. The ability to defragment the disk without unmounting it is desirable for a high availability server

  • Online backup utitlities: Again for high availability servers the capacity to backup and restore complete partitions is a plus. XFS has utilities that allow backup and restore of entire partitions online.

  • Dynamic I-Node Allocation: This allows us to create large number of files in the file system without worries of running out of I-Nodes. All modern file systems, except for ext3, support this feature.

  • Support For Quotas: Limiting the disk usage capacity in a multi user environment is very important to be able to offer services with varying levels of quality of service.

  • Allocation Groups: As far as I know this is an XFS only feature that allows processes running concurrently to allocate space in the file system concurrently without interfering with each other. With this XFS enables free space and inode management operations to proceed in parallel throughout the file system increasing performance on large partitions. Current dual-core/multi-processor systems can take advantage of this feature to further increase overall system performance.


Some Remarks Against XFS


  • XFS is very slow deleting large amounts of small files!!.


    I haven't found yet a situation were I needed to actually delete a lot of small files from a mail server. On the other hand I do have found situations were I need to delete large files (i.e. 4GB database tables and 2GB log files). Also after reading this post it seems that it is possible to tweak XFS to perform even better than ext3 when deleting large number of files.


  • XFS only logs meta data on the journal so you will loose data on a crash


    The only File System that is known to be robust to system crashes and sudden power outages is ext3 and only if it is in ordered or journal mode. All other file systems prefer to log only meta data to improve performance. You must decide between a robust but slower file system (i.e. ext3, ReiserFS) or a faster but less robust file system (i.e. XFS, JFS).

    For a server the trend goes to speed performance rather than robustness to crashes and system failures. The Philosophy behind this trend is that if you are intend to install a server you surely have also a disaster recovery strategy. Having a nice backup/restore plan and more than one independent power source are also part of installing a high availability, high performance server.


  • If you need maildir storage then ReiserFS is the way to go!


    Maybe but the benchmarks show that XFS is not that far behind ReiserFS when handling small files and at the same time is the best handling large files. ReiserFS may be well optimized to handle small files but it has really bad performance problems handling medium to large file sizes.

    Also to note is that the incoming Reiser 4 is the worst performer in all speed tests and is even more CPU hungry than XFS.


  • What about JFS?? I heard it performs equal to or even better than XFS with less CPU load and more robust to sudden crashes!![/B]

    Yes I heard it too and some benchmarks show that JFS can be as good or better than XFS in read/write performance. But as mentioned above JFS is relative new (in the linux camp) and is not as mature nor has as many features as XFS. For example it has some problems with user quotas.


So XFS is the all purpose File System

Sorry to say but XFS is no silver bullet here. As I said my conclusions are based on server application needs. I would not recommend using XFS or any other meta-data only journaling file system on a desktop computer, specially on a laptop that has more probability of power outages. The amount of complaints about all KDE configuration files being zero-length or filled with zeros after a crash talk by themselves (link).

XFS is not robust to crashes and power outages as ext3 can be so I will stay with ext3 for my desktop installations and my laptop. Ext3 is the recommended default by Ubuntu and with some tweaks it is possible to improve it's performance further (see here and here).

Future Work

All my conclusions are based on others hard worked results and opinions of others. To validate my conclusions I have collected a one week worth of qmail log files I am planning to use them to obtain mail usage statistics statistics and create some more realistic disk usage patterns. Current benchmarks are based on synthetic tests that may not reflect real load patterns in real mail or database applications,

Once I obtain some useful statistics from my logs I plan to perform some benchmarks over all file systems mentioned here with their respective performance tweaks. Once I get some results they will be posted here in this blog.

General Resources

http://linuxgazette.net/issue55/florido.html
http://bulma.net/impresion.phtml?nIdNoticia=1154
http://linux.inet.hr/first_benchmarks_of_the_ext4_file_system.html
http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://linuxgazette.net/122/piszcz.html
http://www.debian-administration.org/articles/388
http://adria.fesb.hr/~jsosic/mojbench.html

XFS Resources

http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://en.wikipedia.org/wiki/XFS
http://everything2.com/index.pl?node_id=1479435
https://launchpad.net/ubuntu/+bug/37435
http://www-128.ibm.com/developerworks/linux/library/l-fs10.html
http://www.ubuntuforums.org/showthread.php?t=246969
http://forums.gentoo.org/viewtopic-t-488215.html
https://ols2006.108.redhat.com/reprints/chinner-reprint.pdf
http://everything2.com/index.pl?node_id=1479435

ReiserFS Resources

http://www-128.ibm.com/developerworks/library/l-fs.html
http://www.moon-soft.com/program/bbs/readelite666510.htm
http://en.wikipedia.org/wiki/ReiserFS
http://en.wikipedia.org/wiki/Reiser4

Ext3 Resources

http://www-128.ibm.com/developerworks/linux/library/l-fs8.html
http://www.suseforums.net/index.php?showtopic=29773
http://forums.gentoo.org/viewtopic-t-488215.html
http://en.wikipedia.org/wiki/Ext3
http://en.wikipedia.org/wiki/Ext4

JFS Resources

http://www-128.ibm.com/developerworks/library/l-jfs.html

Wednesday, March 14, 2007

Testing SMTP and POP services from a console

Test a non-secure SMTP server

To test a non-secure SMTP server we simply connect to the port 25 of that machine using telnet:

telnet smtp_ip_address 25
Trying smtp_ip_address...
Connected to smtp_ip_address.
Escape character is '^]'.
220 Welcome to Postfix ESMTP Server
ehlo localhost
250-testmail
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from:auser@internaldomain.org
250 2.1.0 Ok
rcpt to:noone@externaldomain.org
554 5.7.1 : Relay access denied
rcpt to:noone@internaldomain.com
550 5.1.1 : Recipient address rejected: User unknown in local recipient
rcpt to:realuser@internaldomain.com
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
Hello There!!
.
250 2.0.0 Ok: queued as 7ECF05100C6
quit
221 2.0.0 Bye
Connection closed by foreign host.

The green text are the commands you send to the server while the blue text indicates the server responses. The example above is for a simple Postfix SMTP server.

Testing the AUTH command

From the previous example we can see that the Postfix SMTP server supports the AUTH command that allows roaming users to use the server as relay. To test this command we must first encode our authentication credentials in base64 format:

perl -MMIME::Base64 -e 'print encode_base64("\000username\@domain.net\000password")'

You must have the MIME::Base64 perl module in your system (is installed by default in Kununtu). Also make sure you replace the username, domain.net and password in the command to reflect your own valid values.

After the command finishes it will output your authentication credentials as a base64 encoded string that looks like:

AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=

Copy that string somewhere and proceed to connect to the server as in the previous example:

telnet smtp_ip_address 25
Trying smtp_ip_address...
Connected to smtp_ip_address.
Escape character is '^]'.
220 Welcome to Postfix ESMTP Server
ehlo localhost
250-testmail
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
auth plain AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=
235 2.0.0 Authentication successful
mail from:auser@internaldomain.org
250 2.1.0 Ok
rcpt to:noone@externaldomain.org
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
hahahaha
.
250 2.0.0 Ok: queued as 7B6265100C6
quit
221 2.0.0 Bye
Connection closed by foreign host.

In this example we used the "auth plain" command to authenticate ourselves against the SMTP server. Once authenticated we are allowed to relay mails to any domain (not only internal domains) from anywhere (roaming).

A closer look at the the previous example will reveal that when we send the "rcpt to" command to an external domain (noone@externaldomain.org) the server response was a "554 relay access denied". In this example we are authenticated with the server (via the AUTH command) so now the "rcpt to" command was accepted.

Test a non-secure POP server

Similar to SMTP we can test a POP server by connecting to it using telnet:

telnet pop_ip_address 110
Trying pop_ip_address...
Connected to pop_ip_address.
Escape character is '^]'.
+OK Hello there.
user username
+OK Password required.
pass password
+OK logged in.
list
+OK POP3 clients that break here, they violate STD53.
1 520
.
quit
+OK Bye-bye.
Connection closed by foreign host.

Again green are your commands and blue are the server responses. In this example we simply connect to the server, authenticate and get the list of mails in our inbox. We can also see the contents of the mails with the "retr num" command replacing num with the mail number as displayed by the "list" command.

Test secure SMTP/POP servers

If the SMTP/POP server supports encryption (TLS/SSL) we won't be able to test it using simple text telnet connections. In this case we must use the openssl command utility that will take care of all TLS/SSL negotiations, connect to the server and allow us to send plain text commands over the encrypted connection.

For SMTP we can check if the server support TLS encryption by looking at the server response to the "ehlo" command. If we get a "250-STARTTLS" within all the responses then the server supports TLS.

To connect to the SMTP server over a TLS encrypted connection we can use the following command:

openssl s_client -starttls smtp -crlf -connect smtp_ip_address:25

You will get a lot of messages from this command that show the SSL negotiations. You really don't need to care about all those messages. When openssl finishes setting up the encrypted connection you will see the same welcome message we got on the first example:

220 Welcome to Postfix ESMTP Server

From here on you can proceed with plain text commands as in the first two examples.

In the case of POP the server it is a little more difficult to check if it supports TLS/SSL connections. POP uses a different port (995 instead of 110) for SSL encrypted connections. This means that as long as the server uses the default ports if the server has the port 995 open we can assume it supports POP over SSL.

To connect to the POP server we must use the openssl utility command to take care of all the SSL negotiation procedures. The following command will do the trick:

openssl s_client -crlf -connect pop_ip_address:995

After all the SSL certificates are exchanged and the encrypted connection is established we are greeted by the POP server hello message as in the previous POP example:

+OK Hello there.

From this point on we can proceed issuing normal text commands and getting server responses. All the encryption part will be handled automatically by openssl.

References

I learned about testing the AUTH command and SMTP over TLS from the following page: http://qmail.jms1.net/test-auth.shtml

Add encryption to SMTP and POP (TLS/SSL)

In previous posts we configured a fully functional SMTP and POP servers for a single domain with local system users and with roaming capabilities using SASL authentication.

In this post we further improve our setup to allow encrypted communications in SMTP and POP. This way we add a security layer to protect users from mishap from third persons and protect sensitive information.

Creating the SSL certificate for SMTP

The recommended way to do this is to create a certificate request and get it signed by a Certificate Authority but for demonstration purposes I will create a self signed certificate.

For postfix we must create a cert/key pair using the openssl command. In your home directory or a temporary directory you can create the SSL cert/key pair like this:

openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 \
-nodes -keyout smtpd.key -keyform PEM -days 1095 -x509

You will be asked a set of questions that you can fill any way you feel like. This is only to create an identity that must be unique among certificates. Note that the above is a one line command split in two due to space constraints.

Next we create some directories to store the SSL certificates. I like to store them inside the postfix configuration directory.

sudo mkdir /etc/postfix/ssl
sudo mkdir /etc/postfix/ssl/cert
sudo mkdir /etc/postfix/ssl/private

sudo chmod 755 /etc/postfix/ssl/cert
sudo chmod 710 /etc/postfix/ssl/private

Then we move the certificate pair to the newly created directories:

sudo mv smtpd.cert /etc/postfix/ssl/cert
sudo mv smtpd.key /etc/postfix/ssl/private


Enabling TLS in Postfix

Simply edit the /etc/postfix/main.cf file and add the following:

###
# TLS parameters

smtpd_tls_cert_file=/etc/postfix/ssl/cert/smtpd.cert
smtpd_tls_key_file=/etc/postfix/ssl/private/smtpd.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache
smtpd_tls_auth_only = no

###
# END

The "smtpd_tls_auth_only" option can be used for added security. It forces the communication the be encrypted before any credentials (user and passwords) are transmitted. Unfortunately some client's may fail to authenticate with this setting so it defaults to "no".

Finally restart postfix:

sudo /etc/init.d/postfix restart


Enable POP over SSL

If you followed my previous post then you already have installed the required courier-pop-ssl package. If not simply install it with this command:

sudo aptitude install courier-pop-ssl


Generating the SSL certificate for POP

By default Kubuntu uses an auto generated SSL certificate (pop3d.pem) that you can find inside the "/etc/courier" directory. The pop3d.pem file is actually a cert/key pair combined into a single file. For our POP server we will use the same cert/key pair we created for the SMTP server to generate our pop3d.pem file.

In your home or a temporary directory:

sudo cat /etc/postfix/ssl/cert/smtpd.cert > pop3d.pem
sudo openssl gendh >> pop3d.pem
sudo cat /etc/postfix/ssl/private/smtpd.key >> pop3d.pem
sudo chmod 600 pop3d.pem

Now backup the auto generated certificate and replace it with the new one:

sudo cp /etc/courier/pop3d.pem /etc/courier/pop3d.pem.orig
sudo rm /etc/courier/pop3d.pem
sudo cp pop3d.pem /etc/courier

You can put the generated pop3d.pem file anywhere you want as long as you change the path in the courier configuration file (i.e. /etc/courier/pop3d-ssl). For security reasons it may be better to keep it in the default place.

Finally restart the service but this time the SSL one.

sudo /etc/init.d/courier-pop-ssl restart

To force your users to use always SSL simply disable the normal POP server and leave only the one with SSL.

sudo /etc/init.d/courier-pop stop
sudo /etc/init.d/courier-pop-ssl restart

Also to stop courier-pop from starting at boot time we must remove the start up script from the rcX.d directories with the following command:

sudo update-rc.d -f courier-pop remove

This way when the server reboots only the SSL enable POP service will be started. If for some reason you require to start the normal POP service at boot time again you can do it with the command:

sudo update-rc.d courier-pop defaults


Maildir

Do not forget to modify the "/etc/courier/pop3d-ssl" configuration file in Kubuntu Feisty to use ".maildir" instead of the "Maildir" for storage of mails. As with normal POP simply edit the file and replace the value of the MAILDIRPATH variable.

In the case of Ubuntu Server Edition all these services (POP, POP-SSL, IMAP, IMAP-SSL) use the configuration inside "/etc/default/courier" to determine the maildir's directory. Simply make sure you modify the file and replace the MAILDIR value with ".maildir"

Testing

In previous posts we were able to test our configurations by connecting via telnet to the corresponding ports. Now that we have enabled encryption we cannot test with simple text commands via telnet.

It is still possible to test the SMTP and POP services over encrypted connections in a console using the openssl utility but I will explain how to do this in a future post.

Basic POP Server Setup in Kubuntu (Courier)

In a previous post I explained how to set up a very basic SMTP server to deliver mails to users local maildir's and to relay mails to external domains (only for authorized users).

Now to allow users to download/view their emails in their own PC's we must enable a POP or IMAP service. I will only deal with POP here but since the configuration procedure for POP and IMAP in Kubuntu is very similar everything said in this post for POP applies with minor changes to IMAP.

Installing the Software

In Kubuntu installing the courier POP servers cannot be easier:

sudo aptitude install courier-pop courier-pop-ssl

Note that we are also installing the courier-pop-ssl package that allows POP over SSL connections. I will talk more about SSL in a future blog post. During the installation you will be asked if you want directory based configuration files, chose not to and then press ok when asked.

Creating Maildirs

The maildir vs mbox formats to store emails is an old debate and I will not deal with it here. I like maildir storage simply because I had a lot of lock and corrupted file problems in the past due to the "single file" mbox format and migrating to maildir format kept those problems away.

By default Courier-POP in Kubuntu uses the maildir format to store users mail. The only problem I have with the default settings is that it uses a folder called "Maildir" inside each user home directory. I personally prefer to hide the mail's directory from the users to avoid accidents (i.e. total deletion).

Now to enable each user's maildir we must first create the maildir directory:

sudo maildirmake /home/username/.maildir
sudo chown -R username:usergroup /home/username/.maildir
sudo chmod -R 700 /home/username/.maildir

You can replace .maildir with whatever you like. The dot "," at the begining makes that directory hidden so it would not be listed normally in the file manager (i.e. Konqueror) or by issuing "ls" on the console.

Creating the .maildir directory must be done for all current users in the system. For new users the maildir creation can be automated by creating a maildir directory inside the "/etc/skel" template directory:

sudo maildirmake /etc/skel/.maildir

Now every time a new user is created the .maildir folder will be automatically created in that user's home directory.

Configuring Courier POP

The default settings that come with Kubuntu are enough to get the services running. The only change we have is to replace the "Maildir" directory with ".maildir". This is done easily by editing the "/etc/courier/pop3d" configuration file:

sudo kate /etc/courier/pop3d

and

replace MAILDIRPATH=Maildir with MAILDIRPATH=.maildir


In the case of Ubuntu Server the above configuration won't work. Instead of modifying the /etc/courier/pop3d file you must edit the /etc/default/courier file:

sudo kate /etc/default/courier

and

replace MAILDIR=Maildir with MAILDIR=.maildir


Restart and Test

Finally we restart the POP/IMAP service:

sudo /etc/init.d/courier-pop restart

And to test it we connect to the port 110 using telnet:

telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK Hello there.
user username
+OK Password required.
pass userpass
+OK logged in.
list
+OK POP3 clients that break here, they violate STD53.
.
quit
+OK Bye-bye.
Connection closed by foreign host.

As in the previous post the blue text are server response messages and the green text are your input commands. To authenticate you simply use the "user" and "pass" commands. Once authenticated you can list your emails with the "list" command and exit with the "quit" command.

Coming Next

In the old days our server is now ready to go online but these days this configuration would be considered insecure. Current state of the art in servers must provide security mechanisms like encrypted communications to avoid leakage of sensitive/personal information.

In my next post I will explain how to encrypt SMTP and POP using TLS/SSL to have a more secure server ready for the current Internet.

Basic SMTP Server Setup in Kubuntu (Postfix)

Kubuntu is mostly a desktop oriented linux distribution, but this does not mean that it cannot be used as a server. Several guides exist on the Ubuntu Wiki that teach you how to setup different kinds of servers.

This small tutorial will show you how to set up a local SMTP server (Postfix) for a small office or school with a single domain.

This configuration was tested on Kubuntu Feisty Fawn and in Ubuntu Server distributions. This configuration may work also with Dapper and with Edgy but I have not tested it.

Installing Software

In a fresh installation of Kubuntu simply install the postfix and support packages:

sudo aptitude install postfix libsasl2 sasl2-bin libsasl2-modules libdb3-util

Note that the above command installs SASL too that will be used to enable roaming in our server. During the installation of postfix the package manager will give you the opportunity to configure the most basic options of postfix. I simply select not to configure it at the time and latter modify the configurations files with a text editor. This way I get more control on the look of the file and can format it in a easy to read layout.

Configuring the server

The main configuration file main.cf of postfix can be found inside the "/etc/postfix" directory. One of the selling points of Postfix when compared to other SMTP servers like qmail and sendmail is the simplicity of it's configuration file as we will see.

Edit the file with your favorite editor:

sudo vi /etc/postfix/main.cf

or

sudo kate /etc/postfix/main.cf

or (if you prefer Gnome)

sudo gedit /etc/postfix/main.cf

Simply delete everything in there and copy/paste the following configuration. Make sure to read the comments and replace setting values to reflect your domain name, IP addresses, etc.

###
# Search hostnames in the hosts file before querying DNS.
# This option is very when trying a new domain that has not
# been published by your DNS server or when testing a replacement
# server.

smtp_host_lookup = native,dns

###
# Add a nice banner to say Hi to the mail clients.
# Add a message that reflects your company or school

smtpd_banner = Welcome to SchoolName ESMTP Server.

###
# Some default configuration settings
#

append_dot_mydomain = no # appending .domain is the MUA's job.
biff = no

###
# Alias maps and databases (Can be configured to use a Databases)
#

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases

###
# Your host name (FQDN)
# This is the name of the machine your server is being installed.
# Normally this name must be a FQDN (Fully Qualified Domain Name)
# that you need to buy. For testing you can add the name to the
# /etc/hosts file and enable postfix to query that file before querying
# a DNS server. Refer to the first option on this setup file

myhostname = you_host_name
mydomain = your_domain.com
myorigin = $mydomain

###
# Domains this server is the final destination. Any mail address
# whose domain is listed here will be delivered locally and no further
# forwarding will occur.

mydestination = $mydomain, $myhostname, localhost.$mydomain, localhost

###
# List of IP subnets allowed to use this server as mail relay. Only
# machines with these IP addresses will be able to send email to addresses
# other than your own domains defined in the "mydestination" option.

relayhost =
mynetworks_style = subnet
mynetworks = 127.0.0.0/8 192.168.1.1/28 169.168.2.1/28

###
# Allows to add comments to mail recipients as specified on RFC 822.
# (i.e. user+comment@your_domain.com)

recipient_delimiter = +

###
# Allow connections to all network interfaces in all IP protocol versions
#

inet_interfaces = all
inet_protocols = all

###
# Enable support for maildir format. The trailing / is important!
# The trailing / tells postfix it must use maildirs instead of mbox format.
#

home_mailbox = .maildir/

###
# Set user mail quota. (0 -> unlimited)
#

mailbox_size_limit = 0

###
# END
#

Some important notes you must remember:

myhostname/mydomain: Replace your_host_name with a FQDN hostname you have registered and your_domain.com with the domain part of the FQDN.

mydestination: This is the list (space separated) of all the domains for which this server is the final destination. This server will deliver all emails addressed to those domains locally (i.e. to your users maildir directories). As you can see we use the values from the mydomain and myhostname options.

mynetworks: Only the IP addresses (individual or subnets) that you specify here will be able to use the server to relay email to other domains. This means that if you want to send an email to another domain (not included in mydestination option) you must do so from a machine with one of the IP addresses specified here or you will get a relay denied error message.

home_mailbox: The default setting that comes with Kubuntu is set to "Maildir". I like to replace it with ".maildir" so the mail directory is hidden and normal users won't see it on Konqueror. Many times novice users mess with the Maildir folder and then I have to do magic to restore it.

Creating the Aliases database

We must create an alias database to receive mails for standard users (i.e. postmaster) and redirect those mails to a real user. To do this we simply edit the "/etc/aliases" file like:

alias1: root
alias2: root
postmaster: root
root: username

You can add as much aliases as you want. Make sure to create a root alias to a real user (i.e. username) that is the actual administrator of the system.

Once you finish adding aliases simply save the file and regenerate the aliases database file with the following command:

sudo newaliases

This will create the aliases.db file used by postfix to lookup aliases. If this file is not found then Postfix will fail to start.

Testing the server

Restart the postfix service to get your server running:

sudo /etc/init.d/postfix restart

To test if the service is running use the telnet command in console:

telnet your_host_name 25
Trying 127.0.1.1...
Connected to your_host_name.
Escape character is '^]'.
220 Welcome to SchoolName ESMTP Server.
quit
221 2.0.0 Bye
Connection closed by foreign host.

The text in blue is the server responses and the text in green are your input commands. The first response you get from the server is a 220 message with the text you set on the smtpd_banner option on the configuration file. With the quit command you simply disconnect from the server.

Some notes on mail relay

Most of the headaches people get setting mail servers are mostly due to the relay settings. To explain how postfix allows or denies email relay we must first understand what is a local domain and an external domain.

Local domains are those domains your mail server is responsible for and that usually are hosted in the server itself (i.e. the users mailbox are in the server hard disk). This domains are specified with the mydestination option in the configuration file.

For example if mydestination contains the domain "mydomain.edu" then any email the server receives with a destination address "juan@mydomain.edu" will be delivered to the user "juan" on the local machine. If the user "juan" does not exist in the local machine then the email will be bounced back to the sender with an user not found error.

An external domain is domain not hosted in your mail server. This includes all domains in existence except those configured in the mydestination option. If your email server receives an email with destination address "juan@otherdomain.com" then the server will reject or relay the email depending on your relay settings.

The default relay configuration is to reject all emails whose destination is not a local domain (i.e. not listed in mydestination option). This is a good default to avoid spammers to use your server to send spam but this will also restrict your server to receive and send emails locally only.

To allow your users to relay emails to external domains you must use the mynetwork option and specify the IP addresses or IP subnets of your users. If the server receives an email with destination address "juan@otherdomain.com" and the sender IP address is in the list of mynetwork option then the server will relay the email to the otherdomain.com mail server.

The drawback of using the mynetwork option is that the list of IP addresses you need to specify usually is not fixed. Some users may have dynamic IP addresses at home or school and sometimes your boss/professor may need to access his email from abroad (i.e. international meeting) making it almost impossible to maintain the list of addresses.

The solution to this problem is to enable roaming in the server. Roaming was implemented with a pop-before-send method in which users had to retrieve their emails using POP service before being able to relay emails with the SMTP server. Latter the SMTP protocol implemented the AUTH command that allows the server to challenge for a username and password before relaying any emails and this method is what we will tackle now.

Enabling Roaming

Postfix implements the AUTH command that allows users to authenticate themselves with the server before sending emails. This approach is more flexible than modifying the IP addresses list in the postfix main.cf configuration file.

For authentication we will use SASL that is very flexible. We could authenticate users using the system usernames/passwords or create a separate database to allow relaying to a selected groups of users only. In Kubuntu because Postfix runs chrooted in /var/spool/postfix we have change a couple paths to make SASL live in the false root. (ie. /var/run/saslauthd becomes /var/spool/postfix/var/run/saslauthd):

sudo mkdir -p /var/spool/postfix/var/run/saslauthd

Next we edit the SASL init configuration defaults "/etc/default/saslauthd" to look like:

START=yes
MECH_OPTIONS=""
THREADS=5
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
MECHANISMS="pam"

For Ubuntu Server the init defaults are a little different. If you are using Ubuntu Server (recomended) instead of Feisty/Edgy then use these defaults:

START=yes
NAME="saslauthd"
DAEMON="/usr/sbin/${NAME}"
DESC="SASL Authentication Daemon"
PWDIR=/var/spool/postfix/var/run/${NAME}
PIDFILE="${PWDIR}/${NAME}.pid"
MECH_OPTIONS=""
THREADS=5
PARAMS="-c -m ${PWDIR}"
MECHANISMS="pam"

We must also create the SASL's configuration file that tells postfix what authentication daemon it should connect for authentication. Create a file "/etc/postfix/sasl/smtp.conf" and insert the following options:

#pwcheck_method: authdaemond
#pwcheck_method: pwcheck
pwcheck_method: saslauthd
log_level: 3
mech_list: plain login

You can choose between the three authentication daemons: authdaemond, saslauthd and pwcheck. The most flexibe is saslauthd that can be configured to use several authentication methods like shadow, sasldb, pam, etc.

Now we must configure Postfix to enable SASL authentication by adding the following to the /etc/postfix/main.cf file:

###
# Postfix SASL Authentication configuration. To allow relay of mails
# from roaming users.
#

smtpd_sasl_local_domain =
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = /etc/postfix/sasl/smtpd
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination

###
# END

The "permit_mynetworks" makes SASL honor the "mynetwork" configuration option inside the main.cf file. This way users that are inside those IP networks can still relay emails even without authentication.

Ubuntu Server little hack

After configuring the SASL daemon I found that the init script "/etc/init.d/saslauthd" was failing to start the service. After some debugging I found that the directory checking code was failing causing the script to end with no warnings nor errors.

To fix this problem we must comment two small lines on the init script "/etc/init.d/saslauthd":

#dir=`dpkg-statoverride --list $PWDIR`
#test -z "$dir" || createdir $dir

Simply edit the script and search for the two lines above and make sure they have a "#" sign in front. Save and we will be ready for testing.

Restart and Test

We must restart both the saslauthd and postfix services for the new configurations to take effect.

sudo /etc/init.d/saslauthd restart
sudo /etc/init.d/postfix restart

And to test the new configuration we proceed as before but with an additional check for the AUTH command:

telnet your_host_name 25
Trying 127.0.1.1...
Connected to your_host_name.
Escape character is '^]'.
220 Welcome to SchoolName ESMTP Server.
ehlo localhost
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
quit
221 2.0.0 Bye
Connection closed by foreign host.

In this test you have to issue the "ehlo localhost" command and the server will respond with a lot of 250- messages. Each message refers to some feature the server supports. To see if the server supports AUTH simply look for the messages:

250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN

If you see this two messages in the list of responses you get then the server is configured to handle authentication.

Thursday, March 08, 2007

Lost your Akregator feed list? Get it back from the archived data

While using my laptop via ssh (with X forwarding) to read my Akregator's feeds the Internet connection failed and Akregator was forced to close. Once back home I fired up Akregator to finish reading my feeds and to my surprise my feed list was gone!.

The feeds.opml file that contains the list was empty and the feeds.opml~ backup file had only the default entries.

After the shock and some failed recovery attempts I found that the archived feed data was intact (i.e. ~/.kde/share/apps/akregator/Archive) and a closer look at the contents of that directory revealed that the archive filenames are in fact the feed's URL addresses with some minor changes (i.e. "/"'s replaced with "_"'s).

With some bash magic I was able to recover the feed list using the following script:

#!/bin/sh

list=`ls http*mk4`

for i in $list; do

feed=`echo $i | sed -e 's/http___/http:\/\//' \
| sed -e 's/_/\//g' | sed -e 's/.mk4//'`
echo "Adding $feed"
`akregator -a $feed`
done

Copy the script inside the archives directory and make sure you are inside that directory before running it. Once running you will see how Akregator is fired up and the feed list starts filling automatically.

Make sure to make the script executable (i.e. chmod +x script.sh) or simply run it using the bash interpreter (i.e. bash script.sh).

On the down side I was unable to recover the group information and all the feed's were added to the Imported group inside Akregator.

As an additional note I must mention that in the archive directory there are two binary files that seem to contain all the feed list information (including groups). One of the files is called "feedlistbackup.mk4" and it's contents is almost identical to that of the feeds.opml file with some additional binary elements added to it. Maybe someone that knows the binary format can create a simple program to recover the feed list from that file including group information.

Saturday, March 03, 2007

Booting the Thinkpad Rescue and Recovery with GRUB

The IBM Thinkpad laptops T40, T42, T42p, T43 and T43p that we use at our lab come with a 5GB Rescue and Recovery partition that can be booted up to restore the laptops OS to the pre-installed state.

In our lab the first thing our students do when they get one of these laptops is to wipe out the pre-installed Windows XP and install their favorite flavor of Linux. This is all good until I get the laptop back and have to restore it to the original state (school rules).

Our lab has the recovery cd's for the T40 and T42 series but not for the T43 series. And out of care I installed Kubuntu on a T43p laptop without creating the recovery CD's first. Even thought I left the rescue and recovery partition intact, Kubuntu by default writes Grub to the MBR effectively rendering it un-bootable.

Under the circumstances I was ready to ask budget to buy the recovery cd's from IBM (8,000~ Yen) but some research lead me to a better (free) solution using Grub itself.

Lucky me this thread on the Thinkwiki shows how to make Grub boot the rescue and recovery partition by editing it's entry on the menu.lst file.


title IBM Rescue and Recovery
root (hd0,1)
parttype (hd0,1) 0x0b
unhide (hd0,1)
chainloader +1



In the case of Kubuntu the rescue and recovery partition was automatically added to the menu.lst file during the installation and all you need to do is add the two lines in bold. After this when you boot the PC press ESC to get the Grub boot menu and select the IBM Rescue and Recovery partition.

With this I was able to create a recovery DVD's and save 8,000 Yen to the laboratory.

MATLAB xcb_xlib_lock Assertion in Feisty


Important Update: With the last update of Feisty Fawn (Herd 5 and up) using this work around will make your KDE unusable!. KDM will start but suddenly disappear leaving a black screen with a cursor blinking at the top left. From the console startx will run ok but with no means to start KDE from within it.

The good news is that with the new updates Matlab works out of the box without need of this work around at all. If you made a copy of the libxcb-xlib.so.0.0.0 file simply restore it and your system will be back on shape. Make sure to install the newest version of libxcb-xlib and Matlab should work without problems... It does on my system.


After I installed Feisty on my machine I installed and tried Matlab and for my unpleasant surprise Matlab started crashing every time I opened a new window. For example the command "plot" that opens a new window would crash Matlab with the following assertion:

MATLAB: xcb_xlib.c:41: xcb_xlib_lock: Assertion `!c->xlib.lock' failed.
Aborted


My only option at the moment was to keep using Edgy until Mathworks fixes the bug in any of their next releases (something that may never happen). But after I tested Feisty I really did not want to give up on it and as a good Linux user I asked the community and as always got a better solution.

It happens that this xcb_xlib_lock assertion bug is common not only in Matlab but in several other applications used in (K)Ubuntu. Reading the Ubuntu forums (see here and here) you can see that the problem has been addressed in two different ways:

One is to download the libx11-6_1.0.3-5_i386.deb package that comes with Edgy and force a downgrade with:

sudo dpkg --force-all --install libx11-6_1.0.3-5_i386.deb

I have not tried this method but some people report it works without problems.

The second method (the one I used) is to download and modify the source code so the Assertion check is bypassed. This sounds difficult but thanks to the Debian package management system it can be done in a few steps.

First download the source code and all build dependencies in a temp directory (i.e. ~/tmp):

apt-get source libxcb-xlib0
sudo apt-get build-dep libxcb-xlib0

Change to the source directory and edit the libxcb-1.0

cd libxcb-1.0
vi src/xcb_xlib.c

Feel free to use your favorite text editor (I like vim) and comment the following two lines:

assert(!c->xlib.lock); change to: //assert(!c->xlib.lock);

assert(c->xlib.lock); change to: //assert(c->xlib.lock);

This source file has few code lines so finding the lines to comment should be no problem at all. To comment them simple add "//" at the start of the line or you can delete them all together.

Next build the library:

./configure
make

If the configure script fails make sure you have the build-essential packages installed. If the make command finishes without any errors LOG OUT the X session and change to a console terminal. If you do not log out the next step is gonna crash you X very hard. To change to a console terminal press ++ (after you have logged out!) and to return back to the X console press ++.

cp /usr/lib/libxcb-xlib.so.0.0.0 /usr/lib/libxcb-xlib.so.0.0.0.orig
cp ~/tmp/libxcb-1.0/src/.libs/libxcb-xlib.so.0.0.0 /usr/lib

Replace ~/tmp with the temporary directory where you downloaded and compiled the source code. Finally restart your X11 server (reboot the PC to be sure) and now Matlab runs flawlessly on Feisty. You can also delete the tmp directory if you want.

I must mention that this bug is caused by bad programming practices when using the Xlib libraries and not a problem in Xlib itself. Read here and here for more information.

Thursday, March 01, 2007

Video Preview Thumbnails in Digikam

For a long time I thought Digikam did not support video thumbnail previews because I never got those on my system (Kubuntu). Of course this was all my fault for being too lazy to check first. A simple scrap over the Digikam's FAQ showed me:


Why don't I have thumbnails for my video files?
Why thumbnail generation stops at the first video file I have ?


First, check if it's the same in konqueror.

Digikam uses KDE to generate thumbnails and KDE rely on aRts to do that.
So, you should check your aRts configuration. ARts can use xine plugin for video files, so try if xine is able to play your video files. You may have to use additional codecs (eg libxine-extracodecs on debian/ubuntu if you already have libarts1-xine) for that.
Try also to play your video files using xine (if libarts1-xine is installed).

Remember that if it doesn't work in konqueror, it won't work in digikam...


Indeed opening the photos folder in Konqueror did not show any video thumbnails leading me to think this is not a missing feature but a missing configuration or library.

Reading carefully the FAQ entry above it hints about libarts1-xine being installed so I quickly entered the following command:

sudo aptitude install libarts1-xine

And in less than a minute I had video thumbnail previews of all my video clips on Digikam and in Konqueror too.