Subversion + OpenSSH Server + PuTTY Client Setup

This setup is of a subversion server running on Linux (Kubuntu 8.04) and subversion clients (Tortoise SVN) running in Windows (WinXP).

Pre-requisites

I assume you already have a subversion repository working and now you want to access it using ssh tunnelling for security reasons. Also the server must have remote ssh access and be configured to do Private Key authentication.

Setting up Putty

The best SSH client and related tools for Windows are the PuTTY family of programs that can be found at http://www.chiark.greenend.org.uk/~sgtatham/putty/. You can download the individual ".exe" files an put them in your path but I recommend you to use the installer that will install all the tools and set them nicely in your start menu.

Create public/private key pair


Objective

To access the subversion server via ssh you need a public/private key pair. To generate these key pair you can use the PuTTY Key Generator (PuTTYgen.exe) that was installed along with PuTTY installer.

Simply execute the PuTTYgen.exe program and press the "generate" button. After that keep moving your mouse, to generate some randomness, until the progress bar reaches the end. At this point you will get your public and private keys generated.

Make sure you add a "Key passphrase" that will be like a password to access the keys. The comment part is not necessary but is customary to put your email address.

Also make sure you are generating a SSH-2 key pair as SSH-1 is not secure. Choosing between RSA or DSA keys is a matter of application. It is known that DSA is fast at key generation and signing while RSA is faster at verification. So if your application requires a lot of signing (i.e. SSL Web Application) then DSA is good but if your application requires a lot of verifying (i.e. Subversion) then RSA is best. Since we are going to use this key for Subversion I recommend subversion but you are free to use the one you prefer.

Now we need to save your generated keys somewhere in your hard disk. The public key is visible in the PuTTYgen application widget (i.e. the long sequence of letters in the top text box). This key starts usually with "ssh-rsa" or "ssh-dsa" depending on what type of key you generated (RSA or DSA).

To save your public key on disk do not use the "save public key" button as this will generate a key in a format that is not compatible with OpenSSH. Simply copy and paste the key from the text box to a file on disk. You can save this file with whatever name you like but I recommend putting a ".pub" extension just as OpenSSH does.

The private key you must save it using the "save private key" button as this generates a file with ".ppk" extension that is understood only by PuTTY programs. Make sure you put the same name as the public key but with the ".ppk" extension instead of ".pub".

As an additional step if you would like to use your private key with other applications like SftpDrive so these can also connect to the OpenSSH server without need for a password, then you must convert your private key (.ppk file) to the OpenSSH key format. To do this simply select "Conversions" in the PuTTYgen.exe menu and then "Export OpenSSH key" to generate a key that can be used with OpenSSH compatible applications.

Note that with PuTTYgen.exe you can convert your private keys between PuTTY (.ppk) format and OpenSSH format so if you generate your public/private keys using OpenSSH ssh-keygen program you can use them in PuTTY by importing the private key using PuTTYgen.

Setting your public keys

The way the public/private key pair are used is simple. You keep the private key in your local machine usually in a key manager like ssh-agent or pageant and send your public key to all the servers you want to access via ssh.

To set up your public key in the SSH server (assumed OpenSSH server) simply log to the server shell using your account username and password (if you don't have one ask you administrator or if you are the administrator create one yourself). Then copy the contents of your public key file inside the file "~/.ssh/authorized_keys2".

To do this edit the "~/.ssh/authorized_keys2" file with any text editor (Vi, Nano, Kwrite, gEdit) and make sure it looks like in the next figure:



The key must start with ssh-rsa or ssh-dsa depending on the key you generated followed by the key all in one line (i.e. no new lines and spaces) and finally the comment again all in one line.

Finally make sure the authorized_keys2 file has correct permissions by executing "chmod 600 authorized_keys2".

You can set the public key in as many SSH servers as you like so you can access them all using the same private key.

Setting your private key

Your private key must be your most guarded secret and shall not be passed to anyone. Only you should have access to this key because anyone who gets his/her hands on that private key can access all the servers where you installed your public key and we don't want anyone to have that access power do we?

To use a your private key we usually use a SSH authentication agent like ssh-agent in Linux/Unix or pageant in Windows. The PuTTY installer comes with pageant.exe that is the one I use and recommend.

Simply execute pageant.exe from the application menu and a small icon will appear in your system tray. Right click the icon and select "Add key" on the context menu, then browse the directory where you saved your private key (i.e. .ppk) and select it. And that is it....

Testing SSH access

To test open the PuTTY.exe SSH client and create a new session to connect to your SSH server (make sure you save the session). When you open the session it should log you in without asking for any username and passwords. If it asks you for username and password then you did something wrong. Make sure the publickey format is correct on the server and that the private key is set in pageant. Of course make sure the OpenSSH server is running on the remote machine and that pageant is running on the local machine.

Once you can connect to the SSH server without need for a username and password then we can proceed to set up TortoiseSVN to access your Subversion repositories via SSH tunnels.

Setting TortoiseSVN with SSH tunnelling

Once you have SSH set to work with public/private key authentication and have pageant configured configuring TortoiseSVN is a breeze. All you have to do is make sure you use the correct URL format to access the repository.

If you have TortoiseSVN installed try to make a checkout and in the URL put something like:

svn+ssh://username@hostname/dir/to/repo

And you should be able to ckeck out that project over an SSH tunnel. You don't even need to have svnserve running on the server as the svn+ssh scheme tells the TortoiseSVN to start it for you and tunnel it over SSH.

Of course replace username with the username you use to access the SSH server and hostname with the IP address or FQDN of your SSH server. The path to the repository must be absolute path starting from the root "/" directory and have correct access privileges (read/write) for the username you use.


Resources

http://neubia.com/archives/000191.html
http://www.unixwiz.net/techtips/putty-openssh.html

Simple Mail Server (Kubuntu)

If you need a small personal mail server to send log reports and system alarms from your Kubuntu machines to a local and/or external mail address then you may use Exim4 that is the default mail server (MTA) that comes with (K)Ubuntu.

To install is very easy:

1 sudo aptitude install exim4 exim4-base

and to configure simply run:

1 sudo dpkg-reconfigure exim4-config

and in the first dialog (image below) select "internet site". For all next options I used the defaults that would allow local users and services to send emails locally and to remote addresses (i.e. gmail accounts).


Now to test the new server you can send a simple email to your Gmail account:

1 mail -s test mygmail@gmail.com

2 test mail

3 .

4 Cc:

The "-s" switch is the mail subject. After the command simply type whatever you want in the mail body. To finish the email press followed by a dot "." and then again. After this you will be prompted with "Cc:" so press and then check your Gmail account. You should have received an email with the message and subject you used above.

Now say you want to receive LogWatch reports or any other mail directed to root in your Gmail account. Then you can simply create an alias in the "/etc/aliases" file like:

1 # Added by installer for initial user

2 root:   mygmail@gmail.com

then rebuild the aliases database

1 sudo newaliases

If you require more complex mail configurations you may check my previous posts about Sendmail and Postfix setup in (K)Ubuntu here.

Receive Log reports via Email (Ubuntu)

When deploying servers in the hostile Internet a good administrator is faced with the need to monitor all the log files the server produces to ensure it is working correctly and to detect any security treats.

This can be a really time consuming task as a busy server can produce several megabytes worth of log files per day and if there are more than one server, then checking log files one by one is totally impractical, not to say useless.

To alleviate the burden of checking big log files every day I installed logwatch and it has proven to be very useful. It gives a very complete summary of all your log files with the most relevant information and very well presented with per service sub-sections.

In Ubuntu Server installing the logwatch package worked out of the box for almost all my running services (Courier-POP, Postfix, OpenSSH). The relevant command is:

sudo aptitude install logwatch

The only service it did not work was the web server because logwatch is configured to use Apache log files by default while I use Lighttpd as web server.

How to Configure Logwatch to parse Lighttpd log files (Ubuntu)

The easiest way to customize logwatch is to create an override.conf file inside the /etc/logwatch/conf/ directory. To tell logwatch to parse lighttpd log files we create the override.conf file and add the following:

logfiles/http: LogFile = lighttpd/*access.log.1
logfiles/http: LogFile = lighttpd/*access.log
logfiles/http: Archive = lighttpd/*access.log.*.gz

This is assuming the lighttpd log files are inside "/var/log/lighttpd" directory. If they are not change the paths to reflect the location on your system. You can add as many log files as you want (i.e. virtual domains) by adding all three entries above for each log file.

Now you will get some nice reports about the web usage in your server. Make sure you read the HOWTO-Customize-LogWatch file to learn more about logwatch. This file is usually inside the "/usr/share/doc/logwatch" directory in .gz format. To read it you can use the command:

zcat /usr/share/doc/logwatch/HOWTO-Customize-LogWatch.gz | less

Get Logwatch reports to via email

By default Logwatch sends the reports it generates to root. To send the reports to a different local user or external email address you can edit the "/etc/aliases" file like:

1 # Added by installer for initial user

2 root:   myuser, mygmail@gmail.com

and then rebuild the aliases database:

1 sudo newaliases

In the example above all Logwatch reports will be received by the local user "myuser" so you may access the reports via the mbox file at "/var/mail/mysuer" and to the external address mygmail@gmail.com that you may read using the Gmail web interface.

Important Note: by default the mail transfer agent of (K)Ubuntu does not allow relay of messages to external addresses (i.e. gmail addresses). To change this you may follow the instructions to set a small personal mail server here.

Logwatch vs Logcheck

I installed both programs and for me Logwatch is far more useful than logcheck. Logcheck will only parse the log files related to security (i.e. auth.log) and simply send you and email with the access denied entries. The information Logcheck provides is no different that I get by looking at the log files directly.

Logwatch in the other hand provides relevant information not only about security issues but also from all the services running on the server. The information is well summarized and presented in a way it is easy to get a general and a detailed view of the server status and operation.

How to make Webalizer work with Lighttpd in Ubuntu server

To get more visually compelling statistics about your web server usage patterns you can use Google Analytics that is a powerful tool. But if you prefer a simpler alternative but still powerful enough then I recommend installing Webalizer.

In Ubuntu server if you are using Lighttpd instead of Apache make sure to change the configuration file (/etc/webalizer.conf) to point to the corresponding log file (i.e. LogFile /var/log/lighttpd/access.log.1) or it won't work.

Logwatch example report

################### LogWatch 7.1 (11/12/05) ####################
Processing Initiated: Thu May 31 06:25:02 2007
Date Range Processed: yesterday
( 20075月-30 )
Period is day.
Detail Level of Output: 5
Type of Output: unformatted
Logfiles for Host: makarena
##################################################################

--------------------- courier mail services Begin ------------------------

Connections: 100 Times
Protocol POP3 - 100 Times
Host 192.33.11.109 - 1 Time
Host 199.120.17.15 - 8 Times
Host 195.10.13.49 - 3 Times
Host 13.9.18.11 - 88 Times



Logins: 96 Times
Protocol POP3 - 96 Times, 3790856 Bytes
User paprika - 1 Time, 21511 Bytes
Host 14.8.13.19 - 1 Time, 21511 Bytes
User mondongolia - 88 Times, 3389830 Bytes
Host 13.9.18.11 - 88 Times, 3389830 Bytes
User juanito3 - 7 Times, 379515 Bytes
Host 124.10.17.1 - 6 Times, 379515 Bytes
Host 124.10.13.4 - 1 Time, 0 Bytes



---------------------- courier mail services End -------------------------


--------------------- Cron Begin ------------------------



Commands Run:
User root:
run-parts --report /etc/cron.hourly: 24 Time(s)
[ -d /var/lib/php4 ] && find /var/lib/php4/ -type f -cmin +$(/usr/lib/php4/maxlifetime) -print0 | xargs -r -0 rm: 48 Time(s)
[ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -r -0 rm: 48 Time(s)
test -x /usr/sbin/anacron || run-parts --report /etc/cron.daily: 1 Time(s)

---------------------- Cron End -------------------------


--------------------- httpd Begin ------------------------

35.45 MB transferred in 2953 responses (1xx 0, 2xx 2766, 3xx 151, 4xx 36, 5xx 0)
1817 Images (9.34 MB),
17 Documents (11.90 MB),
867 Content pages (9.56 MB),
16 Redirects (0.00 MB),
236 Other (4.66 MB)

Attempts to use known hacks by 1 hosts were logged 1 time(s) from:
20.11.61.11: 1 Time(s)


A total of 1 sites probed the server
20.11.61.11

Requests with error response codes
404 Not Found
/%7Enbalan/JSP/: 1 Time(s)
/_vti_bin/shtml.exe/_vti_rpc: 1 Time(s)
/_vti_inf.html: 1 Time(s)
/comment: 2 Time(s)
/en/home/photo_gallery: 1 Time(s)
/en/user: 3 Time(s)
/favicon.ico: 2 Time(s)
/imagefile/filepath/1110/small/IMG_0189.jpg: 1 Time(s)
/imagefile/filepath/1112/small/IMG_0191.jpg: 2 Time(s)
/imagefile/filepath/1113/small/IMG_0192.jpg: 1 Time(s)
/robots.txt: 17 Time(s)
/~nbalan/Concurrency/html/CIC: 2 Time(s)
/~nbalan/Concurrency/html/index2.html: 1 Time(s)
/~shda/toppage.html: 1 Time(s)

A total of 9 ROBOTS were logged

---------------------- httpd End -------------------------

--------------------- pam_unix Begin ------------------------

cron:
Sessions Opened:
root: 121 Time(s)

sshd:
Sessions Opened:
admin: 3 Time(s)

su:
Sessions Opened:
(uid=0) -> nobody: 3 Time(s)


---------------------- pam_unix End -------------------------


--------------------- POP-3 Begin ------------------------


[POP3] Connections:
=========================
Host | Connections
------------------------------------------------------------- | -----------
::ffff:14.8.13.19 | 1
::ffff:13.9.18.11 | 88
::ffff:84.10.1.15 | 6
::ffff:8.10.13.9 | 1
---------------------------------------------------------------------------
96



[POP3] Logout stats (in MB):
============================
User | Logouts | Downloaded | Mbox Size
--------------------------------------- | ------- | ---------- | ----------
hbr | 7 | 0.36 | 0
admin | 1 | 0.02 | 0
sna | 88 | 3.23 | 0
---------------------------------------------------------------------------
96 | 3.62 | 0.00

---------------------- POP-3 End -------------------------


--------------------- postfix Begin ------------------------



3716764 bytes transferred
204 messages sent
204 messages removed from queue

Top ten senders:
1 messages sent by:
root (uid=0):


SASL Authenticated messages from:
unknown[13.9.18.11]: 2 Time(s)


Connections lost:
Connection lost while CONNECT : 1 Time(s)

---------------------- postfix End -------------------------


--------------------- SSHD Begin ------------------------


Users logging in through sshd:
admin:
13.9.18.15: 2 times
14.7.13.19: 1 time

---------------------- SSHD End -------------------------


--------------------- Disk Space Begin ------------------------

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root 227G 33G 183G 15% /
/dev/sdb1 367G 36G 313G 11% /mnt
/dev/sda5 228M 24M 193M 11% /boot


---------------------- Disk Space End -------------------------


###################### LogWatch End #########################

As you can see the report is well structured and provides relevant information about SMTP/POP services, Web statistics and access security. It even reports that someone is proving the server and that a known vulnerability has been tested on my server... now I can take actions like blocking that IP address from all access to the server using iptables.

From looking at the thousand of megabytes of log files it would have been a little difficult to spot this particular security treat.

File System Benchmarks for Postfix Mail Server

After some research I decided to use XFS file system to host my next mail server (Postfix). This decision was based only on comments and suggestions I read in forums and some benchmarks published on the Internet about linux file systems [1].

I installed my new mail server with Postfix as MTA [2] and Courier as POP [3] and leaved a free 60GB partition to make some benchmark testing. This way I was able to extract some hard numbers to validate my desicion. The results show XFS is indeed the best option when running a mail server, at least for Postfix.

Test Environment

The test setup is very simple: I have one server with Ubuntu Server 6.06 LTS installed with the most recent updates and a second client machine with Kubuntu Feisty running Postal/Rabid [4] software used to load the server with SMTP/POP requests.

The machines specifications are:

	Server Machine	Client Machine
OS	Ubuntu Server 6.06 LTS	Kubuntu Feisty Fawn
CPU	Intel Xeon 2.40GHz	Intel Pentium III 800MHz
RAM	512MB	256MB
Disk	SEAGATE ST373307LC 73408MB	Maxtor 6Y120L0 SCSI 122942 MB

In the server machine I had Ubuntu Server installed with Postfix and Courier-POP configured and added 500 user accounts with Maildir boxes.

The client side is my desktop PC running Kubuntu Feisty where I simply run postal and rabid to load the server with a lot of SMTP and POP requests. I apply no limits on the number of requests per minute as we are testing the disk input/output throughput rather than the mail service itself.

Testing Procedure

On the server side I followed these steps:

• Create test partition with one of the file systems under consideration. Especial care was taken to use the creation options that are known to increase each particular file system performance as stated in various benchmarks and forums.
  
  
  
  • mkfs.ext3 -J size=100 -m 1 -O dir_index,filetype,has_journal
  
  
  • mkfs.reiserfs -b 4096 -s 16386
  
  
  • mkfs.xfs -f -l size=64m -d agcount=16
  
  
  • mkfs.jfs -s 64
  
  
  
  


• Mounted the newly created partition to a testing mount point. Again I tried my best to use the recommended mount options in order to improve each file system performance. In the case of ext3 and reiserfs we tested all three journaling methods they support.
  
  
  
  • mount -t ext3 -o noatime,nodiratime,data=journal
  
  
  • mount -t ext3 -o noatime,nodiratime,data=ordered
  
  
  • mount -t ext3 -o noatime,nodiratime,data=writeback
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=journal
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=ordered
  
  
  • mount -t reiserfs -o noatime,nodiratime,notail,data=writeback
  
  
  • mount -t xfs -o noatime,nodiratime,logbufs=8
  
  
  • mount -t jfs
  
  
  
  


• Copied the necessary files from the root partition to the testing partition to create a fully functional chroot environment and chrooted it:
  


• Once inside the chroot environment I start all services. First deleting all the syslogs and then starting sysklogd, postfix, courier-authdaemon and courier-pop in that order. This way all mailboxes, logs and configuration files are read/write in the file system under test.

On the client side I simply start postal and rabid to send unlimited SMTP and POP requests to the server:

postal -m 5 -r 24000 -t 5 -c 3 -s 0 server_ip userlist.txt
rabid  -p 5 -c 5 -r 24000 -s 0 server_ip userlist.txt

The userlist.txt contains the list of all 500 user accounts with password that were created on the server after installation. The postal command above will establish five SMTP connections to the server and send as much messages as it can with sizes between 0 and 5kbytes to all 500 users. On the other side rabid will establish as much POP connections as it cans to the sever from five different processes and download a maximum of five messages per connection.

Both postal and rabid were executed at the same time on the client machine to send concurrent SMTP/POP requests to the server. After 10 minutes we start "vmstat" on the server to obtain CPU and RAM load statistics:

sudo vmstat 60 30 > /var/log/mail.vmstat

This command will output CPU, Disk and RAM load statistics of the server for 30 minutes in one minute intervals and save them on the mail.vmstat file.

Once vmstat finishes we take the mail.log and mail.vmstat files and analyzed them using some custom ruby scripts.

Results


This graph shows the average number of successful SMTP transactions between Postfix (server) and postal (client) per minute. Each connection represents up to 3 messages with sizes between 0 and 5kbytes. No rocket science is needed to see that XFS outperforms by a large margin all other file systems. I managed to get a maximum of 1700 messages delivered per minute using XFS while JFS follows with about 900 messages per minute. ReiserFS and Ext3 in all jornaling modes never passed the 700 messages per minute rank.


This graph shows the average number of successful POP transactions between Courier-POP (server) and rabid (client) per minute. Each connection established by rabid will download and delete 5 messages from the server.

We can see here that contrary to SMTP we have ReiserFS and Ext3 being the best performers with very high averages. XFS and JFS do not perform that well but with 1800 and 1300 transactions per minute respectively they are not that bad.

The bad performance XFS and JFS get here agrees with the fact that they perform badly when deleting large amounts of small files. If we do not delete the messages from the server after download (or use IMAP instead of POP) we may get better performance statistics.


This graph shows the same as the first graph but instead of connections per minute we show the actual number of bytes that were transmitted per minute. This shows that we actually read/write messages from the disk.

Because we ran postal and rabid at the same time we do not have valid statistics for the number of bytes per minute downloaded via POP. This is because the POP client download rate is limited by the SMTP upload rate. In other words the POP client cannot download from the server more than what the SMTP client sends to the server.

This is easy to verify on the mail log files were you will see a lot of POP connections that retrieved zero bytes. Also you can verify this by graphing the number of bytes per minute for POP and you will see that all bars will be equal or slightly lower but never larger than the values obtained for SMTP.



We have here the CPU utilization (real) obtained by running "vmstat" on the server while it was receiving SMTP/POP requests. We see here that XFS is less CPU intensive that all other file systems except for JFS that outperforms all others by a large margin.

This result is surprising as all benchmarks state that XFS is the more CPU hungry of all the file systems. I took this CPU value from the vmstat output data by adding the user and system CPU utilization values. If this is not a good CPU utilization measurement please let me know.


I am not sure what this graph represents but from the "vmstat" man page this value represents: "wa: Time spent waiting for IO". Since we are interested here in file system performance I assume this value is important so I show it here for those that know what it is. To my understanding this represents average CPU time per minute wasted waiting for IO so high values are bad while low values are good. Comments on this graph are appreciated.

Conclusions

XFS simply rocks as a file system for SMTP and POP servers, period. The results I got showed that XFS is the best for delivery of large amounts of mails, it is good for POP and does not consume all CPU.

After Thoughts

These results show clearly that the file system used can greatly affect the applications performance (mail in this case). It can be interesting to test other applications that make heavy use of disk like File Servers and/or Database Servers or even test other SMTP (qmail) or POP (Cyrus, Dovecot) servers and compare their performance over different file systems and Postfix/Courier.

This is the first benchmark test so I am not expert and may have made mistakes. Any comments to improve or correct my tests are welcome.

References

[1] Best File System for Server Usage
[2] Basic SMTP Server Setup In Kubuntu
[3] Basic POP Server Setup In Kubuntu
[4] Postal - SMTP and POP benchmark program.

Best File System for Server Usage

Problem Review

One of my mail servers failed when the ext3 partition that has the users maildirs ran out of I-Nodes. After this it was clear that ext3 is not scalable enough to handle large number of users, each one with large amounts of files/directories. This is a typical setting for a IMAP/SMTP service that allows users to organize their mails on the server rather than downloading them.

After a lot of research (Google) I found a lot of benchmarks and articles that made one thing clear: there is really no real silver bullet in the journaled file systems camp. For example ReiserFS is famous for being the best handling small files and it is true it does but it has really bad performance when handling big files. On the other hand XFS can handle massively large files really fast but has other problems as high CPU consumption and low performance deleting large amounts of files.

There are a lot more examples like these were one file system shines in one test but leaves a lot to desire on others.

After a week of deliberation I have decided to give XFS a try and use it from now on on all my future server installations.

Why XFS?

After looking at some benchmarks
here and here it is easy to see that XFS performs well or better in almost all tests. For those tests it is not the best it usually falls behind the best by a small margin.

The only benchmark it fails by a large margin is for deleting large number of files. This is one of the most criticized problems of XFS but there are tweaks to improve it (see below).

Another problem often criticized is that XFS seems to be CPU intensive when compared to other file systems. Looking at the benchmarks we can see this is true but we can also see that the high CPU usage is not the highest in all tests.

When evaluating journaled file systems there are a lot of features that must be accounted for. A very nice review of what too look can be found here and here.

The decision between XFS and JFS seems strange as JFS appears to be as good as XFS in all benchmarks using less CPU than XFS. The decision to go with XFS usually comes from the fact that it is more mature and has more features than JFS or other journaling file system in fact.

Some of the features that are important to servers that XFS supports are;

• Extended Inode Attributes: This allow use to add custom attributes to inodes giving use more granular control over the file systems. For example we could add tags to inodes and search files based on the tags.


• Online Defragmentation: Modern file systems are resistant to certain point to fragmentation of file but as the available free space is reduced the amount of fragmentation can increase to an alarming rate. The ability to defragment the disk without unmounting it is desirable for a high availability server


• Online backup utitlities: Again for high availability servers the capacity to backup and restore complete partitions is a plus. XFS has utilities that allow backup and restore of entire partitions online.


• Dynamic I-Node Allocation: This allows us to create large number of files in the file system without worries of running out of I-Nodes. All modern file systems, except for ext3, support this feature.


• Support For Quotas: Limiting the disk usage capacity in a multi user environment is very important to be able to offer services with varying levels of quality of service.


• Allocation Groups: As far as I know this is an XFS only feature that allows processes running concurrently to allocate space in the file system concurrently without interfering with each other. With this XFS enables free space and inode management operations to proceed in parallel throughout the file system increasing performance on large partitions. Current dual-core/multi-processor systems can take advantage of this feature to further increase overall system performance.
  

Some Remarks Against XFS

• XFS is very slow deleting large amounts of small files!!.
  
  
  I haven't found yet a situation were I needed to actually delete a lot of small files from a mail server. On the other hand I do have found situations were I need to delete large files (i.e. 4GB database tables and 2GB log files). Also after reading this post it seems that it is possible to tweak XFS to perform even better than ext3 when deleting large number of files.
  



• XFS only logs meta data on the journal so you will loose data on a crash
  
  
  The only File System that is known to be robust to system crashes and sudden power outages is ext3 and only if it is in ordered or journal mode. All other file systems prefer to log only meta data to improve performance. You must decide between a robust but slower file system (i.e. ext3, ReiserFS) or a faster but less robust file system (i.e. XFS, JFS).
  
  For a server the trend goes to speed performance rather than robustness to crashes and system failures. The Philosophy behind this trend is that if you are intend to install a server you surely have also a disaster recovery strategy. Having a nice backup/restore plan and more than one independent power source are also part of installing a high availability, high performance server.
  



• If you need maildir storage then ReiserFS is the way to go!
  
  
  Maybe but the benchmarks show that XFS is not that far behind ReiserFS when handling small files and at the same time is the best handling large files. ReiserFS may be well optimized to handle small files but it has really bad performance problems handling medium to large file sizes.
  
  Also to note is that the incoming Reiser 4 is the worst performer in all speed tests and is even more CPU hungry than XFS.
  



• What about JFS?? I heard it performs equal to or even better than XFS with less CPU load and more robust to sudden crashes!![/B]
  
  Yes I heard it too and some benchmarks show that JFS can be as good or better than XFS in read/write performance. But as mentioned above JFS is relative new (in the linux camp) and is not as mature nor has as many features as XFS. For example it has some problems with user quotas.
  

So XFS is the all purpose File System

Sorry to say but XFS is no silver bullet here. As I said my conclusions are based on server application needs. I would not recommend using XFS or any other meta-data only journaling file system on a desktop computer, specially on a laptop that has more probability of power outages. The amount of complaints about all KDE configuration files being zero-length or filled with zeros after a crash talk by themselves (link).

XFS is not robust to crashes and power outages as ext3 can be so I will stay with ext3 for my desktop installations and my laptop. Ext3 is the recommended default by Ubuntu and with some tweaks it is possible to improve it's performance further (see here and here).

Future Work

All my conclusions are based on others hard worked results and opinions of others. To validate my conclusions I have collected a one week worth of qmail log files I am planning to use them to obtain mail usage statistics statistics and create some more realistic disk usage patterns. Current benchmarks are based on synthetic tests that may not reflect real load patterns in real mail or database applications,

Once I obtain some useful statistics from my logs I plan to perform some benchmarks over all file systems mentioned here with their respective performance tweaks. Once I get some results they will be posted here in this blog.

General Resources

http://linuxgazette.net/issue55/florido.html
http://bulma.net/impresion.phtml?nIdNoticia=1154
http://linux.inet.hr/first_benchmarks_of_the_ext4_file_system.html
http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://linuxgazette.net/122/piszcz.html
http://www.debian-administration.org/articles/388
http://adria.fesb.hr/~jsosic/mojbench.html

XFS Resources

http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html
http://en.wikipedia.org/wiki/XFS
http://everything2.com/index.pl?node_id=1479435
https://launchpad.net/ubuntu/+bug/37435
http://www-128.ibm.com/developerworks/linux/library/l-fs10.html
http://www.ubuntuforums.org/showthread.php?t=246969
http://forums.gentoo.org/viewtopic-t-488215.html
https://ols2006.108.redhat.com/reprints/chinner-reprint.pdf
http://everything2.com/index.pl?node_id=1479435

ReiserFS Resources

http://www-128.ibm.com/developerworks/library/l-fs.html
http://www.moon-soft.com/program/bbs/readelite666510.htm
http://en.wikipedia.org/wiki/ReiserFS
http://en.wikipedia.org/wiki/Reiser4

Ext3 Resources

http://www-128.ibm.com/developerworks/linux/library/l-fs8.html
http://www.suseforums.net/index.php?showtopic=29773
http://forums.gentoo.org/viewtopic-t-488215.html
http://en.wikipedia.org/wiki/Ext3
http://en.wikipedia.org/wiki/Ext4

JFS Resources

http://www-128.ibm.com/developerworks/library/l-jfs.html

Testing SMTP and POP services from a console

Test a non-secure SMTP server

To test a non-secure SMTP server we simply connect to the port 25 of that machine using telnet:

telnet smtp_ip_address 25
Trying smtp_ip_address...
Connected to smtp_ip_address.
Escape character is '^]'.
220 Welcome to Postfix ESMTP Server
ehlo localhost
250-testmail
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from:auser@internaldomain.org
250 2.1.0 Ok
rcpt to:noone@externaldomain.org
554 5.7.1 : Relay access denied
rcpt to:noone@internaldomain.com
550 5.1.1 : Recipient address rejected: User unknown in local recipient
rcpt to:realuser@internaldomain.com
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
Hello There!!
.
250 2.0.0 Ok: queued as 7ECF05100C6
quit
221 2.0.0 Bye
Connection closed by foreign host.

The green text are the commands you send to the server while the blue text indicates the server responses. The example above is for a simple Postfix SMTP server.

Testing the AUTH command

From the previous example we can see that the Postfix SMTP server supports the AUTH command that allows roaming users to use the server as relay. To test this command we must first encode our authentication credentials in base64 format:

perl -MMIME::Base64 -e 'print encode_base64("\000username\@domain.net\000password")'

You must have the MIME::Base64 perl module in your system (is installed by default in Kununtu). Also make sure you replace the username, domain.net and password in the command to reflect your own valid values.

After the command finishes it will output your authentication credentials as a base64 encoded string that looks like:

AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=

Copy that string somewhere and proceed to connect to the server as in the previous example:

telnet smtp_ip_address 25
Trying smtp_ip_address...
Connected to smtp_ip_address.
Escape character is '^]'.
220 Welcome to Postfix ESMTP Server
ehlo localhost
250-testmail
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
auth plain AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=
235 2.0.0 Authentication successful
mail from:auser@internaldomain.org
250 2.1.0 Ok
rcpt to:noone@externaldomain.org
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
hahahaha
.
250 2.0.0 Ok: queued as 7B6265100C6
quit
221 2.0.0 Bye
Connection closed by foreign host.

In this example we used the "auth plain" command to authenticate ourselves against the SMTP server. Once authenticated we are allowed to relay mails to any domain (not only internal domains) from anywhere (roaming).

A closer look at the the previous example will reveal that when we send the "rcpt to" command to an external domain (noone@externaldomain.org) the server response was a "554 relay access denied". In this example we are authenticated with the server (via the AUTH command) so now the "rcpt to" command was accepted.

Test a non-secure POP server

Similar to SMTP we can test a POP server by connecting to it using telnet:

telnet pop_ip_address 110
Trying pop_ip_address...
Connected to pop_ip_address.
Escape character is '^]'.
+OK Hello there.
user username
+OK Password required.
pass password
+OK logged in.
list
+OK POP3 clients that break here, they violate STD53.
1 520
.
quit
+OK Bye-bye.
Connection closed by foreign host.

Again green are your commands and blue are the server responses. In this example we simply connect to the server, authenticate and get the list of mails in our inbox. We can also see the contents of the mails with the "retr num" command replacing num with the mail number as displayed by the "list" command.

Test secure SMTP/POP servers

If the SMTP/POP server supports encryption (TLS/SSL) we won't be able to test it using simple text telnet connections. In this case we must use the openssl command utility that will take care of all TLS/SSL negotiations, connect to the server and allow us to send plain text commands over the encrypted connection.

For SMTP we can check if the server support TLS encryption by looking at the server response to the "ehlo" command. If we get a "250-STARTTLS" within all the responses then the server supports TLS.

To connect to the SMTP server over a TLS encrypted connection we can use the following command:

openssl s_client -starttls smtp -crlf -connect smtp_ip_address:25

You will get a lot of messages from this command that show the SSL negotiations. You really don't need to care about all those messages. When openssl finishes setting up the encrypted connection you will see the same welcome message we got on the first example:

220 Welcome to Postfix ESMTP Server

From here on you can proceed with plain text commands as in the first two examples.

In the case of POP the server it is a little more difficult to check if it supports TLS/SSL connections. POP uses a different port (995 instead of 110) for SSL encrypted connections. This means that as long as the server uses the default ports if the server has the port 995 open we can assume it supports POP over SSL.

To connect to the POP server we must use the openssl utility command to take care of all the SSL negotiation procedures. The following command will do the trick:

openssl s_client  -crlf -connect pop_ip_address:995

After all the SSL certificates are exchanged and the encrypted connection is established we are greeted by the POP server hello message as in the previous POP example:

+OK Hello there.

From this point on we can proceed issuing normal text commands and getting server responses. All the encryption part will be handled automatically by openssl.

References

I learned about testing the AUTH command and SMTP over TLS from the following page: http://qmail.jms1.net/test-auth.shtml

Add encryption to SMTP and POP (TLS/SSL)

In previous posts we configured a fully functional SMTP and POP servers for a single domain with local system users and with roaming capabilities using SASL authentication.

In this post we further improve our setup to allow encrypted communications in SMTP and POP. This way we add a security layer to protect users from mishap from third persons and protect sensitive information.

Creating the SSL certificate for SMTP

The recommended way to do this is to create a certificate request and get it signed by a Certificate Authority but for demonstration purposes I will create a self signed certificate.

For postfix we must create a cert/key pair using the openssl command. In your home directory or a temporary directory you can create the SSL cert/key pair like this:

openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 \
        -nodes -keyout smtpd.key -keyform PEM -days 1095 -x509

You will be asked a set of questions that you can fill any way you feel like. This is only to create an identity that must be unique among certificates. Note that the above is a one line command split in two due to space constraints.

Next we create some directories to store the SSL certificates. I like to store them inside the postfix configuration directory.

sudo mkdir /etc/postfix/ssl
sudo mkdir /etc/postfix/ssl/cert
sudo mkdir /etc/postfix/ssl/private

sudo chmod 755 /etc/postfix/ssl/cert
sudo chmod 710 /etc/postfix/ssl/private

Then we move the certificate pair to the newly created directories:

sudo mv smtpd.cert /etc/postfix/ssl/cert
sudo mv smtpd.key /etc/postfix/ssl/private

Enabling TLS in Postfix

Simply edit the /etc/postfix/main.cf file and add the following:

###
# TLS parameters

smtpd_tls_cert_file=/etc/postfix/ssl/cert/smtpd.cert
smtpd_tls_key_file=/etc/postfix/ssl/private/smtpd.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache
smtpd_tls_auth_only = no

###
# END

The "smtpd_tls_auth_only" option can be used for added security. It forces the communication the be encrypted before any credentials (user and passwords) are transmitted. Unfortunately some client's may fail to authenticate with this setting so it defaults to "no".

Finally restart postfix:

sudo /etc/init.d/postfix restart

Enable POP over SSL

If you followed my previous post then you already have installed the required courier-pop-ssl package. If not simply install it with this command:

sudo aptitude install courier-pop-ssl

Generating the SSL certificate for POP

By default Kubuntu uses an auto generated SSL certificate (pop3d.pem) that you can find inside the "/etc/courier" directory. The pop3d.pem file is actually a cert/key pair combined into a single file. For our POP server we will use the same cert/key pair we created for the SMTP server to generate our pop3d.pem file.

In your home or a temporary directory:

sudo cat /etc/postfix/ssl/cert/smtpd.cert > pop3d.pem
sudo openssl gendh >> pop3d.pem
sudo cat /etc/postfix/ssl/private/smtpd.key >> pop3d.pem
sudo chmod 600 pop3d.pem

Now backup the auto generated certificate and replace it with the new one:

sudo cp /etc/courier/pop3d.pem /etc/courier/pop3d.pem.orig
sudo rm /etc/courier/pop3d.pem
sudo cp pop3d.pem /etc/courier

You can put the generated pop3d.pem file anywhere you want as long as you change the path in the courier configuration file (i.e. /etc/courier/pop3d-ssl). For security reasons it may be better to keep it in the default place.

Finally restart the service but this time the SSL one.

sudo /etc/init.d/courier-pop-ssl restart

To force your users to use always SSL simply disable the normal POP server and leave only the one with SSL.

sudo /etc/init.d/courier-pop stop
sudo /etc/init.d/courier-pop-ssl restart

Also to stop courier-pop from starting at boot time we must remove the start up script from the rcX.d directories with the following command:

sudo update-rc.d -f courier-pop remove

This way when the server reboots only the SSL enable POP service will be started. If for some reason you require to start the normal POP service at boot time again you can do it with the command:

sudo update-rc.d  courier-pop defaults

Maildir

Do not forget to modify the "/etc/courier/pop3d-ssl" configuration file in Kubuntu Feisty to use ".maildir" instead of the "Maildir" for storage of mails. As with normal POP simply edit the file and replace the value of the MAILDIRPATH variable.

In the case of Ubuntu Server Edition all these services (POP, POP-SSL, IMAP, IMAP-SSL) use the configuration inside "/etc/default/courier" to determine the maildir's directory. Simply make sure you modify the file and replace the MAILDIR value with ".maildir"

Testing

In previous posts we were able to test our configurations by connecting via telnet to the corresponding ports. Now that we have enabled encryption we cannot test with simple text commands via telnet.

It is still possible to test the SMTP and POP services over encrypted connections in a console using the openssl utility but I will explain how to do this in a future post.

Basic POP Server Setup in Kubuntu (Courier)

In a previous post I explained how to set up a very basic SMTP server to deliver mails to users local maildir's and to relay mails to external domains (only for authorized users).

Now to allow users to download/view their emails in their own PC's we must enable a POP or IMAP service. I will only deal with POP here but since the configuration procedure for POP and IMAP in Kubuntu is very similar everything said in this post for POP applies with minor changes to IMAP.

Installing the Software

In Kubuntu installing the courier POP servers cannot be easier:

sudo aptitude install courier-pop courier-pop-ssl

Note that we are also installing the courier-pop-ssl package that allows POP over SSL connections. I will talk more about SSL in a future blog post. During the installation you will be asked if you want directory based configuration files, chose not to and then press ok when asked.

Creating Maildirs

The maildir vs mbox formats to store emails is an old debate and I will not deal with it here. I like maildir storage simply because I had a lot of lock and corrupted file problems in the past due to the "single file" mbox format and migrating to maildir format kept those problems away.

By default Courier-POP in Kubuntu uses the maildir format to store users mail. The only problem I have with the default settings is that it uses a folder called "Maildir" inside each user home directory. I personally prefer to hide the mail's directory from the users to avoid accidents (i.e. total deletion).

Now to enable each user's maildir we must first create the maildir directory:

sudo maildirmake /home/username/.maildir
sudo chown -R username:usergroup /home/username/.maildir
sudo chmod -R 700 /home/username/.maildir

You can replace .maildir with whatever you like. The dot "," at the begining makes that directory hidden so it would not be listed normally in the file manager (i.e. Konqueror) or by issuing "ls" on the console.

Creating the .maildir directory must be done for all current users in the system. For new users the maildir creation can be automated by creating a maildir directory inside the "/etc/skel" template directory:

sudo maildirmake /etc/skel/.maildir

Now every time a new user is created the .maildir folder will be automatically created in that user's home directory.

Configuring Courier POP

The default settings that come with Kubuntu are enough to get the services running. The only change we have is to replace the "Maildir" directory with ".maildir". This is done easily by editing the "/etc/courier/pop3d" configuration file:

sudo kate /etc/courier/pop3d   

    and

replace MAILDIRPATH=Maildir with MAILDIRPATH=.maildir

In the case of Ubuntu Server the above configuration won't work. Instead of modifying the /etc/courier/pop3d file you must edit the /etc/default/courier file:

sudo kate /etc/default/courier   

    and

replace MAILDIR=Maildir with MAILDIR=.maildir

Restart and Test

Finally we restart the POP/IMAP service:

sudo /etc/init.d/courier-pop restart

And to test it we connect to the port 110 using telnet:

telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK Hello there.
user username
+OK Password required.
pass userpass
+OK logged in.
list
+OK POP3 clients that break here, they violate STD53.
.
quit
+OK Bye-bye.
Connection closed by foreign host.

As in the previous post the blue text are server response messages and the green text are your input commands. To authenticate you simply use the "user" and "pass" commands. Once authenticated you can list your emails with the "list" command and exit with the "quit" command.

Coming Next

In the old days our server is now ready to go online but these days this configuration would be considered insecure. Current state of the art in servers must provide security mechanisms like encrypted communications to avoid leakage of sensitive/personal information.

In my next post I will explain how to encrypt SMTP and POP using TLS/SSL to have a more secure server ready for the current Internet.

Basic SMTP Server Setup in Kubuntu (Postfix)

Kubuntu is mostly a desktop oriented linux distribution, but this does not mean that it cannot be used as a server. Several guides exist on the Ubuntu Wiki that teach you how to setup different kinds of servers.

This small tutorial will show you how to set up a local SMTP server (Postfix) for a small office or school with a single domain.

This configuration was tested on Kubuntu Feisty Fawn and in Ubuntu Server distributions. This configuration may work also with Dapper and with Edgy but I have not tested it.

Installing Software

In a fresh installation of Kubuntu simply install the postfix and support packages:

sudo aptitude install postfix libsasl2 sasl2-bin libsasl2-modules libdb3-util

Note that the above command installs SASL too that will be used to enable roaming in our server. During the installation of postfix the package manager will give you the opportunity to configure the most basic options of postfix. I simply select not to configure it at the time and latter modify the configurations files with a text editor. This way I get more control on the look of the file and can format it in a easy to read layout.

Configuring the server

The main configuration file main.cf of postfix can be found inside the "/etc/postfix" directory. One of the selling points of Postfix when compared to other SMTP servers like qmail and sendmail is the simplicity of it's configuration file as we will see.

Edit the file with your favorite editor:

sudo vi /etc/postfix/main.cf

   or

sudo kate /etc/postfix/main.cf

   or (if you prefer Gnome)

sudo gedit /etc/postfix/main.cf

Simply delete everything in there and copy/paste the following configuration. Make sure to read the comments and replace setting values to reflect your domain name, IP addresses, etc.

###
# Search hostnames in the hosts file before querying DNS.
# This option is very when trying a new domain that has not 
# been published by your DNS server or when testing a replacement
# server.

smtp_host_lookup = native,dns

###
# Add a nice banner to say Hi to the mail clients.
# Add a message that reflects your company or school

smtpd_banner = Welcome to SchoolName ESMTP Server.

###
# Some default configuration settings 
#

append_dot_mydomain = no     # appending .domain is the MUA's job.
biff = no

###
# Alias maps and databases (Can be configured to use a Databases)
#

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases

###
# Your host name (FQDN)
# This is the name of the machine your server is being installed.
# Normally this name must be a FQDN (Fully Qualified Domain Name)
# that you need to buy. For testing you can add the name to the
# /etc/hosts file and enable postfix to query that file before querying
# a DNS server. Refer to the first option on this setup file

myhostname = you_host_name
mydomain = your_domain.com
myorigin = $mydomain

###
# Domains this server is the final destination. Any mail address
# whose domain is listed here will be delivered locally and no further 
# forwarding will occur.

mydestination = $mydomain, $myhostname, localhost.$mydomain, localhost

###
# List of IP subnets allowed to use this server as mail relay. Only 
# machines with these IP addresses will be able to send email to addresses
# other than your own domains defined in the "mydestination" option.

relayhost =
mynetworks_style = subnet
mynetworks = 127.0.0.0/8 192.168.1.1/28 169.168.2.1/28

###
# Allows to add comments to mail recipients as specified on RFC 822. 
# (i.e. user+comment@your_domain.com)

recipient_delimiter = +

###
# Allow connections to all network interfaces in all IP protocol versions 
# 

inet_interfaces = all
inet_protocols = all

###
# Enable support for maildir format. The trailing / is important!
# The trailing / tells postfix it must use maildirs instead of mbox format.
#

home_mailbox = .maildir/

###
# Set user mail quota. (0 -> unlimited)
#

mailbox_size_limit = 0

###
# END
#

Some important notes you must remember:

myhostname/mydomain: Replace your_host_name with a FQDN hostname you have registered and your_domain.com with the domain part of the FQDN.

mydestination: This is the list (space separated) of all the domains for which this server is the final destination. This server will deliver all emails addressed to those domains locally (i.e. to your users maildir directories). As you can see we use the values from the mydomain and myhostname options.

mynetworks: Only the IP addresses (individual or subnets) that you specify here will be able to use the server to relay email to other domains. This means that if you want to send an email to another domain (not included in mydestination option) you must do so from a machine with one of the IP addresses specified here or you will get a relay denied error message.

home_mailbox: The default setting that comes with Kubuntu is set to "Maildir". I like to replace it with ".maildir" so the mail directory is hidden and normal users won't see it on Konqueror. Many times novice users mess with the Maildir folder and then I have to do magic to restore it.

Creating the Aliases database

We must create an alias database to receive mails for standard users (i.e. postmaster) and redirect those mails to a real user. To do this we simply edit the "/etc/aliases" file like:

alias1:      root
alias2:      root
postmaster:  root
root:        username

You can add as much aliases as you want. Make sure to create a root alias to a real user (i.e. username) that is the actual administrator of the system.

Once you finish adding aliases simply save the file and regenerate the aliases database file with the following command:

sudo newaliases

This will create the aliases.db file used by postfix to lookup aliases. If this file is not found then Postfix will fail to start.

Testing the server

Restart the postfix service to get your server running:

sudo /etc/init.d/postfix restart

To test if the service is running use the telnet command in console:

telnet your_host_name 25
Trying 127.0.1.1...
Connected to your_host_name.
Escape character is '^]'.
220 Welcome to SchoolName ESMTP Server.
quit
221 2.0.0 Bye
Connection closed by foreign host.

The text in blue is the server responses and the text in green are your input commands. The first response you get from the server is a 220 message with the text you set on the smtpd_banner option on the configuration file. With the quit command you simply disconnect from the server.

Some notes on mail relay

Most of the headaches people get setting mail servers are mostly due to the relay settings. To explain how postfix allows or denies email relay we must first understand what is a local domain and an external domain.

Local domains are those domains your mail server is responsible for and that usually are hosted in the server itself (i.e. the users mailbox are in the server hard disk). This domains are specified with the mydestination option in the configuration file.

For example if mydestination contains the domain "mydomain.edu" then any email the server receives with a destination address "juan@mydomain.edu" will be delivered to the user "juan" on the local machine. If the user "juan" does not exist in the local machine then the email will be bounced back to the sender with an user not found error.

An external domain is domain not hosted in your mail server. This includes all domains in existence except those configured in the mydestination option. If your email server receives an email with destination address "juan@otherdomain.com" then the server will reject or relay the email depending on your relay settings.

The default relay configuration is to reject all emails whose destination is not a local domain (i.e. not listed in mydestination option). This is a good default to avoid spammers to use your server to send spam but this will also restrict your server to receive and send emails locally only.

To allow your users to relay emails to external domains you must use the mynetwork option and specify the IP addresses or IP subnets of your users. If the server receives an email with destination address "juan@otherdomain.com" and the sender IP address is in the list of mynetwork option then the server will relay the email to the otherdomain.com mail server.

The drawback of using the mynetwork option is that the list of IP addresses you need to specify usually is not fixed. Some users may have dynamic IP addresses at home or school and sometimes your boss/professor may need to access his email from abroad (i.e. international meeting) making it almost impossible to maintain the list of addresses.

The solution to this problem is to enable roaming in the server. Roaming was implemented with a pop-before-send method in which users had to retrieve their emails using POP service before being able to relay emails with the SMTP server. Latter the SMTP protocol implemented the AUTH command that allows the server to challenge for a username and password before relaying any emails and this method is what we will tackle now.

Enabling Roaming

Postfix implements the AUTH command that allows users to authenticate themselves with the server before sending emails. This approach is more flexible than modifying the IP addresses list in the postfix main.cf configuration file.

For authentication we will use SASL that is very flexible. We could authenticate users using the system usernames/passwords or create a separate database to allow relaying to a selected groups of users only. In Kubuntu because Postfix runs chrooted in /var/spool/postfix we have change a couple paths to make SASL live in the false root. (ie. /var/run/saslauthd becomes /var/spool/postfix/var/run/saslauthd):

sudo mkdir -p /var/spool/postfix/var/run/saslauthd

Next we edit the SASL init configuration defaults "/etc/default/saslauthd" to look like:

START=yes
MECH_OPTIONS=""
THREADS=5
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
MECHANISMS="pam"

For Ubuntu Server the init defaults are a little different. If you are using Ubuntu Server (recomended) instead of Feisty/Edgy then use these defaults:

START=yes
NAME="saslauthd"
DAEMON="/usr/sbin/${NAME}"
DESC="SASL Authentication Daemon"
PWDIR=/var/spool/postfix/var/run/${NAME}
PIDFILE="${PWDIR}/${NAME}.pid"
MECH_OPTIONS=""
THREADS=5
PARAMS="-c -m ${PWDIR}"
MECHANISMS="pam"

We must also create the SASL's configuration file that tells postfix what authentication daemon it should connect for authentication. Create a file "/etc/postfix/sasl/smtp.conf" and insert the following options:

#pwcheck_method: authdaemond
#pwcheck_method: pwcheck
pwcheck_method: saslauthd
log_level: 3
mech_list: plain login

You can choose between the three authentication daemons: authdaemond, saslauthd and pwcheck. The most flexibe is saslauthd that can be configured to use several authentication methods like shadow, sasldb, pam, etc.

Now we must configure Postfix to enable SASL authentication by adding the following to the /etc/postfix/main.cf file:

###
# Postfix SASL Authentication configuration. To allow relay of mails
# from roaming users.
#

smtpd_sasl_local_domain =
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = /etc/postfix/sasl/smtpd
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks,  reject_unauth_destination

###
# END

The "permit_mynetworks" makes SASL honor the "mynetwork" configuration option inside the main.cf file. This way users that are inside those IP networks can still relay emails even without authentication.

Ubuntu Server little hack

After configuring the SASL daemon I found that the init script "/etc/init.d/saslauthd" was failing to start the service. After some debugging I found that the directory checking code was failing causing the script to end with no warnings nor errors.

To fix this problem we must comment two small lines on the init script "/etc/init.d/saslauthd":

#dir=`dpkg-statoverride --list $PWDIR`
#test -z "$dir" || createdir $dir

Simply edit the script and search for the two lines above and make sure they have a "#" sign in front. Save and we will be ready for testing.

Restart and Test

We must restart both the saslauthd and postfix services for the new configurations to take effect.

sudo /etc/init.d/saslauthd restart
sudo /etc/init.d/postfix restart

And to test the new configuration we proceed as before but with an additional check for the AUTH command:

telnet your_host_name 25
Trying 127.0.1.1...
Connected to your_host_name.
Escape character is '^]'.
220 Welcome to SchoolName ESMTP Server.
ehlo localhost
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
quit
221 2.0.0 Bye
Connection closed by foreign host.

In this test you have to issue the "ehlo localhost" command and the server will respond with a lot of 250- messages. Each message refers to some feature the server supports. To see if the server supports AUTH simply look for the messages:

250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN

If you see this two messages in the list of responses you get then the server is configured to handle authentication.

Faking POP

As part of a networking class at my lab we teach students protocols by asking them to implement a simple POP3 client in Java. The client must connect to the POP3 server and interact with the server as a normal POP client.

As soon as the class started we realized the university POP server uses SSL to encrypt the connection (for obvious security reasons) and asking the students to implement the SSL part of the client was not an option.

We needed a POP server set up quickly before the students could present their programs or at least something that behaves like a POP server. A quick search resulted in FakePOP and the great Ubuntu has it in their repositories.

I simply issued the following command:

sudo aptitude install xinetd fakepop

Then create a xinetd configuration file "/etc/xinet.d/fakepop" that contains the following:

service pop3
{
server = /usr/sbin/in.fakepop
disable = no
socket_type = stream
protocol = tcp
user = nobody
wait = no
}

and restart the xinetd daemon:

sudo /etc/init.d/xinetd restart

and in less than a minute I had a POP3 server (or at least something that behaves like one) on my machine and the class was saved.

Also you can create a custom message to be delivered to anyone connecting to the fakepop server. You can create as many messages as you want by putting them as text files inside the "/etc/fakepop" directory. Some examples can be found in the "/usr/share/doc/fakepop/examples" directory like the one below:

From: System Administrator
Subject: example message - text only
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

This is a simple message.
It contains only text.

Using eruby with Lighttpd

Some times I am asked to develop some web application clients for a particular web service. The clients I have developed include for example J2ME/Doja clients for cellular phones that download information form a special server and fat clients developed on C++/J2SE that access some special service in a corporate server.

Usually the server I am supposed to interact with is developed in parallel with the client I am developing and there are always situations were I finish a client feature but I cannot test it because that particular feature is not implemented yet in the server.

To solve this problem and to avoid discussions with the server developers and the manager I decided to implement a fake server that would behave like the real server to the client but in the back is a simple program that receives the client requests and returns the expected response (or a bogus response if I want to test the client robustness to server failures).

So my dilema was to choose the best server and dynamic language that would allow me to develop small server side Unit Tests to validate my client implementation. From all the possible options Lighttpd combined with eruby offered the most simplicity with the power of Ruby.

Apache/PHP?? since I started to learn Rails it has been very difficult for me to go back to PHP. IIS/ASP?? well from my previous posts you can see I am not a Microsoft fan and I don't want to install a Windows Server just for testing. Rails?? it very good and flexible I don't want to be creating complete Rails applications everytime I need to test a little feature in my application.

Lighttpd in Gentoo Linux

My linux server is running the Gentoo distribution and fortunately Lighttpd is in the portage tree. Installing lighttpd can be done with one command:

# emerge lighttpd

At this time the version of lighttpd is 1.4.11. One of the reasons I choose lighttpd is that configuring Lighttpd is a breeze. With little effort I can have a working HTTP server. The configuration file I use is below. Pay attention to the blue lines that are required to serve eruby pages.

# Load the modules I need
server.modules = (
"mod_rewrite", # For rewriting URLs
"mod_cgi" # For dispaching eruby
)

# Server configuration
server.port = 80
server.username = "lighttpd"
server.groupname = "lighttpd"
server.pid-file = "/var/run/lighttpd.pid"
server.indexfiles = ( "index.html", "index.rhtml" )
server.document-root = "/var/www/localhost/htdocs"
server.errorlog = "/var/www/localhost/htdocs/log/server.log"


mimetype.assign = (
".css" => "text/css",
".gif" => "image/gif",
".htm" => "text/html",
".html" => "text/html",
".jpeg" => "image/jpeg",
".jpg" => "image/jpeg",
".js" => "text/javascript",
".pdf" => "application/pdf",
".png" => "image/png",
".txt" => "text/plain",
)

cgi.assign = ( ".rhtml" => "/usr/bin/eruby" )

This configuration usually goes in the /etc/lighttpd/lighttpd.conf file. Also we need to install eruby that is also in the portage tree:

# emerge eruby

We can now restart the lighttpd server:

# /etc/init.d/lighttpd restart

eruby Hello World

How this all works?? is very simple in fact. When a client requests a rhtml file the server (lighttpd) calls the eruby interpreter with the rhtml file as input. The eruby parses the rhtml file and replaces all the ruby code with their respective outputs (Usually HTML) and returns it to the server. Finally the server sends the generated HTML to the client.

The point here is that the ruby code that is contained in the rhtml file can be anything ruby has to offer and with the power of ruby it can be practically anything!. Let's see a simple example:

Create a file hello.rhtml with the following code and save it in the htdocs directory you configured on the lighttpd.conf file.

<HTML>
<HEAD>
<TITLE><%= "Hello World Page" %></TITLE>
</HEAD>
<BODY>
<%= "Hello World" %>
</BODY>
</HTML>

Anyone familiar with Rails/PHP/ASP/JSP can see easily what this code does. What eruby does is to replace the ruby code in between the <% and %> tags with it's return value. In our case we simply created a String object in the tags.

There is far more to eruby than this and when eruby is combined with other Ruby libraries like CGI, rMagick, Base64, etc with get a rich and powerfull set of tools to create from small test web applications to full blow web applications a la Rails.

Some external links with a little more information.

eRuby: Getting Started with Ruby on the Web

Ruby and the Web

Virtual Domains with Vpopmail on a qmail server.

In previous post I explained how to install a qmail server on a Red Hat ES machine and how to manage the server connections via tcpserver. Now I will add support to virtual domains using vpopmail.

This guide is based on this one with some modifications to fit my needs. The usual disclaimer is that everything written here that is good is from that guide while the bad tidbits are mine.

Preparing Red Hat ES for vpopmail

To ease the work I chose to use the rpm packages that come with Red Hat by default. First we install the MySQL client and server packages that are included on the Red Hat installation CD's.

# rpm -ivh mysql-4.1.7-4.RHEL4.1 mysqlclient10-3.23.58-4.RHEL4.1 mysql-server-4.1.7-4.RHEL4.1

We also install the devel package needed to compile Vpopmail with MySQL support:

# rpm -ivh mysql-devel-4.1.7-4.RHEL4.1

The default installation of MySQL server has a black password for the root account. This of course is a security risk and we must assign a password to the root account as soon as possible.

First start the MySQL server

# /etc/init.d/mysql start

Now we setup a root password

# mysqladmin -u root password 'mysql-root-pwd'

Before we continue to install Vpopmail we need to install the zlib-devel package but I could not find it in any of the Red Hat CD's. To install it I downloaded a zlib-devel rpm from here and installed it. You can use wget to download the file.

# rpm -ivh zlib-devel-1.2.1.2-1.i386.rpm

Installing Vpopmail

We need some preparations before we can compile and install Vpopmail on the machine. First we create a vchkpw group and vpopmail user with home directory set to /home/vpopmail.

# groupadd -g 89 vchkpw
# useradd -g vchkpw -u 89 -d /home/vpopmail vpopmail

Now we create a vpopmail database on the MySQL server and assign a user and password in the database to access it.

# mysql -u root --password="mysql-root-pwd"

# mysql> CREATE DATABASE vpopmail;
# mysql> GRANT select,insert,update,delete,create,drop ON vpopmail.* TO vpopmailuser@localhost IDENTIFIED BY 'vpoppasswd';
# mysql> quit

In the above command we created a database called vpopmail that will store all the vpopmail domains/users/log/config etc. Then we create a user "vpopmailuser" with password "vpoppasswd" with all priviledges on the vpopmail database. Please feel free to change the user and password to your favorite ones.

Next we create a configuration file that vpopmail processes will use to connect to the MySQL server:

# mkdir ~vpopmail/etc
# chown vpopmail.vchkpw ~vpopmail/etc
# echo "localhost|0|vpopmailuser|vpoppasswd|vpopmail" > ~vpopmail/etc/vpopmail.mysql
# chown vpopmail.vchkpw ~vpopmail/etc/vpopmail.mysql
# chmod 640 ~vpopmail/etc/vpopmail.mysql

Don't copy and paste the echo command above. You must change the vpopmailuser and vpoppasswd to reflect the username and password you configured on the database.

Now we are ready to install Vpopmail on the machine. Download the source package and decompress it in anywhere you feel like:

# wget http://kent.dl.sourceforge.net/sourceforge/vpopmail/vpopmail-5.4.13.tar.gz
# tar xvfz vpopmail-5.4.13.tar.gz
# cd vpopmail-5.4.13

Now we configure the source for our needs. Before running the configuration script make sure you know what features you want enabled/disabled. Once we compile changing a feature will require you to reconfigure and recompile the source code. In my case I am interested mostly on roaming and MySQL support:

# ./configure --enable-roaming-users \ # Enable POP-before-SMTP functionality.
--enable-logging=p \ # Log to syslog errors with passwords
--disable-passwd \ # Disable /etc/passwd (or shadow) accounts
--enable-auth-module=mysql \ # Enable MySQL backend support
--enable-auth-logging \ # Record time and ip of last auth attempt
--enable-sql-logging \ # Enable authentication logging to MySQL/Postgres.
--enable-valias \ # Store email aliases in MySQL
--enable-mysql-limits \ # Use MySQL to store limits.
--enable-many-domains # Store all virtual domain users in a single table.

To learn about other options you can always use the command:

# ./configure --help | less

Finally we simply compile by calling the make command:

make
make install-strip

That's it for installing Vpopmail. Now we can create/delete virtual domains and users within those domains using all Vpopmail commands found in the /home/vpopmail/bin directory:

To add a domain :

/home/vpopmail/bin/vadddomain yourdomain.com yourpassword

To add a mailbox:

/home/vpopmail/bin/vadduser someone@yourdomain.com apassword

To remove a mailbox

/home/vpopmail/bin/vdeluser someone@yourdomain.com

To remove a domain :

/home/vpopmail/bin/vdeldomain yourdomain.com

To change a user's password

/home/vpopmail/bin/vpasswd someone@yourdomain.com newpassword

(Or you can do it via qmailadmin)

To lookup info about a user

/home/vpopmail/bin/vuserinfo someone@yourdomain.com

Configureing tcpserver to manage POP3 connections

As with qmail SMTP we would like our clients to connect to the pop server from anywhere in the world. To do so we setup tcpserver in the qmail init script to accept tcp connections to port 110 and excecute the qmail-pop3d command. To do this simply add the next line after the tcpserver invocation for qmail-smtpd inside the start() method

/usr/local/bin/tcpserver -H -R -l test.canmail.jp -v 0 pop3 var/qmail/bin/qmail-popup test.canmail.jp /home/vpopmail/bin/vchkpw /var/qmail/bin/qmail-pop3d Maildir 2>&1 | /var/qmail/bin/splogger pop3d 3 &

Save the qmail init script and restart the service:

#/etc/init.d/qmail restart

Testing SMTP and POP3 via Telnet

Since SMTP and POP are text based protocols we can easily test them via telnet.

First let's create a test domain and a user:

First create the virtual domain. This command will also create a postmaster account and ask you for a password.

# /home/vpopmail/bin/vadddomain testdomain.jp

Next create a test account in the newly create domain

# /home/vpopmail/bin/vadduser testuser@testdomain.jp

Let's try first sending emails to the testuser using our newly installed qmail SMTP. In the next examples the blue lines are our commands and the red ones are the server responses.

From any machine different from the server using a telnet client:
# telnet server_ip_address 25
Trying 218.45.218.139...
Connected to 218.45.218.139.
Escape character is '^]'.
220 example.jp ESMTP
helo example.jp
250 example.jp
mail from: anyaccount@anydomain.jp
250 ok
rcpt to: testuser@testdomain.jp
250 ok
data
354 go ahead
hello world
.
250 ok 1159529928 qp 17735
quit
221 example.jp
Connection closed by foreign host.

With the above commands we send an email to the testuser at the testdomain in the server. You can see the message was delivered as the server responded with a 250 ok message.

Now let's do the same but send a message to a remote account. For example a gmail account.

From any machine different from the server using a telnet client:
# telnet server_ip_address 25
Trying 218.45.218.139...
Connected to 218.45.218.139.
Escape character is '^]'.
220 example.jp ESMTP
helo example.jp
250 example.jp
mail from: testuser@testdomain.jp
250 ok
rcpt to: anyaccount@gmail.com
553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
quit
Connection closed by foreign host.

The server did not allow us to send the message because the gmail.com domain is not configured on the list of rcpthosts of qmail. This is very good since we don't want anyone using our server so relay email (specially spam). In the previous example the message was delivered because the testdomain.jp was added to the rcpthosts list of qmail by the vadddomain command.

But then how can the legitimate users that exist in our server send emails to other domains that are not in the rcpthosts list?? Is not like our users are going to send email only between them is it??. Well that is why I configured Vpopmail with roaming support. With roaming support after a user connects to the POP server, Vpopmail records the IP address of the machine the user connected from and modifies the tcp.smtp file to allow the recorded IP address to relay emails to any other domain.

This is called POP-before-SMTP since to be able to send emails with the server the user first needs to connect at least once with the POP server. Let's try by connecting to the POP server via telnet:

telnet 218.45.218.139 110
Trying 218.45.218.139...
Connected to 218.45.218.139.
Escape character is '^]'.
+OK <17844.1159530536@example.jp>
user testuser@testdomain.jp
+OK
pass testuserpass
+OK
list
+OK
1 269
.
quit
+OK
Connection closed by foreign host.

The list command ask the server to list the testuser emails and as we can see the listing shows one new email in the inbox. This is the email we sent on the first test. After this the IP address of the machine we are running the telnet client must be recorded on the tcp.smtp database and if we retry the second example the email will be accepted for delivery this time. Go ahead.. you can try.

Warning: Check your IPTables

When I did remote tests the first time I could not connect remotely to the port 110 on the server. After some time I realized that Red Hat ES by default has some very strict iptable rules (Firewall) set that would not allow remote POP connections.

The easiest way to allow remote SMTP (port 25) and POP3 (port 110) connections is to modify the iptables configuration file in the /etc/sysconfig directory.

# nano /etc/sysconfig/iptables

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 110 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Make sure the lines in red are in the file and then restart the iptables service:

# /etc/init.d/iptables restart

Final Notes

If you want a first class email server then DO NOT USE this guide. This configuration is the old school of email server configurations for a medium size application with not so much users. Current email servers use SMTP-Auth instead of POP-before-SMTP and some sort of encryption like TLS for SMTP and SSL for POP.

For a real email server that comply with current standards you should look at Cyrus-SASL for SMTP and POP encryption, SMTP-Auth for qmail and Cyrus-IMAP.

Why I still use this old school configuration?? Ask my boss... This email server is to replace a legacy server that has been working for years. Maybe changing the way it works can be annoying to the current users so I was asked to replicate exactly the old server.