Receive Log reports via Email (Ubuntu)

When deploying servers in the hostile Internet a good administrator is faced with the need to monitor all the log files the server produces to ensure it is working correctly and to detect any security treats.

This can be a really time consuming task as a busy server can produce several megabytes worth of log files per day and if there are more than one server, then checking log files one by one is totally impractical, not to say useless.

To alleviate the burden of checking big log files every day I installed logwatch and it has proven to be very useful. It gives a very complete summary of all your log files with the most relevant information and very well presented with per service sub-sections.

In Ubuntu Server installing the logwatch package worked out of the box for almost all my running services (Courier-POP, Postfix, OpenSSH). The relevant command is:

sudo aptitude install logwatch

The only service it did not work was the web server because logwatch is configured to use Apache log files by default while I use Lighttpd as web server.

How to Configure Logwatch to parse Lighttpd log files (Ubuntu)

The easiest way to customize logwatch is to create an override.conf file inside the /etc/logwatch/conf/ directory. To tell logwatch to parse lighttpd log files we create the override.conf file and add the following:

logfiles/http: LogFile = lighttpd/*access.log.1
logfiles/http: LogFile = lighttpd/*access.log
logfiles/http: Archive = lighttpd/*access.log.*.gz

This is assuming the lighttpd log files are inside "/var/log/lighttpd" directory. If they are not change the paths to reflect the location on your system. You can add as many log files as you want (i.e. virtual domains) by adding all three entries above for each log file.

Now you will get some nice reports about the web usage in your server. Make sure you read the HOWTO-Customize-LogWatch file to learn more about logwatch. This file is usually inside the "/usr/share/doc/logwatch" directory in .gz format. To read it you can use the command:

zcat /usr/share/doc/logwatch/HOWTO-Customize-LogWatch.gz | less

Get Logwatch reports to via email

By default Logwatch sends the reports it generates to root. To send the reports to a different local user or external email address you can edit the "/etc/aliases" file like:

1 # Added by installer for initial user

2 root:   myuser, mygmail@gmail.com

and then rebuild the aliases database:

1 sudo newaliases

In the example above all Logwatch reports will be received by the local user "myuser" so you may access the reports via the mbox file at "/var/mail/mysuer" and to the external address mygmail@gmail.com that you may read using the Gmail web interface.

Important Note: by default the mail transfer agent of (K)Ubuntu does not allow relay of messages to external addresses (i.e. gmail addresses). To change this you may follow the instructions to set a small personal mail server here.

Logwatch vs Logcheck

I installed both programs and for me Logwatch is far more useful than logcheck. Logcheck will only parse the log files related to security (i.e. auth.log) and simply send you and email with the access denied entries. The information Logcheck provides is no different that I get by looking at the log files directly.

Logwatch in the other hand provides relevant information not only about security issues but also from all the services running on the server. The information is well summarized and presented in a way it is easy to get a general and a detailed view of the server status and operation.

How to make Webalizer work with Lighttpd in Ubuntu server

To get more visually compelling statistics about your web server usage patterns you can use Google Analytics that is a powerful tool. But if you prefer a simpler alternative but still powerful enough then I recommend installing Webalizer.

In Ubuntu server if you are using Lighttpd instead of Apache make sure to change the configuration file (/etc/webalizer.conf) to point to the corresponding log file (i.e. LogFile /var/log/lighttpd/access.log.1) or it won't work.

Logwatch example report

################### LogWatch 7.1 (11/12/05) ####################
Processing Initiated: Thu May 31 06:25:02 2007
Date Range Processed: yesterday
( 20075月-30 )
Period is day.
Detail Level of Output: 5
Type of Output: unformatted
Logfiles for Host: makarena
##################################################################

--------------------- courier mail services Begin ------------------------

Connections: 100 Times
Protocol POP3 - 100 Times
Host 192.33.11.109 - 1 Time
Host 199.120.17.15 - 8 Times
Host 195.10.13.49 - 3 Times
Host 13.9.18.11 - 88 Times



Logins: 96 Times
Protocol POP3 - 96 Times, 3790856 Bytes
User paprika - 1 Time, 21511 Bytes
Host 14.8.13.19 - 1 Time, 21511 Bytes
User mondongolia - 88 Times, 3389830 Bytes
Host 13.9.18.11 - 88 Times, 3389830 Bytes
User juanito3 - 7 Times, 379515 Bytes
Host 124.10.17.1 - 6 Times, 379515 Bytes
Host 124.10.13.4 - 1 Time, 0 Bytes



---------------------- courier mail services End -------------------------


--------------------- Cron Begin ------------------------



Commands Run:
User root:
run-parts --report /etc/cron.hourly: 24 Time(s)
[ -d /var/lib/php4 ] && find /var/lib/php4/ -type f -cmin +$(/usr/lib/php4/maxlifetime) -print0 | xargs -r -0 rm: 48 Time(s)
[ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -r -0 rm: 48 Time(s)
test -x /usr/sbin/anacron || run-parts --report /etc/cron.daily: 1 Time(s)

---------------------- Cron End -------------------------


--------------------- httpd Begin ------------------------

35.45 MB transferred in 2953 responses (1xx 0, 2xx 2766, 3xx 151, 4xx 36, 5xx 0)
1817 Images (9.34 MB),
17 Documents (11.90 MB),
867 Content pages (9.56 MB),
16 Redirects (0.00 MB),
236 Other (4.66 MB)

Attempts to use known hacks by 1 hosts were logged 1 time(s) from:
20.11.61.11: 1 Time(s)


A total of 1 sites probed the server
20.11.61.11

Requests with error response codes
404 Not Found
/%7Enbalan/JSP/: 1 Time(s)
/_vti_bin/shtml.exe/_vti_rpc: 1 Time(s)
/_vti_inf.html: 1 Time(s)
/comment: 2 Time(s)
/en/home/photo_gallery: 1 Time(s)
/en/user: 3 Time(s)
/favicon.ico: 2 Time(s)
/imagefile/filepath/1110/small/IMG_0189.jpg: 1 Time(s)
/imagefile/filepath/1112/small/IMG_0191.jpg: 2 Time(s)
/imagefile/filepath/1113/small/IMG_0192.jpg: 1 Time(s)
/robots.txt: 17 Time(s)
/~nbalan/Concurrency/html/CIC: 2 Time(s)
/~nbalan/Concurrency/html/index2.html: 1 Time(s)
/~shda/toppage.html: 1 Time(s)

A total of 9 ROBOTS were logged

---------------------- httpd End -------------------------

--------------------- pam_unix Begin ------------------------

cron:
Sessions Opened:
root: 121 Time(s)

sshd:
Sessions Opened:
admin: 3 Time(s)

su:
Sessions Opened:
(uid=0) -> nobody: 3 Time(s)


---------------------- pam_unix End -------------------------


--------------------- POP-3 Begin ------------------------


[POP3] Connections:
=========================
Host | Connections
------------------------------------------------------------- | -----------
::ffff:14.8.13.19 | 1
::ffff:13.9.18.11 | 88
::ffff:84.10.1.15 | 6
::ffff:8.10.13.9 | 1
---------------------------------------------------------------------------
96



[POP3] Logout stats (in MB):
============================
User | Logouts | Downloaded | Mbox Size
--------------------------------------- | ------- | ---------- | ----------
hbr | 7 | 0.36 | 0
admin | 1 | 0.02 | 0
sna | 88 | 3.23 | 0
---------------------------------------------------------------------------
96 | 3.62 | 0.00

---------------------- POP-3 End -------------------------


--------------------- postfix Begin ------------------------



3716764 bytes transferred
204 messages sent
204 messages removed from queue

Top ten senders:
1 messages sent by:
root (uid=0):


SASL Authenticated messages from:
unknown[13.9.18.11]: 2 Time(s)


Connections lost:
Connection lost while CONNECT : 1 Time(s)

---------------------- postfix End -------------------------


--------------------- SSHD Begin ------------------------


Users logging in through sshd:
admin:
13.9.18.15: 2 times
14.7.13.19: 1 time

---------------------- SSHD End -------------------------


--------------------- Disk Space Begin ------------------------

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root 227G 33G 183G 15% /
/dev/sdb1 367G 36G 313G 11% /mnt
/dev/sda5 228M 24M 193M 11% /boot


---------------------- Disk Space End -------------------------


###################### LogWatch End #########################

As you can see the report is well structured and provides relevant information about SMTP/POP services, Web statistics and access security. It even reports that someone is proving the server and that a known vulnerability has been tested on my server... now I can take actions like blocking that IP address from all access to the server using iptables.

From looking at the thousand of megabytes of log files it would have been a little difficult to spot this particular security treat.