Thursday, October 08, 2009

How to create Netem distribution tables

The Netem and Dummynet emulators are widely used in networking research to achieve emulated network environments that can be used to evaluate new protocols and applications under controlled and repeateable conditions. In the case of Netem it is possible to use custom ditribution tables to better approximate the round-trip-time variations in the nework.

Unfortunately there is little information on how to generate this distribution tables based on real measured data. The only reference available is in the iproute2 source code itself and is a little confusing. After some reading and re-reading I think I finally got the magic formula and here I present it so others can benefit of my work.

First download the iproute2 source code:


git clone \
git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git


Then enter the netem subdirectory and compile the maketable and stats utilities:


cd iproute2/netem
gcc -lm -o maketable maketable.c
gcc -lm -o stats stats.c


Next you need to extract some delay statistics of the channel you want to emulate. Suppose you want to emulate the network between a server in Japan and a client PC in Chile. Then simply run a ping between both machines for some period of time. A whole day of ping data would be enough:


ping 192.168.4.111 > mydata.txt


The command above does a ping to the address 192.168.4.111 and stores the output to a file named mydata.txt. The file will contain the round-trip-times between the local machine and the machine with address 192.168.4.111 like shown below:


PING 192.168.4.111 (192.168.4.111) 56(84) bytes of data.
64 bytes from 192.168.4.111: icmp_seq=1 ttl=43 time=306 ms
64 bytes from 192.168.4.111: icmp_seq=2 ttl=43 time=273 ms
64 bytes from 192.168.4.111: icmp_seq=3 ttl=43 time=310 ms
64 bytes from 192.168.4.111: icmp_seq=4 ttl=43 time=300 ms
64 bytes from 192.168.4.111: icmp_seq=5 ttl=43 time=300 ms
64 bytes from 192.168.4.111: icmp_seq=6 ttl=43 time=304 ms
...


The more data you have the better so leave the ping running for some long period of time. Next we need to extract the round-trip-time data only:



cat mydata.txt | grep icmp_seq | cut -d'=' -f4 | cut -d' ' -f1 > myrtt.txt



The above command will extract all the round-trip-times from mydata.txt and put them in a single column in a file named myrtt.txt. Finally we can create a distribution table that Netem understands using the maketable utility:


./maketable myrtt.txt > mydist.dist


This will create a distribution table and store it in the file mydist.dist that must be copied in a special folder (/usr/lib/tc) so Netem can find it:


sudo cp mydist.dist /usr/lib/tc


Additionally we can obtain the mean and standard deviation of the round-trip-time using the stats utility:


./stats myrtt.txt
> mu = 296.636364
> sigma = 12.808378
> rho = -0.103472


This command gives use the mean (mu) the deviation (sigma) and the correlation (rho) of the round-trip-time. And at last we can now use Netem to emulate the channel's round-trip-time using the following command:



sudo tc qdisc add dev eth0 root netem delay 296.636364ms 12.808378ms distribution mydist



Note the the distribution name is the same as that of the file we copied inside the /usr/lib/tc folder but without the .dist extension. Now Netem will provide you with a channel that has similar round-trip-time characteristics as those of the real channel.