Monitoring network bandwidth usage

During the development of my monitoring tool, I've faced a little challenge - find a crossplatform (Linux/FreeBSD) way to monitor network bandwidth usage. Surprisingly, there is no simple way to do it. In this post I'll reveal my research and the solution I've found and implemented.


Linux

One of the ways to do it is using ifconfig output:

$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:23:54:15:54:96
          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::223:54ff:fe15:5496/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:23242 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22369 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:24061454 (24.0 MB)  TX bytes:3265008 (3.2 MB)
          Interrupt:30 Base address:0xe000

We can ask for RX (received) and TX (transmitted) bytes each second, and the differences give us the required speeds. This tool use /proc/net/dev data for showing the statistic.


FreeBSD

This OS have the standard tool - systat.

% systat -if 1

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average                                                       
                     
      Interface           Traffic               Peak                Total
           age0  in      0,000 KB/s        107,444 KB/s            2,230 MB
                 out     0,000 KB/s         10,668 KB/s          372,108 KB


FreeBSD doesn't provide any analogue of /proc/net/dev data, and uses internal structures for gathering the statistic.


Universal way

But there is a way which works on all the platforms: libpcap. The Packet Capture library provides a high level interface to packet capture systems. All packets on the network, even those destined for other hosts, are accessible through this mechanism.

Unfortunately, direct use of libpcap requires root privileges, so some daemon/service have to be used for gathering the statistic. I've stopped on pmacct project, and will explain further how to set up and use it.

pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; its main features are:

• Runs on Linux, BSDs, Solaris and embedded systems
• Support for both IPv4 and IPv6
• Collects data through libpcap, Netlink/ULOG, NetFlow v1/v5/v7/v8/v9 and sFlow v2/v4/v5
• Saves data to a number of backends including memory tables, MySQL, PostgreSQL and SQLite
• Exports data to remote collectors through NetFlow v5/v9 and sFlow v5
• Replicates incoming NetFlow and sFlow packets to remote collectors
• Flexible architecture to tag, filter, redirect, aggregate and split captured data
• Pluggable architecture for easy integration of new capturing environments and data backends
• Careful SQL support: data pre-processing, triggers, dynamic table naming
• It's free, open-source, developed and supported with passion and open mind

Pmacct installation

Ubuntu:

$ sudo apt-get install pmacct

FreeBSD:

$ cd /usr/ports/net-mgmt/pmacct

$ sudo make install clean

After the installation pmacctd daemon and pmacct client tool will be available.


Pmacctd daemon configuring

We have to configure two files - the daemon configuration file and networks file describing the networks which should be monitored.


Daemon configuration file (/etc/pmacct/pmacctd.conf in Ubuntu, /usr/local/etc/pmacct/pmacctd.conf in FreeBSD):

! pmacctd configuration
daemonize: true
pidfile: /var/run/pmacctd.pid
syslog: daemon
! on this interface (age0 or similar for FreeBSD)
interface: eth0

! interested in in and outbound traffic
networks_file: /usr/local/etc/networks.def

aggregate[in]: dst_net
aggregate[out]: src_net
! storage methods
plugin_buffer_size: 2048
plugin_pipe_size: 2048000

plugins: memory[in], memory[out]
imt_path[in]: /tmp/pmacct_in.pipe
imt_path[out]: /tmp/pmacct_out.pipe

Here we set up to listen eth0 interface, aggregate all the network information, and store the information in the memory database. Pmacct client will connect the daemon via pipes declared by imt_path parameters.


Networks file (/usr/local/etc/networks.def):

192.168.1.0/24

My IP-address (192.168.1.X) is in this network, so I listen all the network.


Now you can test the configuration by starting the daemon with it:

$ pmacctd -f pmacctd.conf

If everything is fine, it will start without any complains. For debugging purposes, you can start it without daemonizing and with debug: true option.


Using the pmacct

For getting statistic use the pmacct client:

$ pmacct -s -p /tmp/pmacct_in.pipe
DST_IP PACKETS BYTES
:: 1 56
192.168.1.0 26707 24994698
0.0.0.0 22672 3691429

For a total of: 3 entries

Use the different pipes for getting inbound/outbound statistic. To get the bytes only use clarified requests like:

$ pmacct -N 192.168.1.0 -c dst_net -p /tmp/pmacct_in.pipe

25308311

$ pmacct -N 192.168.1.0 -c src_net -p /tmp/pmacct_out.pipe

4368614

pmacct has a useful option "-r" (reset counters), so if you use it with this option every second you'll have network speed (bytes per second), as I do in my monitoring tool:

FreeBSD daemon

FreeBSD requires one more step - make pmacctd starting during the loading process. For this I've put the pmacctd shell script in the /etc/rc.d:

#!/bin/sh

# PROVIDE: pmacctd
# REQUIRE: DAEMON
# BEFORE:  LOGIN
# KEYWORD: nojail shutdown

. /etc/rc.subr

name="pmacctd"
rcvar=`set_rcvar`
command="/usr/local/sbin/${name}"
command_args="-f /usr/local/etc/pmacctd.conf"
pidfile="/var/run/${name}.pid"

load_rc_config $name
run_rc_command "$1"

And set up /etc/rc.conf to start the daemon during the system loading:

pmacctd_enable="YES"

That's all. Hope this help you in monitoring and auditing your systems.