I live in a part of the world where broadband internet subscriptions come with a monthly data cap. The datacap is relatively large (750GB/month), but streaming 4k content will get you there! So I wanted to keep taps on my data usage. My router is the ideal place to monitor this: all traffic needs to pass through there anyway. And since it runs OpenWRT, I can easily add the component I need.

Interface-level statistics

The first step is relatively easy. The Linux kernel already keeps a counter for each network interface that counts the number of bytes it has sent & received. These counters are exposed through the ifconfig command, but also through the /proc/net/dev "file".

I'm using Prometheus as my monitoring tool, so I wrote a simple shell-script to expose these numbers as prometheus metrics. I should note that I'm not very fluent in Prometheus metric naming, so I might have picked something "wrong".

NAME="wan"
STATS="$(grep pppoe /proc/net/dev )"
echo "network_bytes_total{interface=\"$NAME\",direction=\"rx\"} $(echo "$STATS" | awk '{print $2}')"
echo "network_packets_total{interface=\"$NAME\",direction=\"rx\"} $(echo "$STATS" | awk '{print $3}')"
echo "network_bytes_total{interface=\"$NAME\",direction=\"tx\"} $(echo "$STATS" | awk '{print $10}')"
echo "network_packets_total{interface=\"$NAME\",direction=\"tx\"} $(echo "$STATS" | awk '{print $11}')"

To expose these metrics to Prometheus, I converted the above snippet into a CGI-script. This sounds more complicated than it is. I just added these lines to the top, and linked in into /www/cgi-bin/metrics.

echo "Content-Type: text/plain; version=0.0.4"
echo ""

Host-level statistics

Adding metrics for individual hosts turned out to be a lot more complicated than I anticipated. In the good old days, one could simply use an iptables-table that matched all 254 IP-addresses of the subnet, and call it a day. But we don't live in the good old days anymore, I live in a time where I have native IPv6 connectivity!

IPv6 complicated this idea a lot. Not only is it unfeasable to create a table of all 18446744073709551616 IPv6 addresses in the subnet, IPv6 Privacy Extensions means that there is no way to combine the different addresses for a particular host... The only identifier that is usable in a Dual-Stack environment with IPv6 Privacy Extensions is the Ethernet MAC address of the host.

Once I realised I should use the MAC addresses, the rest seemed simple:

iptables -N accounting
iptables -A FORWARD -j accounting
iptables -A accounting -m mac --mac-source 00:00:5E:01:02:03 -j RETURN
iptables -A accounting -m mac --mac-destination 00:00:5E:01:02:03 -j RETURN

Only, that doesn't work... There is no --mac-destination filter. The reason is pretty simple: we are still in IP-land, so the kernel does not know yet what MAC-address it will sent the frame to. You could get around this by adding a complete ebtables setup, but that seems like the "wrong" solution to track IP-level counters.

The solution is rather complicated. And I couldn't have done it without help from this SuperUser answer. Use the connection tracking logic to carry the MAC-information over to the return packets:

# CONNMARK only works in the mangle table, so we'll do the accounting there
iptables -t mangle -N accounting_out
iptables -t mangle -N accounting_in

iptables -t mangle -A FORWARD -j CONNMARK --restore-mark  # Restore the saved mark from the CONNTRACK table into the IP-level MARK
iptables -t mangle -A FORWARD -i pppoe -j account_in      # do accounting based on the MARK
iptables -t mangle -A FORWARD -o pppoe -j account_out     # do accounting based on MAC, and set MARK
iptables -t mangle -A FORWARD -j CONNMARK --save-mark     # save IP-level MARK into CONNTRACK table

# Assign unique integer IDs to each host:
iptables -t mangle -A account_out -m mac --mac-source 00:00:5E:01:02:03 -m comment --comment "desktop" -j MARK --set-mark 1
iptables -t mangle -A account_in -m mark --mark 1 -m comment --comment "desktop" -j RETURN

iptables -t mangle -A account_out -m mac --mac-source 00:00:5E:04:05:06 -m comment --comment "desktop" -j MARK --set-mark 2
iptables -t mangle -A account_in -m mark --mark 2 -m comment --comment "desktop" -j RETURN

# repeat for ip6tables

Some things that I figured out while trying to get this to work:

• There is a CONNMARK target that allows you to directly save the mark to the conntrack-table. This would allow you to skip the --save-mark rule. This works in iptables, but does not work in ip6tables

• restoring a CONNMARK mark to the IP-layer only works in the mangle table.

Exposing these counters to prometheus isn't very difficult, but the resulting script is rather long and ugly:

iptables -t mangle -vnL account_in --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="4",direction="in",host="\3"} \1\nhost_bytes_total{ipv="4",direction="in",host="\3"} \2%p'
iptables -t mangle -vnL account_out --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="4",direction="out",host="\3"} \1\nhost_bytes_total{ipv="4",direction="out",host="\3"} \2%p'
ip6tables -t mangle -vnL account_in --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="6",direction="in",host="\3"} \1\nhost_bytes_total{ipv="6",direction="in",host="\3"} \2%p'
ip6tables -t mangle -vnL account_out --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="6",direction="out",host="\3"} \1\nhost_bytes_total{ipv="6",direction="out",host="\3"} \2%p'