\1

The munin plugin described in this article can be downloaded here: traffic_accounting. Remember that it must be made executable once it's copied in place.

There seems to be no easy way (if at all) to glean detailed traffic information from the /proc virtual filesystem. General stats can be found there, like number of bytes or packets, but not much information about, for example, traffic from/to specific ports, or specific IP addresses, or specific ICMP types.

The usual trick to get that information is to use iptables and write non-terminating rules that match the traffic of interest; since iptables will keep counters for matching packets and bytes for each rule (if accounting is enabled, which it usually is), that provides a "natural" way to get the information we want. Anything that iptables can match, can be put into a (non-terminating) rule and counters obtained for it.

There's a minor problem, however. There's no official, standard way to access the counters in a defined format (that I know of, at least). While there are a couple of commands that output those counters, their output format may change in the future, although it seems to have been fairly stable so far. We have to live with that, but since we're lazy, and want to avoid parsing every possible output format, we rely on the user to add comments in a specific format to the accounting rules, and then parse only those comments. This is an acceptable tradeoff between implementation complexity and usage simplicity.

What we will do here is to write a munin plugin to collect and graph arbitrary traffic stats. There are other plugins floating around based on the same ideas, but as far as I can see, none as general as this.

General concepts

The basic idea is to create a special chain, that will only contain non-terminating rules, whose only purpose will be to collect counters. This will be done for IPv4 and IPv6 separately (since those are two different rulesets in netfilter terms, managed with two different commands). The suggested setup is to add a rule at the beginning of the INPUT and OUTPUT chains to jump to the management chain. If the machine is forwarding traffic, it may be added to the FORWARD chain (exclusively or in addition to INPUT and OUTPUT, depending on the exact circumstances). This allows a minimal impact on any existing firewall setup without having to mess around with the "real" main rules. Example:

# create accounting chain for IPv4
# iptables -N ACCT4
# insert rules in the INPUT and OUTPUT chains to jump to the accounting chain
# iptables -I INPUT 1 -j ACCT4
# iptables -I OUTPUT 1 -j ACCT4
# same as above, for IPv6
# ip6tables -N ACCT6
# ip6tables -I INPUT 1 -j ACCT6
# ip6tables -I OUTPUT 1 -j ACCT6
# do the same for FORWARD if/as needed

At this stage, non-terminating rules can be added to the ACCT4 and ACCT6 chains at will, without affecting the rest of the firewall.

To preserve counters across reboots, it's recommended that the firewall rules are saved on file using iptables-save/ip6tables-save upon shutdown, and restored using iptables-restore/ip6tables-restore at boot. This is what the majority of the distributions do anyway, so there should be no problem.

Requirements

For this project, we have some requirements:

• It should be possible to choose whether to collect stats about traffic in terms of packets, bytes, or bits;
• It should be possible to have "regular" graphs (ie with a single quantity graphed), or munin's "sent/received" style traffic (as used for example for network interfaces), with positive values representing outgoing traffic, and negative values representing incoming traffic;
• Anything that iptables can match, it should be possible to graph;
• It should be possible to "group" rules so they are totaled and graphed as a single entity;
• The plugin should provide multigraph capabilities, but if the munin server does not support multigraph (old versions), it should be possible to use the plugin as a normal wildcard plugin by creating appropriately named symlinks to it. Thus the plugin defaults to multigraph if the server supports it, but the user can override this via environment variables.

Accounting rules

As said, to avoid parsing the full output of iptables, we rely on the user to add suitable comments to each rule, using iptables' -m comment option. These comments can be easily detected in the output of iptables-save or ip6tables-save because commented rules will contain something like

... -m comment --comment "comment here"

The main purposes of the comments are:

• Identify rules that are relevant for graphing. Rules with no comments are discarded;
• Assign a "tag" to each rule. The tag can be just anything, and identifies the entity to be graphed (which need not correspond to a single rule only, as we'll see: rules with the same tag are aggregated);
• Provide a description to use for the graph captions.

To achieve these goals, the comments must have a predefined syntax, as follows:

=tag= Free descriptive text

Here, "tag" is the tag, and "Free descriptive text" is the comment that will be used for the graph (if you think your tag is descriptive enough for you, you can leave out the descriptive text and the plugin will use the tag as description). Multiple rules can have the same tag, as follows:

• All the rules with the same tag in the same chain are aggregated, their stats summed, and graphed as a single value;
• Using tag_up and tag_down will result in a sent/received style graph, with data matched by the rule tagged _up graphed in the upper half, and data matched by the rule tagged _down graphed in the lower half. Of course the plugin knows nothing about what the rules do, so this could also be abused to put arbitrary rules in the upper and lower half; it just seems more logical to pair rules that describe the two directions of the same traffic.

The two methods described above can be used together, so it's possible to have multiple tag_up rules (will be summed and graphed as a single value in the upper half) and tag_down rules (will be summed and graphed as a single value in the lower half).

An example will make the concepts clear.

Example

Suppose we have a server whose IPv4 address is 192.168.24.1, and its IPv6 address is 2001:db8:f00d::1. This server runs an HTTP daemon on TCP ports 80 and 443 (bound to the IPv4 and the IPv6 addresses), and it also runs an SSH server on port 22 (bound to the IPv4 address only). Access to SSH is restricted and is allowed from IP addresses in the range 10.2.3.0/24 only. What we want is to get a graph of HTTP traffic (in the sent/received style), and a regular graph of SSH traffic received from unauthorized addresses (will be basically just SYN packets for the most part, but for the example it is fine). As a final twist, we want HTTP stats for IPv4 to be aggregated (port 80 and 443 together), but stats for IPv6 separated (separate graphs for port 80 and port 443).

Here are the rules to be added to the accounting chains (line numbers added for readability):

 1 # iptables -A ACCT4 ! -s 10.2.3.0/24 -p tcp --dport 22 -m comment --comment "=unauthssh= Unauthorized SSH traffic received"
 2 # iptables -A ACCT4 -d 192.168.24.1 -p tcp --dport 80 -m comment --comment "=web4_down= Combined web traffic to/from local 80/443"
 3 # iptables -A ACCT4 -s 192.168.24.1 -p tcp --sport 80 -m comment --comment "=web4_up= dummy"
 4 # iptables -A ACCT4 -d 192.168.24.1 -p tcp --dport 443 -m comment --comment "=web4_down= dummy"
 5 # iptables -A ACCT4 -s 192.168.24.1 -p tcp --sport 443 -m comment --comment "=web4_up= dummy"
 6 # ip6tables -A ACCT6 -d 2001:db8:f00d::1 -p tcp --dport 80 -m comment --comment "=web6_down= Web traffic to/from local port 80"
 7 # ip6tables -A ACCT6 -s 2001:db8:f00d::1 -p tcp --sport 80 -m comment --comment "=web6_up= dummy"
 8 # ip6tables -A ACCT6 -d 2001:db8:f00d::1 -p tcp --dport 443 -m comment --comment "=web6ssl_down= Web traffic to/from local port 443"
 9 # ip6tables -A ACCT6 -s 2001:db8:f00d::1 -p tcp --sport 443 -m comment --comment "=web6ssl_up= dummy"

Rule 1 catches unauthorized SSH attempts over IPv4. The rule is tagged as unauthssh (it could be anything, in fact, although probably a descriptive tag is more readable), and the description that will appear in the graph is "Unauthorized SSH traffic received". The rule is the only one with that tag, so it will be graphed on its own graph.

Rules 2 and 3 catch IPv4 TCP traffic to and from local port 80 respectively. They're tagged web4_down and web4_up, so they will be in the same graph (sent/received style).
Rules 4 and 5 catch IPv4 TCP traffic to and from local port 443 respectively. Since we want this traffic to be aggregated with that of port 80, they're also tagged web4_down and web4_up. So overall, rules 2 to 5 will end up in the same graph; the upper half will graph the sum of rules 3 and 5 (outgoing traffic); the lower half will graph the sum of rules 2 and 4 (incoming traffic). This is to demonstrate summing; in practice, one would probably want to see the two graphs separately, as we'll do for IPv6 (see below). Regarding graph descriptions, the plugin takes the one it finds when it first encounter a new tag, so in this case the description from rule 2 will be used: "Combined web traffic to/from local 80/443". The descriptions in rules 3, 4 and 5 can be anything; here it's "dummy", but we could have left it empty. The plugin just ignores them, since they have the same tag as rule 2.

Rules 6 and 7 match IPv6 TCP traffic to/from local port 80 (similar to what rules 2 and 3 do for IPv4). They're tagged web6_down and web6_up, so they (and only they) will be in their own sent/received graph, whose description will be "Web traffic to/from local port 80".

Finally, rules 8 and 9 match IPv6 TCP traffic to/from local port 443 (like rules 4 and 5 did for IPv4). They're tagged web6ssl_down and web6ssl_up, which is a new tag, so they'll be in their own sent/received graph. The description will be "Web traffic to/from local port 443".

That's it for the iptables part for this example. For real cases, one should create all the rules that are relevant to the traffic that one wants to account, using the rules for tagging explained previously, depending on the desired graph layout.

Plugin configuration

Now what's left is to configure the munin plugin, and let it do its job. As said previously, the plugin can operate in multigraph mode, or in wildcard mode whereby one or more appropriately named symlink is created to it.

If multigraph is enabled, the plugin scans the configured IPv4 and iPv6 accounting chains and processes the rules according to the tags it finds; this results in a root cumulative graph being produced, along with all the constituent individual graphs.
If multigraph is disabled, the plugin can only graph a protocol/tag combination at a time, so it expects to be invoked via an appropriately named symlink. The symlink should be as follows:

traffic_accounting_protocol_tag

In our example, without multigraph one would create the following symlinks:

# ln -s /path/to/traffic_accounting /etc/munin/plugins/traffic_accounting_ipv4_unauthssh
# ln -s /path/to/traffic_accounting /etc/munin/plugins/traffic_accounting_ipv4_web4
# ln -s /path/to/traffic_accounting /etc/munin/plugins/traffic_accounting_ipv6_web6
# ln -s /path/to/traffic_accounting /etc/munin/plugins/traffic_accounting_ipv6_web6ssl

The plugin tries to detect if the server supports multigraph; if it doesn't, multigraph is disabled. If the server support multigraph, the plugin checks for the $multipath environment variable and uses its value ("yes" or "no"); if the variable is not present, it defaults to multigraph enabled.

So a typical configuration when using multigraph could be (the values shown for environment variables are the defaults, so they can all be omitted)

[traffic_accounting]
# needs root for iptables-save
user root
# accounting chain for IPv4
env.chain4 ACCT4
# accounting chain for IPv6
env.chain6 ACCT6
# location of iptables
env.iptables /sbin/iptables
# location of ip6tables
env.ip6tables /sbin/ip6tables
# use multigraph yes|no
env.multigraph yes
# what to graph bits|bytes|packets
env.what bytes

When not using multigraph, the configuration might look something like

[traffic_accounting_*]
# needs root for iptables-save
user root
# etc. same as before

and of course as usual with munin plugins more specific values that override the more general ones can be provided by giving a more specific plugin name specification, like

[traffic_accounting_ipv4_*]
# applies to all IPv4 stuff

[traffic_accounting_ipv6_web6ssl]
# applies only to the specific IPv6 SSL stuff
#etc.

That's it. Restart munin-node, wait some time until the data is collected, and look at your traffic.