\1

Many commercial switches allow replication of traffic from one or more ports to one designated port (usually chosen by the user) for monitoring and analysis purposes. Some models offer the option to choose whether to replicate only incoming or outgoing traffic (or both, of course).
Typical uses cases for this are traffic analysis systems like IDS/IPS, but it can also be used for troubleshooting.

This feature goes by many names, among which are "SPAN", "port mirroring", "port monitoring", "monitor mode", "roving" and surely others. Although the actual setup procedures vary from vendor to vendor (or even from model to model), what they do in the end is the same.
There can be differences, however, in the way tagged (ie VLAN) packet are mirrored; in some cases, VLAN tags are stripped from the mirrored copy.

Since Linux implements at least two types of bridging (nowadays used mostly to create virtual networks t/o connect virtual machines), one may wonder whether port mirroring is possible. The answer is yes, although the procedures may be a bit tricky. So let's see how to set up port mirroring under Linux with the two prevailing bridging implementations (Openvswitch and in-kernel bridging), plus another kludge at the end.

Openvswitch

Let's start with Openvswitch, the (by now not so) new, multiplatform, all-singing, all-dancing bridge implementation.

Extremely simplified, openvswitch uses a kernel module to manage the data path (ie, the actual forwarding of frames), and keeps everything else in user space. A daemon (ovs-vswitchd) manages the switch operations (which however can manage multiple bridges, so only one daemon needs to run), and another daemon (ovs-ovsdb) manages the database which contains the various tables that make up the configuration(s) for all the bridges managed by ovs-vswitchd.

Each of these two functions is driven by a corresponding protocol: OpenFlow for the management of flows and data paths (not mandatory), and OVSDB for the management of the switch itself (to add/remove ports, interfaces, bridges etc. and to removal and configuration in general).

In fact, a basic installation of openvswitch runs a local OVSDB daemon, and all the various ovs-vsctl management commands (including those shown below) connect to this local OVSDB instance via a UNIX socket, asking it to carry out the tasks.

So we have our bridge ovsbr0, with three VMs connected respectively to vnet0, vnet1 and vnet2 (of course, everything remains perfectly valid and applicable if we have real physical interfaces instead).

# ovs-vsctl show
...
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "vnet2"
            Interface "vnet2"
        Port "vnet1"
            Interface "vnet1"
        Port "vnet0"
            Interface "vnet0"
...
# ovs-vsctl list bridge ovsbr0
_uuid               : 0141452d-efc1-47f8-a3b4-24f0c2bc1c36
controller          : []
datapath_id         : "00002e454101f847"
datapath_type       : ""
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : [8a547c29-a171-4412-b7ed-b2a1b88815de]
name                : "ovsbr0"
netflow             : []
other_config        : {}
ports               : [1d1da575-73ac-4bac-8e81-1042da415103, a8333e72-cb12-4777-bf55-e339ff41ece1, ccd87251-f61f-47ff-84f3-9e8864e6c2d8, f66298f8-02e8-48cc-a2c8-92181bea2c56]
protocols           : []
sflow               : []
status              : {}
stp_enable          : false

A thing to note (besides the awkward command names, that is) is that in openvswitch, absolutely everything that can be referenced has an UUID (this is by design).
In this case, we see that the switch has three ports (plus the "internal" port that is created by default), whose UUIDs are as shown in the ports field (which is a list of values).
(Each port, in turn, may be and usually is composed of one or more interfaces, which are also objects and have their own UUIDs, but that's not relevant here).

Just to get an idea, to get the actual UUIDs of our ports we can use this command:

# for p in vnet{0..2}; do echo "$p: $(ovs-vsctl get port "$p" _uuid)"; done
vnet0: f66298f8-02e8-48cc-a2c8-92181bea2c56
vnet1: ccd87251-f61f-47ff-84f3-9e8864e6c2d8
vnet2: a8333e72-cb12-4777-bf55-e339ff41ece1

To do mirroring with openvswitch, the first thing to do is to create and add a mirror (doh!) to the bridge.

# ovs-vsctl -- --id=@m create mirror name=mymirror -- add bridge ovsbr0 mirrors @m
cd94ea72-bb7f-4a26-816f-983a085a4bfd

The syntax may look a bit awkward, but it's not complicated (and it's well explained in the ovs-vsctl man page). We're running two commands at once, each command is introduced by --. The first comand creates a mirror named mymirror and, thanks to the --id=@m part, saves its UUID in the "variable" @m, which remains available for later commands. And we use it indeed in the second command, which associates the newly-created mirror mymirror with the bridge ovsbr0.

As said, everything has an UUID, and mirrors are no exceptions: the UUID of the new mirror is output as a result of the (successful) command. Let's check:

# ovs-vsctl list bridge ovsbr0
_uuid               : 0141452d-efc1-47f8-a3b4-24f0c2bc1c36
controller          : []
datapath_id         : "00002e454101f847"
datapath_type       : ""
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : [cd94ea72-bb7f-4a26-816f-983a085a4bfd]
name                : "ovsbr0"
netflow             : []
other_config        : {}
ports               : [1d1da575-73ac-4bac-8e81-1042da415103, a8333e72-cb12-4777-bf55-e339ff41ece1, ccd87251-f61f-47ff-84f3-9e8864e6c2d8, f66298f8-02e8-48cc-a2c8-92181bea2c56]
protocols           : []
sflow               : []
status              : {}
stp_enable          : false

So everything as before, but now our bridge has the mirror (since it's a list, as shown by the fact that it's in square brackets, there can be more than one).

Now that we have our mirror created and in the bridge, we should configure its source ports and destination ports. We want to mirror all traffic going in/out port vnet0, and we want to send it to bridge port vnet2 (where presumably we have a traffing monitoring application).

We must be careful with the terminology here. A mirror has a set of "source" and "destination" ports, but those refer only to origin ports, that is, those whose traffic we want to mirror. If a port is included in the source port set (select_src_port in openvswitch term), its outgoing traffic will be mirrored; if it's included in the destination port set (select_dst_port), its incoming traffic will be mirrored. So if we want to mirror both incoming and outgoing traffic for vnet0, we must include it in both sets:

# f66298f8-02e8-48cc-a2c8-92181bea2c56 is the UUID of vnet0
# ovs-vsctl set mirror mymirror select_src_port=f66298f8-02e8-48cc-a2c8-92181bea2c56 select_dst_port=f66298f8-02e8-48cc-a2c8-92181bea2c56
# ovs-vsctl list mirror mymirror
_uuid               : cd94ea72-bb7f-4a26-816f-983a085a4bfd
external_ids        : {}
name                : mymirror
output_port         : []
output_vlan         : []
select_all          : false
select_dst_port     : [f66298f8-02e8-48cc-a2c8-92181bea2c56]
select_src_port     : [f66298f8-02e8-48cc-a2c8-92181bea2c56]
select_vlan         : []
statistics          : {}

Thanks to the previously introduced --id=@name feature, we could have done the same thing without having to specify the actual UUID of vnet0:

# ovs-vsctl -- --id=@vnet0 get port vnet0 -- set mirror mymirror select_src_port=@vnet0 select_dst_port=@vnet0

In general, this syntax is both clearer and easier, so we're going to use it for the remaining steps.

If we wanted to mirror both vnet0 and vnet1 in both directions, we would do:

# ovs-vsctl \
  -- --id=@vnet0 get port vnet0 \
  -- --id=@vnet1 get port vnet1 \
  -- set mirror mymirror 'select_src_port=[@vnet0,@vnet1]' 'select_dst_port=[@vnet0,@vnet1]'

So the trick is to populate select_src_port and select_dst_port with the (list(s) of) UUIDs of the ports that we're interested in.

So far we've told openvswitch which port(s) we want to mirror, but we haven't said yet to which port we want to send this mirrored traffic. That is the purpose of the output_port attribute, which again is the UUID of the port which will receive the mirrored traffic. In our case, we know that this port is vnet2, so here's how we add it:

# ovs-vsctl -- --id=@vnet2 get port vnet2 -- set mirror mymirror output-port=@vnet2
# ovs-vsctl list mirror mymirror
_uuid               : cd94ea72-bb7f-4a26-816f-983a085a4bfd
external_ids        : {}
name                : mymirror
output_port         : a8333e72-cb12-4777-bf55-e339ff41ece1
output_vlan         : []
select_all          : false
select_dst_port     : [f66298f8-02e8-48cc-a2c8-92181bea2c56]
select_src_port     : [f66298f8-02e8-48cc-a2c8-92181bea2c56]
select_vlan         : []
statistics          : {}

So if we now go to our VM connected to vnet2 we're going to see the mirrored traffic from vnet0. Try it and see.

Now that we have seen the step-by-step procedure, it should not come as a surprise that we could also have done all the above in a single command (reformatted for clarity):

# ovs-vsctl \
  -- --id=@m create mirror name=mymirror \
  -- add bridge ovsbr0 mirrors @m \
  -- --id=@vnet0 get port vnet0 \
  -- set mirror mymirror select_src_port=@vnet0 select_dst_port=@vnet0 \
  -- --id=@vnet2 get port vnet2 \
  -- set mirror mymirror output-port=@vnet2
cd94ea72-bb7f-4a26-816f-983a085a4bfd

A quick and dirty way to mirror all traffic passing thrugh the bridge to a given port is to use the select_all property of the mirror:

# ovs-vsctl -- --id=@vnet2 get port vnet2 -- set mirror mymirror select_all=true output-port=@vnet2
# ovs-vsctl list mirror mymirror
_uuid               : cd94ea72-bb7f-4a26-816f-983a085a4bfd
external_ids        : {}
name                : mymirror
output_port         : a8333e72-cb12-4777-bf55-e339ff41ece1
output_vlan         : []
select_all          : true
select_dst_port     : []
select_src_port     : []
select_vlan         : []
statistics          : {tx_bytes=216769, tx_packets=1400}

Openvswitch mirrors preserve VLAN tags, so the traffic is received untouched.

To remove a specific mirror, the following command can be used:

# ovs-vsctl -- --id=@m get mirror mymirror -- remove bridge ovsbr0 mirrors @m

To remove all existing mirrors from a bridge:

# ovs-vsctl clear bridge ovsbr0 mirrors

Traditional bridging

Before Openvswitch came about, Linux had had (and of course still has) in-kernel bridging since about forever.
This is a much simpler yet functional bridge implementation in the Linux kernel, which provides basic functionality like STP but not much more. In particular, there is no native port mirroring functionality.
But fear not: Linux has a powerful tool which, among a lot of other things, can also mirror traffic. We're talking of the traffic control subsystem (tc for short), which can do all sorts of magic things.
Since it's a generic framework, its capabilities (including mirroring) are not limited to bridges; this means that we can mirror traffic for any interface(s) and send it to any other(s), regardless of whether they are phisical, virtual, part of a bridge or not, etc.

Indeed, for this example we're going to mirror the incoming/outgoing traffic of the interface bond0 and have it copied to the dummy interface dummy0 (very useful for testing). Replace with vnetx/vifx.y/whatever as needed. It works just the same.

First a very brief and simplified recap, since tc is very akin to a black art. Every interface, in Linux, has a so-called queuing discipline (qdisc), which basically defines the criteria that are used to send packets out the interface. This is for outgoing packets; it is also possible, although not usually done, to set a qdisc for incoming traffic, although its usefulness is somewhat limited (but it is definitely used for mirroring).
These qdisc are usually referred to as the "root qdisc" (for outgoing traffic) and "ingress qdisc" (for incoming traffic).
So the idea is: to mirror the traffic for an interface, we configure the relevant qdisc (root and/or ingress) to mirror packets before doing anything else.

To do this, we need to attach a classifier (filter in tc speak) to the relevant qdisc. Simply put, a filter tries to match packets according to some criteria and, if the match succeeds, performs certain actions on them.

Let's start with the code to mirror incoming traffic for an interface, which is simpler. The first thing to do is to establish an ingress qdisc for the interface, as there's none by default:

# tc qdisc add dev bond0 ingress

This creates an ingress qdisc for bond0 and gives it the ffff: identifier (it's always ffff:, for any interface, so no surprises):

# tc qdisc show dev bond0
qdisc ingress ffff: parent ffff:fff1 ----------------

Now, as said, we attach a filter to it. This filter simply matches all packets, and mirrors them to dummy0. A filter is attached to a qdisc, so it must have a reference to the parent. Here's the syntax to create the filter:

# tc filter add dev bond0 parent ffff: \
    protocol all \
    u32 match u8 0 0 \
    action mirred egress mirror dev dummy0

The syntax is arcane (and, in this case, not really immediately understandable), but there are basically 3 parts. Let's break it down. The first part is the filter creation linked to the parent qdisc for interface bond0:

tc filter add dev bond0 parent ffff:

Then come the matching rules; first, we say that the match should be attempted on any protocol, since we want all the traffic:

protocol all

This is not yet part of the actual filter; it's just part of the syntax that tc needs to know which packets it should attempt to apply actual matching rules to (ok, it is effectively a filter, but not in the tc sense).
Then we give the actual filter rule:

u32 match u8 0 0

This is the syntax used to tell the u32 filter that, of the packets it's seeing (that is, all of them), all should be matched. "u32" informs the parser that a u32 match follows, and the actual matching happens in the "u8 0 0" part, which, in simple language, returns true if the first byte of the packet (u8), ANDed with 0, gives 0. Some basic knowledge of bitwise operations tells us that X AND 0 == 0 for any X, so the match is always true.

Finally, the third part of the command specifies the action that is to be executed on matching packets (again, all of them):

action mirred egress mirror dev dummy0

Here we use the mirred action, which basically has two modes of operation: mirror (which is what we want here) to, er, mirror the packet, and redirect, to, uhm, redirect it. Both do their job using the device specified in the "dev" argument. As for the "egress" part, that's the only supported mode as of this writing.

If we wanted to mirror to multiple devices, all we would have to do is to specify multiple actions:

action mirred egress mirror dev dummy0 \
action mirred egress mirror dev dummy1 ...

So if you've made it so far, you'll be happy to know that applying these rules for outgoing traffic is almost the same, just a bit more complicated. The thing is, unlike the ingress case, interfaces normally do have an egress (outgoing) qdisc, but we can't attach filters directly to it since it's a classless qdisc ("classless" just means that it can't have "child" classes and filters). So the first thing to do is add a classful egress qdisc; once we've done that, the filter is attached in the same way as for the ingress qdisc.
As a side note, the mq qdisc found in wireless interfaces, despite claiming to be classful, doesn't seem to support direct filter attachment.

If we add a classful qdisc, we should decide which one to use, since there are a few of them. The most common ones are PRIO, CBQ and HTB. Of these, the simplest is PRIO, which is what we're going to use for our example. So without further ado, let's add our classful egress qdisc to our interface:

# tc qdisc add dev bond0 handle 1: root prio

We choose to give it the handle 1:; we could as well have used 100: or 42:, it doesn't matter as long as we use the same number when attaching the filter.
Once we have a classful qdisc to play, we can finally attach the filter to it, exactly in the same way as we did for the ingress qdisc:

# tc filter add dev bond0 parent 1: \
    protocol all \
    u32 match u8 0 0 \
    action mirred egress mirror dev dummy0

Now, let's bring dummy0 up and check:

# ip link set dummy0 up
# tcpdump -e -v -n -i dummy0
tcpdump: WARNING: dummy0: no IPv4 address assigned
tcpdump: listening on dummy0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:56:41.237966 00:13:72:af:11:23 > 00:16:3e:fd:aa:67, ethertype IPv4 (0x0800), length 153: (tos 0x0, ttl 64, id 57195, offset 0, flags [DF], proto TCP (6), length 139)
    192.168.1.3.17569 > 192.168.1.232.514: Flags [P.], cksum 0x84b9 (correct), seq 3603440679:3603440766, ack 1213686729, win 229, options [nop,nop,TS val 1217617195 ecr 69837571], length 87
18:56:41.238131 00:16:3e:fd:aa:67 > 00:13:72:af:11:23, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 51990, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.232.514 > 192.168.1.3.17569: Flags [.], cksum 0x9889 (correct), ack 87, win 1307, options [nop,nop,TS val 69844202 ecr 1217617195], length 0
...
18:57:06.687832 00:26:b9:72:16:99 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 14, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.7.1.1 is-at 00:26:b9:72:16:99, length 46

As can be seen above, VLAN tags are copied.

So to sum it up, here's how to enable bidirectional mirroring from bond0 to dummy0;

sif=bond0
dif=dummy0

# ingress
tc qdisc add dev "$sif" ingress
tc filter add dev "$sif" parent ffff: \
          protocol all \
          u32 match u8 0 0 \
          action mirred egress mirror dev "$dif"

# egress
tc qdisc add dev "$sif" handle 1: root prio
tc filter add dev "$sif" parent 1: \
          protocol all \
          u32 match u8 0 0 \
          action mirred egress mirror dev "$dif"

Of course, to mirror traffic for multiple source interfaces, the above (all or only half of it, depending on whether we want traffic in both or only one direction) should be repeated for each of them.

To remove the mirroring, it's enough to delete the root and ingress qdiscs from all the involved source interfaces (the default root qdisc will be restored automatically):

tc qdisc del dev bond0 ingress
tc qdisc del dev bond0 root

Daemonlogger

So, just for the sake of it, let's see another method to mirror traffic under Linux.

There's a nice utility called daemonlogger, which, according to its description, "is able to log packets to file or mirror to another interface", which sounds just like what we are looking for. Debian has it in its standard repositories.

A quick read of the man page shows that we can use it as follows:

# daemonlogger -i bond0 -o dummy0
[-] Interface set to bond0
[-] Log filename set to "daemonlogger.pcap"
[-] Tap output interface set to dummy0[-] Pidfile configured to "daemonlogger.pid"
[-] Pidpath configured to "/var/run"
[-] Rollover size set to 18446744071562067968 bytes
[-] Rollover time configured for 0 seconds
[-] Pruning behavior set to oldest IN DIRECTORY

-*> DaemonLogger <*-
Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved

sniffing on interface bond0

At this point, tcpdump on dummy0 gives us all the traffic of bond0. Admittedly, less sophisticated than both Openvswitch and tc, but definitely much more "quick and dirty". It's also worth mentioning that it supports BPF filters just like tcpdump, so traffic can be filtered out before mirroring.
Nevertheless, a word of caution; the README file says, at the end:

This code is largely untested and probably completely shoddy.