Scenario: KVM virtualization host running several bridged guests. The host has a bridge interface br0 that starts out containing only eth0, and other interfaces are dynamically added and removed from the bridge as guests are started and stopped.
The problem is, the host seems to randomly suffer some loss of connectivity (from a few to 30-40 seconds) when some guest is started or stopped. Initially one might think of something related to STP, but it turns out that it is disabled (and even then, ports appearing or disappearing should not affect existing ports).
What happens here is that, when a new guest is started, a tap interface is created and enslaved to the bridge (the tap interface is usually connected to the guest's own ethernet interface). This interface has a name like vnet5, for example. Minor but important detail, this tap interface has a more or less random MAC address. (Note that we're talking of the host-side tap interface, not the guest-side virtual interface, whose MAC address is configurable in the guest's configuration file or on the command line). It is theoretically possible to set the MAC address of this host-side tap interface, but it's quite involved and basically nobody does that (and also virt-manager has no provision to do that).
Now, by default bridge interfaces in Linux use, for their MAC address, the lowest MAC address among the enslaved interfaces. So if the newly created interface has a lower MAC, the bridge changes its MAC address and uses that of the new interface. When the guest is stopped, the tap interface is removed from the bridge and destroyed, at which point the bridge's MAC address has to change again. Depending on when and how often guests are started/stopped, the host's bridge interface may be changing its MAC address quite often.
Of course, this may (and does) cause connectivity problems for traffic to/from the host (fortunately, traffic to/from the guests is not affected), because neighboring computers and network devices suddenly have to update their ARP caches with the new MAC address before they can talk to the host. Depending on the traffic patterns, this may take anything from a few seconds to several tens of seconds.
Fortunately, there is a way to ensure that the bridge's MAC address is fixed and never changes, thus entirely avoiding the problem. Thanks to this thread and this thread by people with similar problems, I found out that if the bridge's MAC address is forced to a specific value, the bridge "remembers" that and makes the address permanent. But there's a caveat: the address must belong to one of the devices enslaved to the bridge. In our example (and probably also in the most common case), the ideal candidate is obviously eth0, which is permanently enslaved.
There are two ways to set the MAC address of an interface. The old way is using ifconfig:
# 00:0a:e7:2c:44:2a is eth0's MAC address # ifconfig br0 hw ether 00:0a:e7:2c:44:2a
The newer and better way is with iproute2:
# ip link set br0 address 00:0a:e7:2c:44:2a
And also, it is desirable to set the permanent MAC address at boot time, so for example under Debian/Ubuntu the following line can be used in the configuration file /etc/network/interfaces:
auto br0 iface br0 inet static address 10.4.0.3 netmask 255.255.255.0 gateway 10.4.0.254 bridge_ports eth0 bridge_stp off post-up ip link set br0 address 00:0a:e7:2c:44:2a
Or you can use:
auto br1
iface br1 inet static
...
bridge_hw 00:22:aa:44:bb:cc
...
source: http://wiki.debian.org/BridgeNetworkConnections
it is very helpful and get it fixed.
Thanks
Thanks for the useful post. This scenario also applies to OpenVZ containers when configured with veth devices. An important thing to note (in my experience at least) is that whenever you make any changes to the bridge using brctl (deleting/adding interfaces) the MAC address selection happens again and will revert your previous "ifconfig" or "ip link set" commands. With OpenVZ, I get around this by adding "$ifconfig $VZHOSTBR hw ether $($ifconfig eth0 | awk '{print $5; exit}')" to the vznetaddbr script (called when starting a container if "CONFIG_CUSTOMIZED=yes" in $VEID.conf) and vps.umount (called when stopping a container).
Your post and the post at http://www.linuxweblog.com/blogs/sandip/20080814/bridge-networking-on-openvz-containers-using-veth-devices were both very helpful in helping me to find and solve a pretty tricky problem today.
-Paul
I'm wondering if this can be configured without specifying the mac address and instead of using something like bridge_hw `ip link show eth1 | awk '/ether/ {print $2}'` in the /etc/network/interfaces -file. It probably fails now, because eth1 is not yet configured when the command is executed
Hey, folks, this doesn't work for me, Look what I got :
auto br1
iface br1 inet static
address 192.168.2.105
network 192.168.2.2
netmask 255.255.255.0
broadcast 192.168.2.255
gateway 192.168.2.1
bridge_ports bond0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
# post-up ip link set br1 address 00:50:fc:f8:b1:2a
Your line is commented and it works, if i get out the comment it just stops working... ! There is a bond-bridge setup of eth1 and eth0, bond-mode 1 (redundance)...so what can be wrong ?
OK, IT works, I had a typo in address ! The address mac address you give MUST be one of your enslaved ethernet cards.
This is probably for some sort of security reasons, so you can't spoof and create chaos in the network by artificially replicating the same mac. I mistyped a character to be sure of that :/
Thanks, nice "how-to".
Actually one should read "post-up ip link ..." instead of "port-up ip link ..."
Thank you again
Doh! Right, I fixed it. Thanks!
Thank you very much for these very clear and detailled explanations. This problem was annoying me for a while.
Do you see a rationale for the bridge MAC@ dance any way?
To be honest, I don't know why the default behavior is to have a dynamic MAC address for the bridge. The important thing is that there is now the option of setting a sticky address.
This is great. I've been running into this problem on a new server and couldn't figure out what was causing it. Not only did you explain why this was happening on my new server but also why I'd never seen it on the older systems I have running QEMU/KVM/libvirt, the bridged interface MAC started with 00 so the chances of a randomly generated MAC from a VNET device taking over the bridge were pretty small. Thanks!
Yes, it seems that the tendency that recent libvirt/KVM have to give host-side tap interfaces MAC addresses starting with fe:52 or so (though it looks like that could be changed with a lot of work, which noone wants to do) made the problem much more apparent.