Sometimes, for instance when having a limited number of Network Interface Cards (NICs) on a system that will be used for a Linux hosted platform virtualization solution (and you’re running Fedora 13), the easiest approach to giving each of the guests “direct” access to a network is to configure the physical devices as bridges on the host.
This will permit the libvirt virtualization (management) abstraction interface to easily build “briges of bridges” that in turn let a Kernel Virtual Machine (KVM) guest get it’s own “public” (only in quotes because I happen to think the average bear would not be so silly as to put their Linux/KVM host directly onto the internet. Right???) IP address and route its traffic directly onto the ether (via the lower levels of the IP stack of the host environment).
There are, as is the case with all things Linux or UNIX, a couple of ways to skin this particular bear (sorry, that’s bad!), but the one that makes the most sense to me is to have init take care of the configuration as part of the system boot process (when the network service executes). And doing that, although in its simplest form requires access to a terminal window and a text editor on the Fedora host, is actually very simple, once you know what you’re doing. Hopefully, the following will help you learn (if you don’t already know and are only reading this because you’re looking around and are a very bored individual).
At a high level, all you have to do is edit the /etc/sysconfig/network-scripts/ifcfg-ethX file for the Ethernet (eth) device you want to bridge, specify that the device will belong to a bridge named <bridgename> and then create a /etc/sysconfig/network-scripts/ifcfg-<bridgename> file that configures the bridge (assigning it a way to obtain an IP address – probably static, routing and DNS configuration information).
Prior to configuring the system to use bridging, I had configured static IP addresses on the two physical interfaces I will be creating bridges for. Since that configuration was being obsoleted in order to use the same interfaces for guest-to-iSCSI-SAN traffic, I edited and created the above mentioned configuration files. All of the edited items are in bold. Before you edit and create these files, life gets a little easier if you:
- Back up the original ifcfg-eth[N] files to some other location than /etc/sysconfig/network-scripts/
- # ifdown eth[N]
Then, as an example, the two-to-four files you need (ifcfg-eth[N], ifcfg-eth[N+1] and ifcfg-bridge[N] and ifcfg-bridge[N+1]).
/etc/sysconfig/network-scripts/ifcfg-eth2:
# Intel Corporation 82571EB Gigabit Ethernet Controller DEVICE=eth2 BOOTPROTO=none HWADDR=00:15:17:6C:97:94 #IPADDR=X.X.Z.231 #NETMASK=255.255.255.0 #PREFIX=24 #DEFROUTE=yes ONBOOT=yes TYPE=Ethernet #DNS1=[DNS-SERVER1] #DNS2=[DNS-SERVER2] IPV6INIT=no USERCTL=no IPV4_FAILURE_FATAL=yes NAME="System eth2" BRIDGE=iscsi-bridge0 MTU=9000
/etc/sysconfig/network-scripts/ifcfg-eth3:
# Intel Corporation 82571EB Gigabit Ethernet Controller DEVICE=eth3 #IPADDR=X.Y.Z.232 #NETMASK=255.255.255.0 #PREFIX=24 #DEFROUTE=yes #IPV4_FAILURE_FATAL=yes HWADDR=00:15:17:6C:97:95 ONBOOT=yes BOOTPROTO=none TYPE=Ethernet BRIDGE=iscsi-bridge1 IPV6INIT=no USERCTL=no NAME="System eth3" MTU=9000
/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge0:
DEVICE=iscsi-bridge0 ONBOOT=yes TYPE=Bridge IPADDR=X.Y.Z.231 NETMASK=255.255.255.0 STP=off MTU=9000 DELAY=0
/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge1:
DEVICE=iscsi-bridge1 ONBOOT=yes TYPE=Bridge IPADDR=X.Y.Z.232 NETMASK=255.255.255.0 MTU=9000 STP=off DELAY=0
Although there’s a perfectly valid way of achieving the same thing through the virsh/libvirtd management interface to libvirt as well as with the Network Manager tools, my preference is to make this configuration “stick” using the old network init service. The problem(s) I see with the NetworkManager/libvirtd approach is twofold:
- Timing of NetworkManager start-up (not all that early) relative to the Open-iSCSI stack startup (early)
- Timing of libvirtd start-up (one of the last services to get called) relative to other iSCSI volumes needing to be available for the host environment.
So, for this example, disable NetworkManager as a boot service and enable the network service:
# service NetworkManager stop # chkconfig NetworkManager off # chkconfig network on # ifup iscsi-bridge0 # ifup iscsi-bridge1
And, Bob’s yer uncle (or, at least, he should be!). To verify that everything is working properly, ping an IP target that should be reachable from the bridge device interface(s) only:
# ping -I iscsi-bridge0
# ping -I iscsi-bridge1
By the way: If you use bridged interfaces, no iSCSI volumes on the host (for guests only, in other words) and have iptables enabled on the host (which you should), make sure to configure your host iptables to leave the bridged interfaces alone. For details, see the – soon to be created, I promise – post about performance tuning the Linux IPv4 environment for iSCSI-initiators on this site. Alternatively, you can grab the information pertaining to the relevant bridge sysctl.conf entries from the KVM scalability white paper Red Hat published (and I provided most of the content for in my previous career).
On my host, I have two NICs: one for our primary network (eth0/br0), one on our dedicated iSCSI network (eth2/br2) which mounts an iSCSI volume on the host. On my guests, I want to use iSCSI and so I have two NICs for each guest, one bridging br0 for all our regular traffic, and one bridging br2 where I intended to carry traffic from my guest iSCSI initiators. Unfortunately, the system comes to a half for 30 seconds or more on each iSCSI discovery or node connection on a guest. I see in your final paragraph NOT to use iSCSI volumes on the host if using iSCSI on the bridged network. Is this likely my problem? Can I still use iSCSI on multiple guests over the same bridged host NIC without issue? Why might these restrictions be in place?
It’s difficult to tell from your comment, to be honest. I don’t know the distribution or version of the Linux kernel you’re using, etc, etc. Generally speaking, doing a “#iscsiadm -m discovery -t st” should not bring the actual bridge interfaces or physical interfaces into a state that would cause some sort of hickup (temporary link reset), but that’s what it sounds like is happening to you.
I’d probably start by looking at my switch configuration. Do you have spanning tree configured on the ports – could be you’re seeing ports being disabled by the switch to mitigate switch loops, etc. “Port flapping” can often be the result of SPT recalculations (which can take up to 30+ seconds.
Also, I’d look into whether or not NetworkManager is involved (on servers I _always_ turn Network Manager off since I don’t see the point of having it active. It’s predominantly designed to be a client side tool (i.e. for laptops or systems inconsistently connected to different networks/wireless networks) and has – in my view – no purpose in a server environment with permanent & direct links.