This will permit the libvirt virtualization (management) abstraction interface to easily build “briges of bridges” that in turn let a Kernel Virtual Machine (KVM) guest get it’s own “public” (only in quotes because I happen to think the average bear would not be so silly as to put their Linux/KVM host directly onto the internet. Right???) IP address and route its traffic directly onto the ether (via the lower levels of the IP stack of the host environment).

There are, as is the case with all things Linux or UNIX, a couple of ways to skin this particular bear (sorry, that’s bad!), but the one that makes the most sense to me is to have init take care of the configuration as part of the system boot process (when the network service executes). And doing that, although in its simplest form requires access to a terminal window and a text editor on the Fedora host, is actually very simple, once you know what you’re doing. Hopefully, the following will help you learn (if you don’t already know and are only reading this because you’re looking around and are a very bored individual).

At a high level, all you have to do is edit the /etc/sysconfig/network-scripts/ifcfg-ethX file for the Ethernet (eth) device you want to bridge, specify that the device will belong to a bridge named <bridgename> and then create a /etc/sysconfig/network-scripts/ifcfg-<bridgename> file that configures the bridge (assigning it a way to obtain an IP address – probably static, routing and DNS configuration information).

Prior to configuring the system to use bridging, I had configured static IP addresses on the two physical interfaces I will be creating bridges for. Since that configuration was being obsoleted in order to use the same interfaces for guest-to-iSCSI-SAN traffic, I edited and created the above mentioned configuration files. All of the edited items are in bold. Before you edit and create these files, life gets a little easier if you:

1. Back up the original ifcfg-eth[N] files to some other location than /etc/sysconfig/network-scripts/
2. # ifdown eth[N]

Then, as an example, the two-to-four files you need (ifcfg-eth[N], ifcfg-eth[N+1] and ifcfg-bridge[N] and ifcfg-bridge[N+1]).

/etc/sysconfig/network-scripts/ifcfg-eth2:

# Intel Corporation 82571EB Gigabit Ethernet Controller
DEVICE=eth2
BOOTPROTO=none
HWADDR=00:15:17:6C:97:94
#IPADDR=X.X.Z.231
#NETMASK=255.255.255.0
#PREFIX=24
#DEFROUTE=yes
ONBOOT=yes
TYPE=Ethernet
#DNS1=[DNS-SERVER1]
#DNS2=[DNS-SERVER2]
IPV6INIT=no
USERCTL=no
IPV4_FAILURE_FATAL=yes
NAME="System eth2"
BRIDGE=iscsi-bridge0
MTU=9000

/etc/sysconfig/network-scripts/ifcfg-eth3:

# Intel Corporation 82571EB Gigabit Ethernet Controller
DEVICE=eth3
#IPADDR=X.Y.Z.232
#NETMASK=255.255.255.0
#PREFIX=24
#DEFROUTE=yes
#IPV4_FAILURE_FATAL=yes
HWADDR=00:15:17:6C:97:95
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
BRIDGE=iscsi-bridge1
IPV6INIT=no
USERCTL=no
NAME="System eth3"
MTU=9000

/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge0:

DEVICE=iscsi-bridge0
ONBOOT=yes
TYPE=Bridge
IPADDR=X.Y.Z.231
NETMASK=255.255.255.0
STP=off
MTU=9000
DELAY=0

/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge1:

DEVICE=iscsi-bridge1
ONBOOT=yes
TYPE=Bridge
IPADDR=X.Y.Z.232
NETMASK=255.255.255.0
MTU=9000
STP=off
DELAY=0

Although there’s a perfectly valid way of achieving the same thing through the virsh/libvirtd management interface to libvirt as well as with the Network Manager tools, my preference is to make this configuration “stick” using the old network init service. The problem(s) I see with the NetworkManager/libvirtd approach is twofold:

• Timing of NetworkManager start-up (not all that early) relative to the Open-iSCSI stack startup (early)
• Timing of libvirtd start-up (one of the last services to get called) relative to other iSCSI volumes needing to be available for the host environment.

So, for this example, disable NetworkManager as a boot service and enable the network service:

# service NetworkManager stop
# chkconfig NetworkManager off
# chkconfig network on
# ifup iscsi-bridge0
# ifup iscsi-bridge1

And, Bob’s yer uncle (or, at least, he should be!). To verify that everything is working properly, ping an IP target that should be reachable from the bridge device interface(s) only:

# ping -I iscsi-bridge0

# ping -I iscsi-bridge1

By the way: If you use bridged interfaces, no iSCSI volumes on the host (for guests only, in other words) and have iptables enabled on the host (which you should), make sure to configure your host iptables to leave the bridged interfaces alone. For details, see the – soon to be created, I promise – post about performance tuning the Linux IPv4 environment for iSCSI-initiators on this site. Alternatively, you can grab the information pertaining to the relevant bridge sysctl.conf entries from the KVM scalability white paper Red Hat published (and I provided most of the content for in my previous career).

The problem I’ve had, until recently, was providing actual – objective – data as a means to help illustrate my points.  For instance, I could not clearly illustrate how a snoop filter on the CPU interconnect can improve the linearity of the workload scalability in a virtualized environment (see Fig. 1).

Fig. 1: Average response time with pinned vs. un-pinned processors

I was unable to demonstrate benefits of the NUMA aware scheduler that the Linux kernel uses and how it does improve performance. (In figure 2, it’s represented by the improvement in average response times from the web-servers included in the workload) when your workloads run with memory interleaving disabled – see Fig. 2 and 3. Unless, for support reasons, your application vendor explicitly tells you otherwise, of course!

Fig. 2: Average Response Times - Non-interleaved Memory Config

Fig. 3: Average Response Times - Interleaved memory

I also used to have a hard time explaining how and why to tune the Linux kernel for these systems. For instance, I only suspected how little (none) tuning of the host platform is required in order to drive pretty significant numbers of guests  (98) in these environments – see Fig. 4. But, if you engage in some very minor tuning activities of the network stack, how those very same workload performance results can be extended even further (to 256 guests) – see Fig. 5:

Fig. 4: The system has not been tuned beyond it's "out of the box" state.

Fig. 5: System is tuned and exhibiting linear scalability to 256 KVM guests

As part of a joint documentation effort with Red Hat, all of the data collected has been brought together in a Reference Architecture document  – “Scaling RHEL 5.4 + KVM up to 256 Guests” available for free from Red Hat’s website.

We obviously picked the guest density to prove a point about the platform, however it’s worth mentioning that 256 guests does not represent the upper bound for the platform. It only represents where we thought the density went (far) beyond what is reasonable to expect in a production environment this day in age.

As a consequence, in my environment, I’ve got a couple of KVM guests that are running Fedora 12 with Red Hat Cluster v3.0.6 installed. That’s not really “living on the edge”. The “living on the edge” part of that configuration is that the two guests share a clustered file system. This clustered file system is hosted on a DRDB replicated volume between two standard internal SATA drives hosted on two different KVM host systems. And these host systems are, in turn their own Fedora 12 based Red Hat Cluster.

Obviously, there are plenty of opportunities for data to go “missing” (get corrupted/get lost/disappear/etc) in a configuration like this. And I thought I’d been able to eliminate them all.

That was what I thought, until I ran one of the KVM guests on one of the hosts, and the other on the other. My GFS2 file system wasn’t impressed! And I was stumped. DRBD had been configured with synchronous replication (let’s not talk about the performance impact of that decision, shall we…?) but obviously the data wasn’t being committed simultaneously to both drives^[1].

I now suspect that’s happening because the KVM hosts were caching the data on the guests behalf. Could be a very spiffy performance boost^[2] but causes all sorts of problems for my clustered applications that rely on the data in the file system being where it’s supposed to be.

So, I had to dig around a little and discovered  that Qemu/KVM/libvirt actually supports setting the caching properties for the ‘physical’ devices backing its virtual hard drives (i.e. the hard drives or container files exported to the guest as “disks”). And it’s – if you’re using the CLI interfaces for managing KVM, libvirtd & virsh – fairly easy to set it to what you want/need it to be.

The caching properties you can set are:

• writeback
• writethrough
• none
• default

Unfortunately, I’ve not been able to locate some way to set this while creating the guest with virt-manager. However, virt-install does let you set it, and if the guest is inactive (i.e. not running), you can set it by editing the <driver> tag.

For example:

<disk type='block' device='disk'>
   <driver name='qemu' cache='none'/>
   <source dev='/dev/mapper/sharedVG01-www--local'/>
   <target dev='vde' bus='virtio'/>
</disk>

# virsh version

Apropos:

[1] = I know, I know. A DRBD mirror set up to use “protocol C” doesn’t, technically, commit the data simultaneously to both devices. It only “looks” like that because the write() operation does not return success until the data has been successfully written on the “remote” device as well as the local one.

[2] = It is, actually. As an example, the reason why the likes of Xen, KVM, etc have been able to post IO benchmarks that are more than 100% the performance of the underlying hardware is because the host environment caches the data on the guests behalf. Looks good on benchmarks. Not so much if your host fails before the data have been flushed from the host cache onto the physical disk devices. Applications tend to get cranky when data they “know” was committed to persistent storage is missing…