We tried everything. Different bonding algorithms (balanced-alb, 802.3ad, balanced-rr, etc), different IO testing tools, different network configurations, messing with the rr_io_min settings, tuning the network stack itself (increasing read/write buffer sizes) and then we got desperate and changed the IO schedulers (For the record: I consider that “an act of desperation” since I would never expect an IO scheduler change to give us a 100% increase).

Since we didn’t configure the physical aspects of the network and we’re playing with equipment hosted in a rather large and very established Lab SAN, it took me a while to think of and accept that it might be a physical problem and not a host configuration one. It was probably about time to walk to the switches and get a visual on the port status for the host links, the Controller port lights and any ISLs or (if present) stacking links.

Guess what… the ISL between the switch connected to the test host and the switch connected to the EqualLogic group was operating at 1/2 its intended capacity.

“Mystery” solved.

You know, sometimes the blinking lights are the most important troubleshooting tool. Sometimes, the color of the blinking lights are the difference between wasting 3 hours and not. And, sometimes, a person with more than 10 years experience designing, configuring and reviewing SANs in mission critical environments can be reminded – in very embarrassing ways – that you should always verify the physical aspects of the configuration first and that simple is more often than not better than the alternative(s). Even if the simple approach includes a hike, a flight of stairs and entering into the noisy data center.

As a consequence, in my environment, I’ve got a couple of KVM guests that are running Fedora 12 with Red Hat Cluster v3.0.6 installed. That’s not really “living on the edge”. The “living on the edge” part of that configuration is that the two guests share a clustered file system. This clustered file system is hosted on a DRDB replicated volume between two standard internal SATA drives hosted on two different KVM host systems. And these host systems are, in turn their own Fedora 12 based Red Hat Cluster.

Obviously, there are plenty of opportunities for data to go “missing” (get corrupted/get lost/disappear/etc) in a configuration like this. And I thought I’d been able to eliminate them all.

That was what I thought, until I ran one of the KVM guests on one of the hosts, and the other on the other. My GFS2 file system wasn’t impressed! And I was stumped. DRBD had been configured with synchronous replication (let’s not talk about the performance impact of that decision, shall we…?) but obviously the data wasn’t being committed simultaneously to both drives^[1].

I now suspect that’s happening because the KVM hosts were caching the data on the guests behalf. Could be a very spiffy performance boost^[2] but causes all sorts of problems for my clustered applications that rely on the data in the file system being where it’s supposed to be.

So, I had to dig around a little and discovered  that Qemu/KVM/libvirt actually supports setting the caching properties for the ‘physical’ devices backing its virtual hard drives (i.e. the hard drives or container files exported to the guest as “disks”). And it’s – if you’re using the CLI interfaces for managing KVM, libvirtd & virsh – fairly easy to set it to what you want/need it to be.

The caching properties you can set are:

• writeback
• writethrough
• none
• default

Unfortunately, I’ve not been able to locate some way to set this while creating the guest with virt-manager. However, virt-install does let you set it, and if the guest is inactive (i.e. not running), you can set it by editing the <driver> tag.

For example:

<disk type='block' device='disk'>
   <driver name='qemu' cache='none'/>
   <source dev='/dev/mapper/sharedVG01-www--local'/>
   <target dev='vde' bus='virtio'/>
</disk>

# virsh version

Apropos:

[1] = I know, I know. A DRBD mirror set up to use “protocol C” doesn’t, technically, commit the data simultaneously to both devices. It only “looks” like that because the write() operation does not return success until the data has been successfully written on the “remote” device as well as the local one.

[2] = It is, actually. As an example, the reason why the likes of Xen, KVM, etc have been able to post IO benchmarks that are more than 100% the performance of the underlying hardware is because the host environment caches the data on the guests behalf. Looks good on benchmarks. Not so much if your host fails before the data have been flushed from the host cache onto the physical disk devices. Applications tend to get cranky when data they “know” was committed to persistent storage is missing…