A quick request on the #selinux-fedora IRC channel on freenode.net clarified the whole thing, but I thought it worth posting so I won’t forget.

You may have realized, if you’ve read any of the other posts in this blog, that I have a multi-instance Apache Server configuration that shares a single GFS2 file system between two OS instances (they’re actually VMs) and I hosts a few different WordPress blogs and a Photo Gallery using this configuration. The pictures and source files for the blogs and gallery are mounted at a non-standard location (non-standard in a Fedora/httpd context) and since I recently re-enabled SELinux in my environment and set it to “Enforcing”, I’ve had to make some additions to the file context (fcontext) portions of the SELinux policy so as to ensure that any “.autorelabel” and # restorecon commands will set the wordpress and gallery source files to the correct file context (fcontext).

Basically, I’ve been creating file context regular expressions that align with my web server layout using the # semanage fcontext -a -t <context> “<file path regex>” command. At the time I created the contexts, I never noticed the fact that I was using two different contexts (they both worked, so it wasn’t obvious to me that I’d done this);

httpd_sys_rw_context_t

httpd_sys_content_rw_t

But, when I ran ” #restorecon -RFv <path to html root>” from both of my OS instances, I noticed that they were both labeling the same (shared) directories with the two different fcontexts. And that made me wonder…  What on earth was/is the difference between the two (and, secondarily, why do they both exist)?

To answer the first question first; “What on earth is the difference between the httpd_sys_rw_content_t fcontext and the httpd_sys_content_rw_t fcontext?”;

None! Zip! Nada! They’re aliases of each other and have the exact same meaning. Thus, if you have a directory where you wish to let the httpd (Apache) server write to – for instance an “upload” folder – you can set it to either context and SELinux will let the HTTPD Daemon write to the location as long as the remaining UNIX permissions are correct.

The 2nd question is harder to answer (or, in other words; I don’t have an authoritative answer to that question), but I can speculate that it’s a compatibility “play” to not break previously set fcontexts (“previously” being “fcontexts set by other selinux-policy versions and/or previous versions of the Fedora distribution itself”).

We tried everything. Different bonding algorithms (balanced-alb, 802.3ad, balanced-rr, etc), different IO testing tools, different network configurations, messing with the rr_io_min settings, tuning the network stack itself (increasing read/write buffer sizes) and then we got desperate and changed the IO schedulers (For the record: I consider that “an act of desperation” since I would never expect an IO scheduler change to give us a 100% increase).

Since we didn’t configure the physical aspects of the network and we’re playing with equipment hosted in a rather large and very established Lab SAN, it took me a while to think of and accept that it might be a physical problem and not a host configuration one. It was probably about time to walk to the switches and get a visual on the port status for the host links, the Controller port lights and any ISLs or (if present) stacking links.

Guess what… the ISL between the switch connected to the test host and the switch connected to the EqualLogic group was operating at 1/2 its intended capacity.

“Mystery” solved.

You know, sometimes the blinking lights are the most important troubleshooting tool. Sometimes, the color of the blinking lights are the difference between wasting 3 hours and not. And, sometimes, a person with more than 10 years experience designing, configuring and reviewing SANs in mission critical environments can be reminded – in very embarrassing ways – that you should always verify the physical aspects of the configuration first and that simple is more often than not better than the alternative(s). Even if the simple approach includes a hike, a flight of stairs and entering into the noisy data center.

This will permit the libvirt virtualization (management) abstraction interface to easily build “briges of bridges” that in turn let a Kernel Virtual Machine (KVM) guest get it’s own “public” (only in quotes because I happen to think the average bear would not be so silly as to put their Linux/KVM host directly onto the internet. Right???) IP address and route its traffic directly onto the ether (via the lower levels of the IP stack of the host environment).

There are, as is the case with all things Linux or UNIX, a couple of ways to skin this particular bear (sorry, that’s bad!), but the one that makes the most sense to me is to have init take care of the configuration as part of the system boot process (when the network service executes). And doing that, although in its simplest form requires access to a terminal window and a text editor on the Fedora host, is actually very simple, once you know what you’re doing. Hopefully, the following will help you learn (if you don’t already know and are only reading this because you’re looking around and are a very bored individual).

At a high level, all you have to do is edit the /etc/sysconfig/network-scripts/ifcfg-ethX file for the Ethernet (eth) device you want to bridge, specify that the device will belong to a bridge named <bridgename> and then create a /etc/sysconfig/network-scripts/ifcfg-<bridgename> file that configures the bridge (assigning it a way to obtain an IP address – probably static, routing and DNS configuration information).

Prior to configuring the system to use bridging, I had configured static IP addresses on the two physical interfaces I will be creating bridges for. Since that configuration was being obsoleted in order to use the same interfaces for guest-to-iSCSI-SAN traffic, I edited and created the above mentioned configuration files. All of the edited items are in bold. Before you edit and create these files, life gets a little easier if you:

1. Back up the original ifcfg-eth[N] files to some other location than /etc/sysconfig/network-scripts/
2. # ifdown eth[N]

Then, as an example, the two-to-four files you need (ifcfg-eth[N], ifcfg-eth[N+1] and ifcfg-bridge[N] and ifcfg-bridge[N+1]).

/etc/sysconfig/network-scripts/ifcfg-eth2:

# Intel Corporation 82571EB Gigabit Ethernet Controller
DEVICE=eth2
BOOTPROTO=none
HWADDR=00:15:17:6C:97:94
#IPADDR=X.X.Z.231
#NETMASK=255.255.255.0
#PREFIX=24
#DEFROUTE=yes
ONBOOT=yes
TYPE=Ethernet
#DNS1=[DNS-SERVER1]
#DNS2=[DNS-SERVER2]
IPV6INIT=no
USERCTL=no
IPV4_FAILURE_FATAL=yes
NAME="System eth2"
BRIDGE=iscsi-bridge0
MTU=9000

/etc/sysconfig/network-scripts/ifcfg-eth3:

# Intel Corporation 82571EB Gigabit Ethernet Controller
DEVICE=eth3
#IPADDR=X.Y.Z.232
#NETMASK=255.255.255.0
#PREFIX=24
#DEFROUTE=yes
#IPV4_FAILURE_FATAL=yes
HWADDR=00:15:17:6C:97:95
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
BRIDGE=iscsi-bridge1
IPV6INIT=no
USERCTL=no
NAME="System eth3"
MTU=9000

/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge0:

DEVICE=iscsi-bridge0
ONBOOT=yes
TYPE=Bridge
IPADDR=X.Y.Z.231
NETMASK=255.255.255.0
STP=off
MTU=9000
DELAY=0

/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge1:

DEVICE=iscsi-bridge1
ONBOOT=yes
TYPE=Bridge
IPADDR=X.Y.Z.232
NETMASK=255.255.255.0
MTU=9000
STP=off
DELAY=0

Although there’s a perfectly valid way of achieving the same thing through the virsh/libvirtd management interface to libvirt as well as with the Network Manager tools, my preference is to make this configuration “stick” using the old network init service. The problem(s) I see with the NetworkManager/libvirtd approach is twofold:

• Timing of NetworkManager start-up (not all that early) relative to the Open-iSCSI stack startup (early)
• Timing of libvirtd start-up (one of the last services to get called) relative to other iSCSI volumes needing to be available for the host environment.

So, for this example, disable NetworkManager as a boot service and enable the network service:

# service NetworkManager stop
# chkconfig NetworkManager off
# chkconfig network on
# ifup iscsi-bridge0
# ifup iscsi-bridge1

And, Bob’s yer uncle (or, at least, he should be!). To verify that everything is working properly, ping an IP target that should be reachable from the bridge device interface(s) only:

# ping -I iscsi-bridge0

# ping -I iscsi-bridge1

By the way: If you use bridged interfaces, no iSCSI volumes on the host (for guests only, in other words) and have iptables enabled on the host (which you should), make sure to configure your host iptables to leave the bridged interfaces alone. For details, see the – soon to be created, I promise – post about performance tuning the Linux IPv4 environment for iSCSI-initiators on this site. Alternatively, you can grab the information pertaining to the relevant bridge sysctl.conf entries from the KVM scalability white paper Red Hat published (and I provided most of the content for in my previous career).

But before we get into the nitty-gritty of that, I just wanted to touch on a problem I’ve been having in setting up multipath over IPv4 on my Fedora 13 host. Now, keep in mind that the Fedora host I’m using has been “tweaked” and “messed with” (a lot), so it ain’t no “fresh install” here… That said, I’ve been having problems getting both of my array-facing NICs to successfully ping the EqualLogic Group IP address.

So, if you’re able to ping the EqualLogic array’s group IP address from both Ethernet NICs, setting up the iSCSI initiator as well as the DM multipath module(s) is very much a “straight forward” exercise.

Create two new open-iscsi initiator interfaces

# iscsiadm --mode iface --op=new --interface iscsi-0

# iscsiadm --mode iface --op=new --interface iscsi-1

Validate that the new interfaces are stored

# iscsiadm --mode iface

default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
iscsi-0 tcp,<empty>,<empty>,<empty>,<empty>
iscsi-1 tcp,<empty>,<empty>,<empty>,<empty>

Bind the physical NICs to the newly created initiator interfaces

# iscsiadm --mode iface --op=update --interface iscsi-0 --name=iface.net_ifacename --value=eth2

# iscsiadm --mode iface --op=update --interface iscsi-1 --name=iface.net_ifacename --value=eth3

Validate the bindings

# iscsiadm --mode iface --interface iscsi-0

# BEGIN RECORD 2.0-872
iface.iscsi_ifacename = iscsi-0
iface.net_ifacename = eth2 
iface.ipaddress = <empty>
iface.hwaddress = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
# END RECORD

# iscsiadm --mode iface --interface iscsi-1

# BEGIN RECORD 2.0-872
iface.iscsi_ifacename = iscsi-1
iface.net_ifacename = eth3 
iface.ipaddress = <empty>
iface.hwaddress = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
# END RECORD

# iscsiadm --mode iface

default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
iscsi-0 tcp,<empty>,<empty>,eth2,<empty>
iscsi-1 tcp,<empty>,<empty>,eth3,<empty>

Now that we’ve got the initiator interfaces configured and bound to the physical 1 or 10 GigE NICs, we need to make sure every initiator NIC is logged into the available targets. By the way, you can also configure your Fedora host environment to use bridged interfaces if you’ve got a limited number of NICs in the system and want to use KVM/libvirt to import LUNs directly to a virtualized guest instance.

The easiest way to log in all of the interfaces you’ve defined above to the available (the LUNs/Volumes are “available” and “seen” because you’ve set up the ACL on the LUN/Volume to permit access from this host, right?) targets  is to simply list^[1] the target(s) you’ve discovered previously and re-issue the # iscsiadm –mode node –login –target iqn.2001-05.com.equallogic:<ID of Volume/LUN> command.

The next step now is to edit/add the contents of the /etc/multipath.conf (or, if you’re so inclined: /etc/multipath/multipath.conf which can represent a symlink target for /etc/multipath.conf) file, assign an alias that can be access from /dev/mapper/<alias> for a given (multipathed) device and “do something” (use as an LVM2 volume, a raw device, host a file system on or whatever)

So, before we “do the work” to configure the Multipath environment, we’ll need to either add (or ensure) the following information is present in the /etc/multipath.conf file (by the way, the multipath utilities assume [expect] its configuration file to be readable from the /etc/multipath.conf location!):

## Use user friendly names, instead of using WWIDs as names.
defaults {
     user_friendly_names yes
     find_multipaths yes
}

## Blacklist local devices (no need to have them accidentally "multipathed")
blacklist {
     wwid scsi-SATA_WDC_WD800AAJS-1_WD-WMAM9ANC0895
     devnode "^sd[a]$"
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
     devnode "^hd[a-z]"
     devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

## For EqualLogic PS series RAID arrays
devices {
     device {
          vendor                  "EQLOGIC"
          product                 "100E-00"
          path_grouping_policy    multibus
          getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
          features                "1 queue_if_no_path"
          path_checker            readsector0
          path_selector           "round-robin 0"
          failback                immediate
          rr_min_io               10
          rr_weight               priorities
     }
}

## Assign human-readable name  (aliases) to the World-Wide ID's (names) of the devices we're wanting to manage
## Identify the WWID by issuing # scsi_id --whitelisted --page=0x83 /dev/<path to sdX for one of the device paths>

multipaths {
     multipath {
          wwid                    36090a01850e08bd883c4a45bfa0330ba
          alias                     benchmark2
     }
     multipath {
          wwid                    36090a01850e04bd683c4745bfa03b07c
          alias                     benchmark1
     }
}

# service multipathd start
# chkconfig multipathd on

Verify that the device mapper multipath configuration is ‘complete’ with the following commands:

# multipath -v2
# multipath -ll

benchmark2 (36090a01850e08bd883c4a45bfa0330ba) dm-0 EQLOGIC,100E-00
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
   |- 6:0:0:0 sdd 8:48 active ready running
   `- 5:0:0:0 sdb 8:16 active ready running
benchmark1 (36090a01850e04bd683c4745bfa03b07c) dm-1 EQLOGIC,100E-00
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
   |- 4:0:0:0 sde 8:64 active ready running
   `- 3:0:0:0 sdc 8:32 active ready running

[1] = # iscsiadm –mode node

It’s pretty straight forward, really.

At its simplest and  highest(ish) level:

1. Install the iSCSI initiator management utilities for Fedora 13 (# yum install iscsi-initiator-utils)
2. Identify (change) the iqn for your Fedora 13 host (# {cat|vi} /etc/iscsi/initiatorname.iscsi)
3. Configure your target(s), define LUN(s) and set up ACLs for the LUNs (Volumes) you’re wanting to use in your EqualLogic group
4. Scan the iSCSI target (EqualLogic group) (# iscsiadm –mode discovery –type sendtargets –portal <ip for EQL group>:3260)
5. Log in to the target & volume (LUN) you’re wanting to use (# iscsiadm –mode node –login –target <iqn for LUN/Volume on EQL array> )
6. Repeat for all of the volumes/LUNs you’re wanting to access from the Fedora 13 host
7. Use the volumes/LUNs you’ve “imported”
8. Optionally (I recommend doing this for clarity), log out and remove (delete) all of the Volumes (LUNs) you’re not using on the Fedora host
   1. List all of the discovered volumes/LUNs from the target: (# iscsiadm –mode node)
   2. Log out the volumes/LUN’s you do not intend to access from the host (# iscsiadm –mode node –logout –target <iqn of Volume/LUN> )
   3. Delete the volume(s)/LUN(s) you don’t want: (# iscsiadm –mode node –target <iqn of Volume/LUN> –op delete )

If you don’t like “long” option names, substitute;

• –mode with -m
• –target with -T
• –login with -l
• –logout with -u
• –op with -o
• ‘sendtargets‘ with ‘st‘

From now on, whenever the iscsid daemon is started during boot, the host will log on to the volume(s)/LUNs during boot, the /dev/ entries will be created – but not be persistent across boots, so make sure you use these LUNs with either LVM or use the UUID= or file system LABEL=<name> when mounting file systems hosted on these Volume(s)/LUNs! – and the devices will be accessible from the host.  More on creating UDEV rules for your iSCSI Volume(s)/LUN(s) in the next  installment of the series.

Of course, now there’s configuring for Volume/LUN path availability, etc, etc. Multipath configuration which is documented by DELL in a tech report paper on our EqualLogic support web site (requires a valid support contract/warranty for login)- and I’ll summarize in another post.

I’ve got a configuration in mind which I’ll document over time, probably. Sorry for being so incredibly vague on the time-line, but my job isn’t to document Linux and EqualLogic configurations so I’m only doing this “for fun” between learning about the PS Series arrays (have to do a live demo “soon”) and various other job tasks.

There are plenty of people out there, using hardware that the hardware vendors have not classified as “Mission Critical”, for what amounts to being highly “mission critical” solutions for their businesses, and vice versa (people using servers that are “mission-critical” for workloads that really aren’t). Examples of this use could include mail servers, firewalls, small but critical databases, etc, etc. There are also examples that are time dependent in nature; For instance, what would happen if a file and print service that was being used by the finance department to generate various time sensitive reports went off-line during the end of quarter or year processing…?

I know, it’s a contrived example, but it does actually demonstrate the point. It is how a  given system is used within the business that defines its degree of criticality, not the server it runs on!

I’ll be the first to concede that hardware certainly can help ensure that a mission critical service/workload remains available to its end users (and can, sometimes, be very good at ensuring the service/workload is not available too!) but it’s by no means the only or most critical component in that equation!

What defines “availability”?

To some, the “number of nines” – i.e. “five nines” means 99.999% availability – is the gold standard. I tend to agree that the “number of nines” is certainly not only one of the best known metrics for availability, it’s also a very good one. It captures the essence of the problem; How much time did my service spend being available to its consumers versus the amount of time it was unavailable. And a typical “Mission Critical” environment tends to operate within the 4-5 “nines” range.

Since “four nines” represents about 52 minutes of downtime, and “five nines” about 5 minutes of downtime per year, most people tend to only think in terms of large, expensive, “enterprise class” servers when thinking about architecting for mission critical workloads. Why? Because, historically speaking, these systems were the only ones where it was technologically feasible to include the level of silicon and hardware designs containing the capabilities needed in order to guarantee four or five nines at a price point customers (you) were willing to accept.

However, with the  growing acceptance of virtualization technologies such as VM Ware, Microsoft hyper B., Citrix and server and the various open source virtualization technologies included in the major commercial Linux distributions, the picture of what a “mission-critical” server looks like has changed.

It’s now possible to quickly build “always on” system architectures based on industry-standard hardware and extremely flexible hypervisor’s and management software. And you can do it for  what amounts to pennies on the dollar compared to the traditionally very expensive and often proprietary systems from the likes of Bull, Fujitsu, NEC, IBM,  Sun (now Oracle), HP, etc.

In my view, the greatest paradigm shift in terms of highly available mission-critical workload designs, follows on the heels of one key enabling technology. The advent and acceptance of live-migration in production environments, basically means that there are no  technological limits to how flexible your application environment can be. If you were to pair “always-on” live migration, resource management, high availability services, data replication and a little bit of ingenuity, you could with relative ease have a fault tolerant solution for your application environment.  Or, if your resource needs aren’t too great, you could leverage the VMware ESX 4.0 fault tolerance capability. Basically the sky’s the limit!

So, to answer my first question, yes PCs can be used for mission-critical workloads. The introduction of virtualization technologies to the x86 platform only makes your job of ensuring availability for those mission-critical workloads a whole lot easier. At least, that’s my opinion.

The problem I’ve had, until recently, was providing actual – objective – data as a means to help illustrate my points.  For instance, I could not clearly illustrate how a snoop filter on the CPU interconnect can improve the linearity of the workload scalability in a virtualized environment (see Fig. 1).

Fig. 1: Average response time with pinned vs. un-pinned processors

I was unable to demonstrate benefits of the NUMA aware scheduler that the Linux kernel uses and how it does improve performance. (In figure 2, it’s represented by the improvement in average response times from the web-servers included in the workload) when your workloads run with memory interleaving disabled – see Fig. 2 and 3. Unless, for support reasons, your application vendor explicitly tells you otherwise, of course!

Fig. 2: Average Response Times - Non-interleaved Memory Config

Fig. 3: Average Response Times - Interleaved memory

I also used to have a hard time explaining how and why to tune the Linux kernel for these systems. For instance, I only suspected how little (none) tuning of the host platform is required in order to drive pretty significant numbers of guests  (98) in these environments – see Fig. 4. But, if you engage in some very minor tuning activities of the network stack, how those very same workload performance results can be extended even further (to 256 guests) – see Fig. 5:

Fig. 4: The system has not been tuned beyond it's "out of the box" state.

Fig. 5: System is tuned and exhibiting linear scalability to 256 KVM guests

As part of a joint documentation effort with Red Hat, all of the data collected has been brought together in a Reference Architecture document  – “Scaling RHEL 5.4 + KVM up to 256 Guests” available for free from Red Hat’s website.

We obviously picked the guest density to prove a point about the platform, however it’s worth mentioning that 256 guests does not represent the upper bound for the platform. It only represents where we thought the density went (far) beyond what is reasonable to expect in a production environment this day in age.

I was looking to grow a shared (clustered) GFS2 file system I’m using for my web server and did the “usual” magic of extending the underlying logical volume (clvm managed), extend the disk label with fdisk and fsck -f the device so as to ensure there’s nothing latent in the metadata or data to cause my “grow” command to fail.

Then I issued

# gfs2_growfs /webroot

Which failed with the following error:

[snip] GFS2 rindex not found.  Please run gfs2_fsck

Well, that’s interesting….

Turns out an update to  GFS2 didn’t get reflected in gfs2_grow, so it’s looking in the wrong place for the rindex. The problem has been addressed upstreamand you can (I hope) expect to see the next version of the gfs2-utils (3.0.7) contain the fix.

UPDATE: The linked .rpm for Fedora 12 resolves the problem. The bug is a minor fix, thankfully, so I’d be surprised if a fixed version of the gfs2-utils package will not be included when they make the Cluster v3.0.7 packages available in the fedora-updates repository.

Thanks to the stellar help from the folks in the #linux-cluster channel on the Freenode IRC network! (fabbione, bob_home and abhi)

As a consequence, in my environment, I’ve got a couple of KVM guests that are running Fedora 12 with Red Hat Cluster v3.0.6 installed. That’s not really “living on the edge”. The “living on the edge” part of that configuration is that the two guests share a clustered file system. This clustered file system is hosted on a DRDB replicated volume between two standard internal SATA drives hosted on two different KVM host systems. And these host systems are, in turn their own Fedora 12 based Red Hat Cluster.

Obviously, there are plenty of opportunities for data to go “missing” (get corrupted/get lost/disappear/etc) in a configuration like this. And I thought I’d been able to eliminate them all.

That was what I thought, until I ran one of the KVM guests on one of the hosts, and the other on the other. My GFS2 file system wasn’t impressed! And I was stumped. DRBD had been configured with synchronous replication (let’s not talk about the performance impact of that decision, shall we…?) but obviously the data wasn’t being committed simultaneously to both drives^[1].

I now suspect that’s happening because the KVM hosts were caching the data on the guests behalf. Could be a very spiffy performance boost^[2] but causes all sorts of problems for my clustered applications that rely on the data in the file system being where it’s supposed to be.

So, I had to dig around a little and discovered  that Qemu/KVM/libvirt actually supports setting the caching properties for the ‘physical’ devices backing its virtual hard drives (i.e. the hard drives or container files exported to the guest as “disks”). And it’s – if you’re using the CLI interfaces for managing KVM, libvirtd & virsh – fairly easy to set it to what you want/need it to be.

The caching properties you can set are:

• writeback
• writethrough
• none
• default

Unfortunately, I’ve not been able to locate some way to set this while creating the guest with virt-manager. However, virt-install does let you set it, and if the guest is inactive (i.e. not running), you can set it by editing the <driver> tag.

For example:

<disk type='block' device='disk'>
   <driver name='qemu' cache='none'/>
   <source dev='/dev/mapper/sharedVG01-www--local'/>
   <target dev='vde' bus='virtio'/>
</disk>

# virsh version

Apropos:

[1] = I know, I know. A DRBD mirror set up to use “protocol C” doesn’t, technically, commit the data simultaneously to both devices. It only “looks” like that because the write() operation does not return success until the data has been successfully written on the “remote” device as well as the local one.

[2] = It is, actually. As an example, the reason why the likes of Xen, KVM, etc have been able to post IO benchmarks that are more than 100% the performance of the underlying hardware is because the host environment caches the data on the guests behalf. Looks good on benchmarks. Not so much if your host fails before the data have been flushed from the host cache onto the physical disk devices. Applications tend to get cranky when data they “know” was committed to persistent storage is missing…

One of the engineers I’ve been working with to help me get a better understanding for what the performance and scalability of a scale-up environment when being used to run Red Hat Virtualization (Xen) as well as the Kernel Virtual Machine (KVM) is changing roles within the company. Good for him!

Unfortunately, this means that I’ll have to do some of the data-collection work on my own. On top of my daily responsibilities. So, we met this week to try and figure out how we could compress what’s been, thus far, a multi-month effort into something a bit more schedule friendly for the both of us. And I think we’ve got a reasonable plan, all things considered.

One of the achilles heels of almost any of the virtualization solutions, thus far, has been the IO throughput (and latency) for disk related IO operations. If you look at testing done by various vendors and the Open Source community at large, the throughput has been either “not great” or inconsistent. The “not great” element of this hasn’t, historically, really been that huge of a deal for customers, since they seem to have planned around it by not including disk IO intensive workloads into their consolidation/virtualization plans.

However, more and more customers are looking to virtualize everything running on their server platforms in an effort to save power, cooling and management costs. As a result, the “not great” performance behavior has become enough of an issue that all of the virtualization vendors now support (or will in the near future), minimally, para-virtualized IO drivers and/or other performance optimizations.

Consequently, they also appear to be pushing Linux (and Windows) further “back” into their data centers and looking to use the two for more and more critical tasks, even in a virtualized/consolidated context. An on-going problem for myself and our customers has been the amount of fact-based information available in terms of how to best configure these environments to optimize IO performance or to help customers understand the actual limitations and benefits of the various IO options in a virtualized environment.

Also, there’s very little in terms of stated best practices for using things like raw devices, volume manager backed devices, file systems or file containers for the various types of workloads. So, I was hoping we’d be able to provide some of that in a presentation at a large conference this summer.  And my original plan was that I’d only be the consumer of the data, not the creator of it. Thus the “best laid plans of mice and men” statement early on.

So, for the next couple of weeks, I’ll be trying to collect whatever the engineer is unable to collect. Then I’ll have to graph it, ‘gussy it up’ (with pretty colors), add some configuration recommendations and bring it all with me to the sessions I have planned for the conference.