The problem I’ve had, until recently, was providing actual – objective – data as a means to help illustrate my points.  For instance, I could not clearly illustrate how a snoop filter on the CPU interconnect can improve the linearity of the workload scalability in a virtualized environment (see Fig. 1).

Fig. 1: Average response time with pinned vs. un-pinned processors

I was unable to demonstrate benefits of the NUMA aware scheduler that the Linux kernel uses and how it does improve performance. (In figure 2, it’s represented by the improvement in average response times from the web-servers included in the workload) when your workloads run with memory interleaving disabled – see Fig. 2 and 3. Unless, for support reasons, your application vendor explicitly tells you otherwise, of course!

Fig. 2: Average Response Times - Non-interleaved Memory Config

Fig. 3: Average Response Times - Interleaved memory

I also used to have a hard time explaining how and why to tune the Linux kernel for these systems. For instance, I only suspected how little (none) tuning of the host platform is required in order to drive pretty significant numbers of guests  (98) in these environments – see Fig. 4. But, if you engage in some very minor tuning activities of the network stack, how those very same workload performance results can be extended even further (to 256 guests) – see Fig. 5:

Fig. 4: The system has not been tuned beyond it's "out of the box" state.

Fig. 5: System is tuned and exhibiting linear scalability to 256 KVM guests

As part of a joint documentation effort with Red Hat, all of the data collected has been brought together in a Reference Architecture document  – “Scaling RHEL 5.4 + KVM up to 256 Guests” available for free from Red Hat’s website.

We obviously picked the guest density to prove a point about the platform, however it’s worth mentioning that 256 guests does not represent the upper bound for the platform. It only represents where we thought the density went (far) beyond what is reasonable to expect in a production environment this day in age.

One of the engineers I’ve been working with to help me get a better understanding for what the performance and scalability of a scale-up environment when being used to run Red Hat Virtualization (Xen) as well as the Kernel Virtual Machine (KVM) is changing roles within the company. Good for him!

Unfortunately, this means that I’ll have to do some of the data-collection work on my own. On top of my daily responsibilities. So, we met this week to try and figure out how we could compress what’s been, thus far, a multi-month effort into something a bit more schedule friendly for the both of us. And I think we’ve got a reasonable plan, all things considered.

One of the achilles heels of almost any of the virtualization solutions, thus far, has been the IO throughput (and latency) for disk related IO operations. If you look at testing done by various vendors and the Open Source community at large, the throughput has been either “not great” or inconsistent. The “not great” element of this hasn’t, historically, really been that huge of a deal for customers, since they seem to have planned around it by not including disk IO intensive workloads into their consolidation/virtualization plans.

However, more and more customers are looking to virtualize everything running on their server platforms in an effort to save power, cooling and management costs. As a result, the “not great” performance behavior has become enough of an issue that all of the virtualization vendors now support (or will in the near future), minimally, para-virtualized IO drivers and/or other performance optimizations.

Consequently, they also appear to be pushing Linux (and Windows) further “back” into their data centers and looking to use the two for more and more critical tasks, even in a virtualized/consolidated context. An on-going problem for myself and our customers has been the amount of fact-based information available in terms of how to best configure these environments to optimize IO performance or to help customers understand the actual limitations and benefits of the various IO options in a virtualized environment.

Also, there’s very little in terms of stated best practices for using things like raw devices, volume manager backed devices, file systems or file containers for the various types of workloads. So, I was hoping we’d be able to provide some of that in a presentation at a large conference this summer.  And my original plan was that I’d only be the consumer of the data, not the creator of it. Thus the “best laid plans of mice and men” statement early on.

So, for the next couple of weeks, I’ll be trying to collect whatever the engineer is unable to collect. Then I’ll have to graph it, ‘gussy it up’ (with pretty colors), add some configuration recommendations and bring it all with me to the sessions I have planned for the conference.