We tried everything. Different bonding algorithms (balanced-alb, 802.3ad, balanced-rr, etc), different IO testing tools, different network configurations, messing with the rr_io_min settings, tuning the network stack itself (increasing read/write buffer sizes) and then we got desperate and changed the IO schedulers (For the record: I consider that “an act of desperation” since I would never expect an IO scheduler change to give us a 100% increase).

Since we didn’t configure the physical aspects of the network and we’re playing with equipment hosted in a rather large and very established Lab SAN, it took me a while to think of and accept that it might be a physical problem and not a host configuration one. It was probably about time to walk to the switches and get a visual on the port status for the host links, the Controller port lights and any ISLs or (if present) stacking links.

Guess what… the ISL between the switch connected to the test host and the switch connected to the EqualLogic group was operating at 1/2 its intended capacity.

“Mystery” solved.

You know, sometimes the blinking lights are the most important troubleshooting tool. Sometimes, the color of the blinking lights are the difference between wasting 3 hours and not. And, sometimes, a person with more than 10 years experience designing, configuring and reviewing SANs in mission critical environments can be reminded – in very embarrassing ways – that you should always verify the physical aspects of the configuration first and that simple is more often than not better than the alternative(s). Even if the simple approach includes a hike, a flight of stairs and entering into the noisy data center.

The problem I’ve had, until recently, was providing actual – objective – data as a means to help illustrate my points.  For instance, I could not clearly illustrate how a snoop filter on the CPU interconnect can improve the linearity of the workload scalability in a virtualized environment (see Fig. 1).

Fig. 1: Average response time with pinned vs. un-pinned processors

I was unable to demonstrate benefits of the NUMA aware scheduler that the Linux kernel uses and how it does improve performance. (In figure 2, it’s represented by the improvement in average response times from the web-servers included in the workload) when your workloads run with memory interleaving disabled – see Fig. 2 and 3. Unless, for support reasons, your application vendor explicitly tells you otherwise, of course!

Fig. 2: Average Response Times - Non-interleaved Memory Config

Fig. 3: Average Response Times - Interleaved memory

I also used to have a hard time explaining how and why to tune the Linux kernel for these systems. For instance, I only suspected how little (none) tuning of the host platform is required in order to drive pretty significant numbers of guests  (98) in these environments – see Fig. 4. But, if you engage in some very minor tuning activities of the network stack, how those very same workload performance results can be extended even further (to 256 guests) – see Fig. 5:

Fig. 4: The system has not been tuned beyond it's "out of the box" state.

Fig. 5: System is tuned and exhibiting linear scalability to 256 KVM guests

As part of a joint documentation effort with Red Hat, all of the data collected has been brought together in a Reference Architecture document  – “Scaling RHEL 5.4 + KVM up to 256 Guests” available for free from Red Hat’s website.

We obviously picked the guest density to prove a point about the platform, however it’s worth mentioning that 256 guests does not represent the upper bound for the platform. It only represents where we thought the density went (far) beyond what is reasonable to expect in a production environment this day in age.