That said, no AFIK, you do not need to use static routes (assuming the dynamic ones are correct). The key is to ensure that the MPIO engine is configured for Round Robin (or Shortest Queue First, if available) and that, if the target ports are on the same sub-net (i.e. 192.168.1.0/255.255.255.0), the sysctl.conf tunables mentioned elsewhere on this site are set correctly.

When using multipath with Centos 5.6 I can see traffic splitted on different target IPs on the same SAN, but its only using 1 interface to do that, it just doesnt split the traffic on both interfaces. It do failover however.

Can you update your posts? Does it need to have static routes to make it work spreading the bandwidth on both nics or multipath needs something special to be configured?

Besr regards.

I’d probably start by looking at my switch configuration. Do you have spanning tree configured on the ports – could be you’re seeing ports being disabled by the switch to mitigate switch loops, etc. “Port flapping” can often be the result of SPT recalculations (which can take up to 30+ seconds.

Also, I’d look into whether or not NetworkManager is involved (on servers I _always_ turn Network Manager off since I don’t see the point of having it active. It’s predominantly designed to be a client side tool (i.e. for laptops or systems inconsistently connected to different networks/wireless networks) and has – in my view – no purpose in a server environment with permanent & direct links.

Now, onto the throughput. The reason I moved from etherchannel (just like you did) is because Dell told me I should be using multipath as it’s the ‘supported’ method (same as you).

They said if I use multipath then connections to the group IP will get distributed among the eth0 and eth1 interfaces and I’ll be able to realise more than the ~1gbps limit which I had with the etherchannel.

Obviously you’re not seeing these benefits, and although I’m yet to test my changes, I can see connections are still only being made to eth0 on the EL SAN.

I feel we’re both possibly being spun out a bit by Dell. I’m going to try and get to the bottom of it, but I wondered if you had anything to add to your experience, since it was a few months ago now.

Thanks.

i forgot to mention: i switched this morning from the bonded method to the “recommended” method with multipath and individual NICS. I’ve had the bonded working great for 11.5mths, but have always been disappointed by the performance – I used the bonded/MLT (LACP) because i wanted the redundancy at the network end of things.

Since i thought this was the bottle neck, i’ve now tried the multipath/4 single NICs to see if there’s much difference.

With my initial tests, the IP address i have assigned to eth5, I’ve mounted a share using NFS on a separate system – interestingly, all of this traffic actually goes via the eth3 interface – it’s listed first in the routing table (it’s on the same subnet as eth2-5) – i’d expect the eth5 since that’s where the IP is “located”

I then saw the routing table, and that seems to be the reason (i’d read your previous post on the rp_filter, and that was already defaulted to 0 – it’s a RHEL5 system).

Reading the kernel docs for “arp_filter” (not rp_filter), it almost implies that i need to setup source-based routing:
“arp_filter – BOOLEAN
1 – Allows you to have multiple network interfaces on the same
subnet, and have the ARPs for each interface be answered
based on whether or not the kernel would route a packet from
the ARP’d IP out that interface (therefore you must use source
based routing for this to work). In other words it allows control
of which cards (usually 1) will respond to an arp request.

0 – (default) The kernel can respond to arp requests with addresses
from other interfaces. This may seem wrong but it usually makes
sense, because it increases the chance of successful communication.
IP addresses are owned by the complete host on Linux, not by
particular interfaces. Only for more complex setups like load-
balancing, does this behaviour cause problems.

arp_filter for the interface will be enabled if at least one of
conf/{all,interface}/arp_filter is set to TRUE,
it will be disabled otherwise”

Nevertheless, i tested it somewhat (it’s actually a production system so limited scope here). my test with dd if=/dev/zero of=testfile count=50000000 which results in a 26GB file. Whilst looking at iostat, i see traffic over all 4 “paths”. This worked out speed wise at ~896mbit/sec which seems awfully close to 1gbit (when taking into account the overhead of network protocols). And the strange thing, i see traffic on all 4 nics. It’s just an awful co-incidence that the write speed seems to be almost exactly at 1gbit … (it’s a PE2950 w/ 16gb ram). The unit PS6000XV had negligible other load at that time.

Re-running my DD test:

[user@host]# time dd if=/dev/zero of=testfile count=50000000
50000000+0 records in
50000000+0 records out
25600000000 bytes (26 GB) copied, 191.673 seconds, 134 MB/s

real 3m11.733s
user 0m19.957s
sys 2m51.032s

again, very close to 1Gbit…

So i’m either stuck with the fact that i’ve got a mis-configuration somewhere, or that the performance of the unit really is ~1gbit in terms of writes, or that the way to get a performing system still alludes me.

Also, how come your eth{2,3,4,5} all have IP addresses? If you’re using the bonding driver, I would have expected them all to be IP-less slaves (see “SLAVE” in the output from #ifconfig eth{2,3,4,5} with “bond0″ (or whatever you decide to call it) as the only entity having an IP address.

The output you included reminds me more of a solution w/no bonding enabled (i.e. the standard EqualLogic recommended configuration) and the dm-multipath driver loaded & configured?

Destination Gateway Genmask Flags MSS Window irtt Iface
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth5
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2

in this case, eth3 takes all the load, despite there being 4 iscsi sessions open from each of the interfaces (one session per interface)

in effect, while there might be load-balancing via multipath, everything is bottle-necked through one NIC?

For a supported configuration, Dell would tell you to split the bond, configure the NIC’s presently bonded as physical devices with IP addresses (i.e. 10.10.10.1, 10.10.10.2, etc) on the same subnet as the group IP (if possible) and then configure multipath.conf/multipathd.

The following would also work, but I’m not clear on whether or not it’s an officially supported configuration, since you’ve already got a bonded interface up, have you considered simply using the iSCSI initiator utilities to configure two (or more, depending on the size of your bond) iSCSI session interfaces, re-discover the LUN(s), re-log in to them and start the multipathd service.

# iscsiadm -m iface -I bond0_1 –op=new
# iscsiadm -m iface -I bond0_2 –op=new
# iscsiadm -m iface -I bond0_1 –op=update -n iface.net_ifacename -v bond0
# iscsiadm -m iface -I bond0_2 –op=update -n iface.net_ifacename -v bond0
# iscsiadm -m discovery -I bond0_1 -t st -p
# iscsiadm -m discovery -I bond0_2 -t st -p

I tend to remove the LUNs/Volumes I’m not using on the host (like the vss-control LUN, etc with # iscsiadm -m node –op=delete -T

# iscsiadm -m node –login -T [iqn for LUN/Volume]

Edit /etc/multipath.conf and start the multipathd service as well as check that the /dev/mapper device is present ( # multipath -v2 ; multipath -ll )

Again: I’ve not confirmed that we’ll actually support this setup (meaning, I know it works in my own sandbox environment, but YMMV and if you call support they could be telling you it’s not officially supported)