i forgot to mention: i switched this morning from the bonded method to the “recommended” method with multipath and individual NICS. I’ve had the bonded working great for 11.5mths, but have always been disappointed by the performance – I used the bonded/MLT (LACP) because i wanted the redundancy at the network end of things.

Since i thought this was the bottle neck, i’ve now tried the multipath/4 single NICs to see if there’s much difference.

With my initial tests, the IP address i have assigned to eth5, I’ve mounted a share using NFS on a separate system – interestingly, all of this traffic actually goes via the eth3 interface – it’s listed first in the routing table (it’s on the same subnet as eth2-5) – i’d expect the eth5 since that’s where the IP is “located”

I then saw the routing table, and that seems to be the reason (i’d read your previous post on the rp_filter, and that was already defaulted to 0 – it’s a RHEL5 system).

Reading the kernel docs for “arp_filter” (not rp_filter), it almost implies that i need to setup source-based routing:
“arp_filter – BOOLEAN
1 – Allows you to have multiple network interfaces on the same
subnet, and have the ARPs for each interface be answered
based on whether or not the kernel would route a packet from
the ARP’d IP out that interface (therefore you must use source
based routing for this to work). In other words it allows control
of which cards (usually 1) will respond to an arp request.

0 – (default) The kernel can respond to arp requests with addresses
from other interfaces. This may seem wrong but it usually makes
sense, because it increases the chance of successful communication.
IP addresses are owned by the complete host on Linux, not by
particular interfaces. Only for more complex setups like load-
balancing, does this behaviour cause problems.

arp_filter for the interface will be enabled if at least one of
conf/{all,interface}/arp_filter is set to TRUE,
it will be disabled otherwise”

Nevertheless, i tested it somewhat (it’s actually a production system so limited scope here). my test with dd if=/dev/zero of=testfile count=50000000 which results in a 26GB file. Whilst looking at iostat, i see traffic over all 4 “paths”. This worked out speed wise at ~896mbit/sec which seems awfully close to 1gbit (when taking into account the overhead of network protocols). And the strange thing, i see traffic on all 4 nics. It’s just an awful co-incidence that the write speed seems to be almost exactly at 1gbit … (it’s a PE2950 w/ 16gb ram). The unit PS6000XV had negligible other load at that time.

Re-running my DD test:

[user@host]# time dd if=/dev/zero of=testfile count=50000000
50000000+0 records in
50000000+0 records out
25600000000 bytes (26 GB) copied, 191.673 seconds, 134 MB/s

real 3m11.733s
user 0m19.957s
sys 2m51.032s

again, very close to 1Gbit…

So i’m either stuck with the fact that i’ve got a mis-configuration somewhere, or that the performance of the unit really is ~1gbit in terms of writes, or that the way to get a performing system still alludes me.

Also, how come your eth{2,3,4,5} all have IP addresses? If you’re using the bonding driver, I would have expected them all to be IP-less slaves (see “SLAVE” in the output from #ifconfig eth{2,3,4,5} with “bond0″ (or whatever you decide to call it) as the only entity having an IP address.

The output you included reminds me more of a solution w/no bonding enabled (i.e. the standard EqualLogic recommended configuration) and the dm-multipath driver loaded & configured?

Destination Gateway Genmask Flags MSS Window irtt Iface
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth5
10.100.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2

in this case, eth3 takes all the load, despite there being 4 iscsi sessions open from each of the interfaces (one session per interface)

in effect, while there might be load-balancing via multipath, everything is bottle-necked through one NIC?

For a supported configuration, Dell would tell you to split the bond, configure the NIC’s presently bonded as physical devices with IP addresses (i.e. 10.10.10.1, 10.10.10.2, etc) on the same subnet as the group IP (if possible) and then configure multipath.conf/multipathd.

The following would also work, but I’m not clear on whether or not it’s an officially supported configuration, since you’ve already got a bonded interface up, have you considered simply using the iSCSI initiator utilities to configure two (or more, depending on the size of your bond) iSCSI session interfaces, re-discover the LUN(s), re-log in to them and start the multipathd service.

# iscsiadm -m iface -I bond0_1 –op=new
# iscsiadm -m iface -I bond0_2 –op=new
# iscsiadm -m iface -I bond0_1 –op=update -n iface.net_ifacename -v bond0
# iscsiadm -m iface -I bond0_2 –op=update -n iface.net_ifacename -v bond0
# iscsiadm -m discovery -I bond0_1 -t st -p
# iscsiadm -m discovery -I bond0_2 -t st -p

I tend to remove the LUNs/Volumes I’m not using on the host (like the vss-control LUN, etc with # iscsiadm -m node –op=delete -T

# iscsiadm -m node –login -T [iqn for LUN/Volume]

Edit /etc/multipath.conf and start the multipathd service as well as check that the /dev/mapper device is present ( # multipath -v2 ; multipath -ll )

Again: I’ve not confirmed that we’ll actually support this setup (meaning, I know it works in my own sandbox environment, but YMMV and if you call support they could be telling you it’s not officially supported)