Are you only seeing IP traffic on one of two (or more) interfaces in a Fedora 13 environment? I was, and here I’ll try to explain how I went about resolving (at least working around) the problem for my multi-homed dedicated iSCSI network.
So, as part of playing in my lab environment to configure Fedora 13 and an iSCSI RAID array (EqualLogic PS Series array) for multipathing, I ran into a bit of a snag:
Successful ping to [group-ip] from first iSCSI NIC
# ping -I eth2 [group-ip] PING [group-ip] ([group-ip]) from [eth2-ip] eth2: 56(84) bytes of data. 64 bytes from [group-ip]: icmp_seq=1 ttl=255 time=0.103 ms 64 bytes from [group-ip]: icmp_seq=2 ttl=255 time=0.075 ms 64 bytes from [group-ip]: icmp_seq=3 ttl=255 time=0.093 ms ^C --- [group-ip] ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2927ms rtt min/avg/max/mdev = 0.075/0.090/0.103/0.013 ms# tcpdump -i eth2 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes 08:30:55.601799 IP [eth2-ip] > [group-ip]: ICMP echo request, id 1650, seq 1, length 64 08:30:55.601933 IP [group-ip] > [eth2-ip]: ICMP echo reply, id 1650, seq 1, length 64 08:30:56.602616 IP [eth2-ip] > [group-ip]: ICMP echo request, id 1650, seq 2, length 64 08:30:56.602693 IP [group-ip] > [eth2-ip]: ICMP echo reply, id 1650, seq 2, length 64 08:30:57.602755 IP [eth2-ip] > [group-ip]: ICMP echo request, id 1650, seq 3, length 64 08:30:57.602831 IP [group-ip] > [eth2-ip]: ICMP echo reply, id 1650, seq 3, length 64
But failed ping to [group-ip] from second iSCSI NIC
# ping -I eth3 [group-ip] PING [group-ip] ([group-ip]) from [eth3-ip] eth3: 56(84) bytes of data. ^C --- [group-ip] ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2630ms# tcpdump -i eth3 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes 08:31:01.496852 IP [eth3-ip] > [group-ip]: ICMP echo request, id 1906, seq 1, length 64 08:31:01.496970 IP [group-ip] > [eth3-ip]: ICMP echo reply, id 1906, seq 1, length 64 08:31:02.496568 IP [eth3-ip] > [group-ip]: ICMP echo request, id 1906, seq 2, length 64 08:31:02.496660 IP [group-ip] > [eth3-ip]: ICMP echo reply, id 1906, seq 2, length 64 08:31:03.496591 IP [eth3-ip] > [group-ip]: ICMP echo request, id 1906, seq 3, length 64 08:31:03.496650 IP [group-ip] > [eth3-ip]: ICMP echo reply, id 1906, seq 3, length 64
Being, what appears to be, above average dense about this, I spent hours trying to figure out what was wrong. I’m still not convinced I figured out why this is happening (root cause), but basically the sysctl tunable net.ipv4.conf.{all,eth*,default}.rp_filter seemingly defaulting to 1 in Fedora 13, running version 2.6.33.6-147.fc13.x86_64 of the kernel, results in only one of the two NICs connected and configured for the dedicated iSCSI network in my lab environment responding to ICMP or TCP traffic (responses) from the group IP.
For some reason, if my Fedora host communicates with another (any?) host on the same subnet, it exhibits none of this behavior!
To work around this problem set you’ve got a few commands to issue:
The least work intensive (yet most disruptive and a whole lot like using a sledge hammer for the finishing touches!) option is to change net.ipv4.conf.default.rp_filter = 1 to 0 (zero) in /etc/sysctl.conf, and reboot. One (big) problem with this is the case where you’ve got other multi-homed networks connected to the same host and you want traffic filtering between NICs on the same subnet (not sure why you’d want that and you’re not using the bonding driver, but…).
Alternatively, you can issue the following command for each of your iSCSI network-facing NICs (# sysctl -w net.ipv4.conf.eth{0,1..N}.rp_filter=0) and keep your systems and workloads running. Basically, issue the above command for each of your eth, bridge or bond device attached to your dedicated IP network (and expected to read/write to/from the EqualLogic group) and voila(!) the NICs will “permit” packets from the group IP again. NOTE: This (# sysctl -w …) isn’t persistent across reboots!)
Successful ping to [group-ip] from second iSCSI NIC
# ping -I eth2 [group-ip] PING [group-ip] ([group-ip]) from [eth2-ip] eth2: 56(84) bytes of data. 64 bytes from [group-ip]: icmp_seq=1 ttl=255 time=0.103 ms 64 bytes from [group-ip]: icmp_seq=2 ttl=255 time=0.075 ms 64 bytes from [group-ip]: icmp_seq=3 ttl=255 time=0.093 ms ^C --- [group-ip] ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2927ms rtt min/avg/max/mdev = 0.075/0.090/0.103/0.013 ms# tcpdump -i eth2 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes 08:11:38.006766 IP [eth2-ip] > [group-ip]: ICMP echo request, id 19057, seq 1, length 64 08:11:38.006861 IP [group-ip] > [eth2-ip]: ICMP echo reply, id 19057, seq 1,length 64 08:11:39.007030 IP [eth2-ip] > [group-ip]: ICMP echo request, id 19057, seq 3, length 64 08:11:39.007098 IP [group-ip] > [eth2-ip]: ICMP echo reply, id 19057, seq 2,length 64 08:11:40.006615 IP [eth2-ip] > [group-ip]: ICMP echo request, id 19057, seq 3, length 64 08:11:40.006688 IP [group-ip] > [eth2-ip]: ICMP echo reply, id 19057, seq 3,length 64
AND successful ping to [group-ip] from second iSCSI NIC
# ping -I eth3 [group-ip] PING [group-ip] ([group-ip]) from [eth3-ip] eth3: 56(84) bytes of data. 64 bytes from [group-ip]: icmp_seq=1 ttl=255 time=0.103 ms 64 bytes from [group-ip]: icmp_seq=2 ttl=255 time=0.075 ms 64 bytes from [group-ip]: icmp_seq=3 ttl=255 time=0.093 ms ^C --- [group-ip] ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2927ms rtt min/avg/max/mdev = 0.075/0.090/0.103/0.013 ms# tcpdump -i eth3 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes 08:11:44.074841 IP [eth3-ip] > [group-ip]: ICMP echo request, id 19313, seq 1, length 64 08:11:44.074938 IP [group-ip] > [eth3-ip]: ICMP echo reply, id 19313, seq 1,length 64 08:11:45.075025 IP [eth3-ip] > [group-ip]: ICMP echo request, id 19313, seq 2, length 64 08:11:45.075123 IP [group-ip] > [eth3-ip]: ICMP echo reply, id 19313, seq 2, length 64
Update just to make sure I dot the i’s, etc; The sysctl settings do not cascade up to bridges built on top of a physical interface that’s had a specific setting, like net.ipv4.conf.[NIC].rp_filter configured. Thus you have to set .rp_filter=0 for all bridges as well.