Bugzilla Bug 753

I have the following setup 

eth1 -- +      ___vlanXXX-br_(bridge)___
        |      |                       |
      bond0 -> | vlanXXX       vif0.0 -|- veth0 (has the only IP AND MAC)
        |      |_______________________|
eth0 -- +

Arp traffic don't get though. Everything works, If i run tcpdump on any of the
interfaces. It doesn't even have to be in promiscuous mode, running tcpdump
with -p (Don’t  put  the  interface into promiscuous mode) makes arp entries
get set too.

Without tcpdump running no arp entries get set. If you manualy enter an arp
entry for a host it works fine.

I'll attach my startup script, ip addr show, brctl show, /proc/net/vlan/config,
and /proc/net/bonding/bond0, and dmesg output

--- startup script ---
#!/bin/sh
# read in config for dom0
. /etc/xen/network/dom0

modprobe bonding
modprobe 8021q
modprobe bridge

mac=`ip link show eth0 | grep 'link\/ether' | sed -e 's/.*ether
\(..:..:..:..:..:..\).*/\1/'`
ip link set bond0 down arp off
ip link set bond0 addr fe:ff:ff:ff:ff:ff
ip addr flush bond0
ip link set bond0 up
ifenslave bond0 eth0 eth1

vconfig set_name_type VLAN_PLUS_VID_NO_PAD

# vlans are stored in /etc/xen/network/vlans one number per line
# it'll create a vlanXXX-br bridge for eath vlan

for vlan in `cat /etc/xen/network/vlans` ; do
   vconfig add bond0 ${vlan}
   ip link set vlan${vlan} up
   brctl addbr vlan${vlan}-br
   brctl stp vlan${vlan}-br off
   brctl setfd vlan${vlan}-br off
   #ip link set vlan${vlan} down arp off
   brctl addif vlan${vlan}-br vlan${vlan}
   ip link set vlan${vlan}-br up
done

# setup dom0's ip addr on one of the vlan bridgs
# as defined in /etc/xen/network/dom0
brctl addif vlan${dom0_VLAN}-br vif0.0

ip link set veth0 addr $mac arp on
ip link set vif0.0 up
ip link set veth0 up
ethtool -K veth0 tx off
ifconfig veth0 mtu 1400

ip addr add ${dom0_IPADDR} broadcast ${dom0_BOADCAST} dev veth0
ip route add default via ${dom0_GATEWAY} dev veth0

--- ip addr show output ----
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: vif0.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
3: veth0: <BROADCAST,MULTICAST,UP> mtu 1400 qdisc noqueue
    link/ether 00:0d:60:4e:48:80 brd ff:ff:ff:ff:ff:ff
    inet 172.21.97.10/24 brd 172.21.97.255 scope global veth0
4: eth0: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0
qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
5: eth1: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0
qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,NOARP,MASTER,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
7: vlan151: <BROADCAST,MULTICAST,NOARP,MASTER,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
8: vlan151-br: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

----- brctl show -----
# brctl show
bridge name     bridge id               STP enabled     interfaces
vlan151-br              8000.feffffffffff       no              vlan151
                                                        vif0.0
---- /proc/net/vlan/config ---
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_PLUS_VID_NO_PAD
vlan151        | 151  | bond0

--- /proc/net/bonding/bond0 ----
Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0d:60:4e:48:80

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0d:60:4e:48:81


------ dmesg output -----
[    4.241856] NET: Registered protocol family 2
[    5.110174] IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
[    5.111697] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    5.113015] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    5.114387] TCP: Hash tables configured (established 8192 bind 8192)
[    5.115718] TCP reno registered
[    5.117130] TCP bic registered
[    5.118477] NET: Registered protocol family 1
[    5.119811] NET: Registered protocol family 8
[    5.121138] NET: Registered protocol family 20
[   13.583165] ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) ->
IRQ 18
[   13.608167] eth0: Tigon3 [partno(BCM95704A41) rev 2002 PHY(serdes)]
(PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:4e:48:80
[   13.611700] eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0]
WireSpeed[0] TSOcap[1]
[   13.614293] eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
[   13.615686] ACPI: PCI Interrupt 0000:01:00.1[B] -> GSI 17 (level, low) ->
IRQ 19
[   13.640281] eth1: Tigon3 [partno(BCM95704A41) rev 2002 PHY(serdes)]
(PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:4e:48:81
[   13.643278] eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0]
WireSpeed[0] TSOcap[1]
[   13.645883] eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
[   37.972543] 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
[   37.973966] All bugs added by David S. Miller <davem@redhat.com>
[   38.470076] Bridge firewalling registered
[   38.561883] NET: Registered protocol family 17
[   38.920085] bonding: bond0: Warning: failed to get speed and duplex from
eth0, assumed to be 100Mb/sec and Full.
[   38.922690] bonding: bond0: enslaving eth0 as an active interface with an up
link.
[   39.290077] bonding: bond0: Warning: failed to get speed and duplex from
eth1, assumed to be 100Mb/sec and Full.
[   39.292682] bonding: bond0: enslaving eth1 as an active interface with an up
link.
[   39.350118] vlan151: dev_set_promiscuity(master, 1)
[   39.351448] device eth0 entered promiscuous mode
[   39.352738] device eth1 entered promiscuous mode
[   39.354082] device bond0 entered promiscuous mode
[   39.355426] device vlan151 entered promiscuous mode
[   39.359806] vlan151-br: port 1(vlan151) entering learning state
[   39.379655] device vif0.0 entered promiscuous mode
[   39.385422] vlan151-br: port 2(vif0.0) entering learning state
[   43.350243] tg3: eth0: Link is up at 1000 Mbps, full duplex.
[   43.351526] tg3: eth0: Flow control is off for TX and off for RX.
[   43.700256] tg3: eth1: Link is up at 1000 Mbps, full duplex.
[   43.701601] tg3: eth1: Flow control is off for TX and off for RX.
[   54.360062] vlan151-br: topology change detected, propagating
[   54.361410] vlan151-br: port 1(vlan151) entering forwarding state
[   54.380060] vlan151-br: topology change detected, propagating
[   54.381401] vlan151-br: port 2(vif0.0) entering forwarding state

I can confirm it, same happens for me with Xen 3.0.2.
Previously, without bonding, everything worked fine.

When I set up bonding, ARP traffic doesn't seem to work properly, and ARP
entries have to be set up manually (i.e., for the default gateway).

This looks like a linux networking issue rather than a Xen one, but I'm
interested in the soloution. I suspect bridging and bonding aren't normally
used together. You might want to consider using a routed xen setup e.g. with
the network-route and vif-route scripts.

Back to bridging, what happens if you use bonding for high-availability
active-backup rather than link aggregation? (mode=1 when loading the bonding
driver, as I recall)

I tried option bonding mode=0 though 6 with the same results. If it is a linux
networking issue how do we proceed to get it fixed?

Eli

(In reply to comment #2)
> This looks like a linux networking issue rather than a Xen one, but I'm
> interested in the soloution. I suspect bridging and bonding aren't normally
> used together. You might want to consider using a routed xen setup e.g. with
> the network-route and vif-route scripts.
> 
> Back to bridging, what happens if you use bonding for high-availability
> active-backup rather than link aggregation? (mode=1 when loading the bonding
> driver, as I recall)
> 

Do you really need to use bridging? I'd go with a routed soloution unless
you're prepared to do a bit of digging (which would be good as it would be nice
to fix this).

Unless i'm missing something i don't think routing will work on our network,
vlans are required for any network connection and each domU is going to end up
on a diffrent vlan. I think i would need a diffrent ip for each vlan on the
dom0 and then route the domU in that vlan to the dom0 ip in that vlan, thus
doubling the ip I need (and probably couldn't get).

I guess i could use nat and give the dom0 all the real ips and domU fake ips
and one to one nat everything, but that seems like a pain, and while i can't
think of any now it mite cause other problems.

For now i can run briging + vlans without bonding, i just have no fault
tolrance on network cards. 

I'll keep digging It seems the next step is probably to send a bug report to
either the maintainers of the bridging code or the bonding code. I'll see what
i can do to reproduce this without vlans or xen to narrow it down.

Eli

(In reply to comment #4)
> Do you really need to use bridging? I'd go with a routed soloution unless
> you're prepared to do a bit of digging (which would be good as it would be nice
> to fix this).
> 

Ok i've got nowhere to test bonding without vlans so its going to be hard to
isolate the problem. So whats the next step in getting this fixed?

Eli

It'll be tedious, but perhaps set the scenario up using another physical NIC
and another machine connected via a crossover cable?

That should generate a good bug report to fire into the linux networking folk
(or put it back in our court) 

Hmm, the issue seems to be deeper.

My script when starting xend without bonding looks like that:


#!/bin/sh -x

dir=$(dirname "$0")

# below is needed, as oopses/panics occured when it was not there
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0 

# VLANs and attached bridges
"$dir/network-bridge" "$@" vifnum=1 netdev=eth0.100 bridge=xenbr100 
"$dir/network-bridge" "$@" vifnum=2 netdev=eth0.101 bridge=xenbr101
"$dir/network-bridge" "$@" vifnum=3 netdev=eth0.102 bridge=xenbr102


With bonding, I needed to change the script:

#!/bin/sh -x

dir=$(dirname "$0")

"$dir/network-bridge" "$@" vifnum=0 netdev=bond0 bridge=xenbr0 

# needed, as bonding is changed, otherwise no connectivity
ifenslave pbond0 eth0 eth1

# script executes fine up to this point

# below ths point, kernel oopses, but doesn't panic
"$dir/network-bridge" "$@" vifnum=1 netdev=bond0.100 bridge=xenbr100 
"$dir/network-bridge" "$@" vifnum=2 netdev=bond0.101 bridge=xenbr101
"$dir/network-bridge" "$@" vifnum=3 netdev=bond0.102 bridge=xenbr102


This is the oops (the oops happens whether I use "ifenslave pbond0..." or not):

Sep 13 14:39:37 syn2 ifplugd(xenbr100)[3692]: ifplugd 0.28 initializing.
Sep 13 14:39:37 syn2 ifplugd(xenbr100)[3692]: Using interface
xenbr100/00:00:00:00:00:00 with driver <bridge> (version: 2.1)
Sep 13 14:39:37 syn2 ifplugd(xenbr100)[3692]: Using detection mode: SIOCETHTOOL
Sep 13 14:39:37 syn2 ifplugd(xenbr100)[3692]: Initialization complete, link
beat detected.
Sep 13 14:39:37 syn2 ifplugd(xenbr100)[3692]: Executing
'/etc/ifplugd/ifplugd.action xenbr100 up'.
Sep 13 14:39:37 syn2 ifplugd(xenbr100)[3692]: Program executed successfully.
Sep 13 14:39:37 syn2 kernel: bond0.100: del 33:33:ff:0e:7d:2c mcast address
from vlan interface
Sep 13 14:39:37 syn2 kernel: bond0.100: del 33:33:ff:0e:7d:2c mcast address
from master interface
Sep 13 14:39:37 syn2 kernel: bond0.100: del 33:33:00:00:00:01 mcast address
from vlan interface
Sep 13 14:39:37 syn2 kernel: bond0.100: del 33:33:00:00:00:01 mcast address
from master interface
Sep 13 14:39:37 syn2 kernel: bonding: Warning: Cannot create
/proc/net/bonding/<NULL>
Sep 13 14:39:37 syn2 kernel: Unable to handle kernel NULL pointer dereference
at virtual address 00000228
Sep 13 14:39:37 syn2 kernel:  printing eip:
Sep 13 14:39:37 syn2 kernel: c01acc83
Sep 13 14:39:37 syn2 kernel: *pde = ma 00000000 pa fffff000
Sep 13 14:39:37 syn2 kernel: Oops: 0000 [#1]
Sep 13 14:39:37 syn2 kernel: SMP
Sep 13 14:39:37 syn2 kernel: Modules linked in: bridge crc32c libcrc32c ipv6
iscsi_tcp libiscsi scsi_transport_iscsi tg3 bonding 8021q loop hw_random tsdev
ext3 jbd sd_mod ahci ata_piix libata scsi_mod raid1 md_mod
Sep 13 14:39:37 syn2 kernel: CPU:    0
Sep 13 14:39:37 syn2 kernel: EIP:    0061:[sysfs_remove_group+35/128]    Not
tainted VLI
Sep 13 14:39:37 syn2 kernel: EIP:    0061:[<c01acc83>]    Not tainted VLI
Sep 13 14:39:37 syn2 kernel: EFLAGS: 00010202   (2.6.16-xen #6)
Sep 13 14:39:37 syn2 kernel: EIP is at sysfs_remove_group+0x23/0x80
Sep 13 14:39:37 syn2 kernel: eax: 00000000   ebx: c92127a0   ecx: 00000007  
edx: 000001f8
Sep 13 14:39:37 syn2 kernel: esi: c6491a80   edi: c920a498   ebp: c6491810  
esp: c43abe48
Sep 13 14:39:37 syn2 kernel: ds: 007b   es: 007b   ss: 0069
Sep 13 14:39:37 syn2 kernel: Process ip (pid: 3749, threadinfo=c43aa000
task=c1215030)
Sep 13 14:39:37 syn2 kernel: Stack: <0>c43abe54 c91feaaf 00000007 c9212d24
0000000a c920946f 000001f8 c92127a0
Sep 13 14:39:37 syn2 kernel:        c9203084 c6491a80 c6491800 c9212664
c6491800 c012e01d c9212664 0000000a
Sep 13 14:39:37 syn2 kernel:        c6491800 0000000a c6491800 fc3591d7
c02add53 c0435940 0000000a c6491800
Sep 13 14:39:37 syn2 kernel: Call Trace:
Sep 13 14:39:38 syn2 kernel:  [<c91feaaf>] bond_create_proc_entry+0x8f/0xa0
[bonding]
Sep 13 14:39:38 syn2 kernel:  [<c920946f>] bond_destroy_sysfs_entry+0x1f/0x30
[bonding]
Sep 13 14:39:38 syn2 kernel:  [<c9203084>] bond_netdev_event+0x84/0xc0
[bonding]
Sep 13 14:39:38 syn2 kernel:  [notifier_call_chain+45/80]
notifier_call_chain+0x2d/0x50
Sep 13 14:39:38 syn2 kernel:  [<c012e01d>] notifier_call_chain+0x2d/0x50
Sep 13 14:39:38 syn2 kernel:  [dev_change_name+387/512]
dev_change_name+0x183/0x200
Sep 13 14:39:38 syn2 kernel:  [<c02add53>] dev_change_name+0x183/0x200
Sep 13 14:39:38 syn2 kernel:  [find_get_page+36/128] find_get_page+0x24/0x80
Sep 13 14:39:38 syn2 kernel:  [<c0142bd4>] find_get_page+0x24/0x80
Sep 13 14:39:38 syn2 kernel:  [dev_ifsioc+396/912] dev_ifsioc+0x18c/0x390
Sep 13 14:39:38 syn2 kernel:  [<c02adf5c>] dev_ifsioc+0x18c/0x390
Sep 13 14:39:38 syn2 kernel:  [dev_ioctl+891/1072] dev_ioctl+0x37b/0x430
Sep 13 14:39:38 syn2 kernel:  [<c02ae52b>] dev_ioctl+0x37b/0x430
Sep 13 14:39:38 syn2 kernel:  [fd_install+44/112] fd_install+0x2c/0x70
Sep 13 14:39:38 syn2 kernel:  [<c016567c>] fd_install+0x2c/0x70
Sep 13 14:39:38 syn2 kernel:  [do_ioctl+53/160] do_ioctl+0x35/0xa0
Sep 13 14:39:39 syn2 kernel:  [<c017af35>] do_ioctl+0x35/0xa0
Sep 13 14:39:39 syn2 kernel:  [vfs_ioctl+93/816] vfs_ioctl+0x5d/0x330
Sep 13 14:39:39 syn2 kernel:  [<c017affd>] vfs_ioctl+0x5d/0x330
Sep 13 14:39:39 syn2 kernel:  [sys_ioctl+136/160] sys_ioctl+0x88/0xa0
Sep 13 14:39:39 syn2 kernel:  [<c017b358>] sys_ioctl+0x88/0xa0
Sep 13 14:39:39 syn2 kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Sep 13 14:39:39 syn2 kernel:  [<c0105109>] syscall_call+0x7/0xb
Sep 13 14:39:39 syn2 kernel: Code: 83 c4 08 5b 5e c3 89 f6 57 53 83 ec 0c 8b 5c
24 1c 8b 54 24 18 8b 3b 85 ff 74 4d 31 c0 b9 ff ff ff ff f2 ae f7 d1 49 89 4c
24 08 <8b> 42 30 89 44 24 04 8b 03 89 04 24 e8 bc 9a fc ff 89 c7 89 f8
Sep 13 14:39:47 syn2 kernel:  <7>xenbr100: no IPv6 routers present

I have the same configuration but without the vlan.

Tomasz Chmielewski has wrote that we must define arp entry manualy but where ?

I'll keep digging too.

Hi,

This is just a comment to support all others users that have the same problem
:)

I've tested various vlan + bonding architecture and unfortunately not works.
Finally I've used a solution with just vlan without bonding from this site 

http://felipe-alfaro.org/blog/category/xen/

So, I hope that this bug and ethtool and mtu tweaks required to use simple vlan
and bonding will be fixed asap

I think the better way should be to configure vlan inside the dom0
Currently I've configured vlan in each domU loading 8021q module ... not very
flexible isn't it ... with  mtu tweak to 1496 

I've used bi-xeon Supermicro server with  2 x gigabit intel cards ...
that support perfeclty vlan ... normaly ;)

Regards

I have the same problem.

Hi,

I believe your problem is as follows,
veth0 sends arp-request for ex. IP 1.2.3.4 to mac broadcast.
vlanXXX-br forwards the frame and marks "veth0 mac" as belonging to vif0.0
port.
bond0 sends the frame through ex. eth0.
The switch connected to eth0 floods all ports except the incoming port (eth0),
and marks "veth0 mac" as belonging to eth0 port.
The flood includes the eth1 port.
The bridge vlanXXX-br forwards the frame and marks "veth0 mac" as belonging to
vlanXXX port.

The destination (1.2.3.4) sends arp reply to "veth0 mac".
The switch connected to eth0/eth1, sends the reply to the eth0 port.
The bridge vlanXXX-br blocks the frame as it believes the destination belongs
to vlanXXX port.

If you use a kernel <= 2.6.16.y you must apply the following patch:
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff_plain;h=8f903c708fcc2b579ebf16542bf6109bad593a1d;hp=ebe19a4ed78d4a11a7e01cdeda25f91b7f2fcb5a

If your NIC have vlan HW acceleration and the kernel <= 2.6.17-rc5, you must
also apply the following patch:
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff_plain;h=7ea49ed73c8d0d0bdf7c11fc18c61572d2d22176;hp=774bd8613d0188cda72408f8ffce258d73953cad

(In reply to comment #12)
> Hi,
> 
> I believe your problem is as follows,
> veth0 sends arp-request for ex. IP 1.2.3.4 to mac broadcast.
> vlanXXX-br forwards the frame and marks "veth0 mac" as belonging to vif0.0
> port.
> bond0 sends the frame through ex. eth0.
> The switch connected to eth0 floods all ports except the incoming port (eth0),
> and marks "veth0 mac" as belonging to eth0 port.
> The flood includes the eth1 port.
> The bridge vlanXXX-br forwards the frame and marks "veth0 mac" as belonging to
> vlanXXX port.
> 
> The destination (1.2.3.4) sends arp reply to "veth0 mac".
> The switch connected to eth0/eth1, sends the reply to the eth0 port.
> The bridge vlanXXX-br blocks the frame as it believes the destination belongs
> to vlanXXX port.

You can verify the above as follows:
Make sure domU net is quiet...

In dom0 "watch -n 1 brctl showmacs vlanXXXX-br".
Note the port containing "vif0.0 mac".
In domU "ping non-existing-addr-on-domU-lan" on other console/xterm/...
If you see "veth0 mac" on other port than "vif0.0 mac", the patches from
comment #12 helps

Regards,
/Kristian Høgh

Yeeeeppppyyyy !

It's work greet for me!
Thx a lot !