Bugzilla – Bug 495
Port-forwarding in Dom0 causes corrupt TCP checksums in DomU
Last modified: 2006-02-28 21:47:56 CST
I encountered a very odd problem which I eventually found a workaround for, but I wanted to make sure the developers are aware of it. The basic idea is that I created a NAT-ed domU using the standard scripts and kernels from the 2.6.12.6-xen3_7.1_fc4 RPM. I can make outgoing connections from DomU and it is otherwise fine. I then forwarded an external port to the domU: iptables -t nat -A PREROUTING -p tcp --dst 15.4.89.26 --dport 11014 \ -j DNAT --to 10.202.107.174:22 15.4.89.26 and 11014 are the external IP address and external port, respectively and 10.202.107.174 and 22 are the internal IP address and port, respectively. I then did > ssh -p 11014 15.4.89.26 from another machine. This hung. Upon closer inspection, I saw that DomU was sending some of its TCP packets with a corrupt TCP checksum (?!): [root@klai-tycoon ~]# tcpdump -i eth0 -nvvvvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 17:34:34.202033 IP (tos 0x10, ttl 63, id 1746, offset 0, flags [DF], proto 6, length: 60) 15.4.89.35.47694 > 10.202.107.174.ssh: S [tcp sum ok] 845312449:845312449(0) win 5840 <mss 1460,sackOK,timestamp 257914643 0,nop,wscale 2> 17:34:34.225891 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 6, length: 60) 10.202.107.174.ssh > 15.4.89.35.47694: S [tcp sum ok] 3227641594:3227641594(0) ack 845312450 win 5792 <mss 1460,sackOK,timestamp 4294947271 257914643,nop,wscale 2> 17:34:34.202227 IP (tos 0x10, ttl 63, id 1748, offset 0, flags [DF], proto 6, length: 52) 15.4.89.35.47694 > 10.202.107.174.ssh: . [tcp sum ok] 1:1(0) ack 1 win 1460 <nop,nop,timestamp 257914643 4294947271> 17:34:34.236769 IP (tos 0x0, ttl 64, id 22332, offset 0, flags [DF], proto 6, length: 72) 10.202.107.174.ssh > 15.4.89.35.47694: P [bad tcp cksum ded9 (->9f8f)!] 1:21(20) ack 1 win 1448 <nop,nop,timestamp 4294947275 257914643> 17:34:34.446430 IP (tos 0x0, ttl 64, id 22334, offset 0, flags [DF], proto 6, length: 72) 10.202.107.174.ssh > 15.4.89.35.47694: P [bad tcp cksum ded9 (->9f7a)!] 1:21(20) ack 1 win 1448 <nop,nop,timestamp 4294947296 257914643> 17:34:34.866394 IP (tos 0x0, ttl 64, id 22336, offset 0, flags [DF], proto 6, length: 72) 10.202.107.174.ssh > 15.4.89.35.47694: P [bad tcp cksum ded9 (->9f50)!] 1:21(20) ack 1 win 1448 <nop,nop,timestamp 4294947338 257914643> 17:34:35.706410 IP (tos 0x0, ttl 64, id 22338, offset 0, flags [DF], proto 6, length: 72) 10.202.107.174.ssh > 15.4.89.35.47694: P [bad tcp cksum ded9 (->9efc)!] 1:21(20) ack 1 win 1448 <nop,nop,timestamp 4294947422 257914643> This only happens for traffic forwarded through Dom0. I have no idea why this happens. After much debugging, I found that executing > iptables -t nat --list on DomU fixes this problem. In particular, having the iptable_nat kernel module loaded in DomU causes the correct checksums to be used.
I have this bug too, with a Xen unstable nightly tarball downloaded on 2006/02/02. Here is a tcpdump of a simple TCP connection made from a domU. Note that the initial 3 way handshake goes OK, but thereafter all TCP packets have a bad checksum. tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 15:18:47.955933 IP (tos 0x10, ttl 63, id 55208, offset 0, flags [DF], length: 60) 192.168.2.250.4361 > 80.68.91.176.80: S [tcp sum ok] 87812993:87812993(0) win 5840 <mss 1460,sackOK,timestamp 146473 0,nop,wscale 2> 15:18:47.989850 IP (tos 0x0, ttl 55, id 0, offset 0, flags [DF], length: 60) 80.68.91.176.80 > 192.168.2.250.4361: S [tcp sum ok] 2338351:2338351(0) ack 87812994 win 5792 <mss 1460,sackOK,timestamp 164554683 146473,nop,wscale 0> 15:18:47.990004 IP (tos 0x10, ttl 63, id 55209, offset 0, flags [DF], length: 52) 192.168.2.250.4361 > 80.68.91.176.80: . [tcp sum ok] 1:1(0) ack 1 win 1460 <nop,nop,timestamp 146482 164554683> 15:18:51.900292 IP (tos 0x10, ttl 63, id 55210, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->4dca)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 147460 164554683> 15:18:52.134882 IP (tos 0x10, ttl 63, id 55211, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->4d8f)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 147519 164554683> 15:18:52.606902 IP (tos 0x10, ttl 63, id 55212, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->4d19)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 147637 164554683> 15:18:53.550959 IP (tos 0x10, ttl 63, id 55213, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->4c2d)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 147873 164554683> 15:18:55.567116 IP (tos 0x10, ttl 63, id 55214, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->4a35)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 148377 164554683> 15:18:59.583385 IP (tos 0x10, ttl 63, id 55215, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->4649)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 149381 164554683> 15:19:07.135834 IP (tos 0x10, ttl 63, id 55216, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->3ee9)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 151269 164554683> 15:19:22.240792 IP (tos 0x10, ttl 63, id 55217, offset 0, flags [DF], length: 68) 192.168.2.250.4361 > 80.68.91.176.80: P [bad tcp cksum fe8b (->3029)!] 1:17(16) ack 1 win 1460 <nop,nop,timestamp 155045 164554683> I am using the standard NAT scripts. Rich (rich at annexia dot org)
And I can also confirm that simply loading the iptable_nat module _in the domU_ fixes this bug - I have no idea why. Rich.
*** This bug has been marked as a duplicate of 447 ***