Moby: Docker container not reachable, bad TCP checksums
Hi,
We are unable to reach the ports n the container directly or mapped.
INFO
Docker version 1.5.0, build a8a31ef
Debian 7 backported kernel 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1~bpo70+1 (2015-04-27) x86_64 GNU/Linux
Containers: 14
Images: 29
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 57
Execution Driver: native-0.2
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 4
Total Memory: 7.815 GiB
Name: xxxxx
ID: PRMV:P7KM:5U6J:PYGZ:I3LQ:QWSB:ANFJ:VA23:URM4:JQOG:7TLM:4PTI
WARNING: No memory limit support
WARNING: No swap limit support
This is our additional settings:
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:26765 errors:0 dropped:0 overruns:0 frame:0
TX packets:30835 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2173785 (2.0 MiB) TX bytes:63950837 (60.9 MiB)
vethc356415 Link encap:Ethernet HWaddr 9e:e7:da:08:87:c6
inet6 addr: fe80::9ce7:daff:fe08:87c6/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:200 errors:0 dropped:0 overruns:0 frame:0
TX packets:147 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:12828 (12.5 KiB) TX bytes:9398 (9.1 KiB)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 xxxxxxxxxxxx 0.0.0.0 UG 0 0 0 eth0
xxxxxxxxxx 0.0.0.0 255.255.255.240 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
iptables -nL DOCKER
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.20 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 172.17.0.20 tcp dpt:80
The running container with the interface vethc356415 is a Discourse container. These are its host and network settings:
"HostConfig": {
"Binds": [
"/var/discourse/shared/standalone:/shared",
"/var/discourse/shared/standalone/log/var-log:/var/log"
],
"CapAdd": null,
"CapDrop": null,
"ContainerIDFile": "",
"Devices": [],
"Dns": null,
"DnsSearch": null,
"ExtraHosts": null,
"IpcMode": "",
"Links": null,
"LxcConf": [],
"NetworkMode": "bridge",
"PidMode": "",
"PortBindings": {
"22/tcp": [
{
"HostIp": "",
"HostPort": "2222"
}
],
"80/tcp": [
{
"HostIp": "",
"HostPort": "10001"
}
]
},
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"RestartPolicy": {
"MaximumRetryCount": 0,
"Name": "always"
},
"SecurityOpt": null,
"VolumesFrom": null
},
"NetworkSettings": {
"Bridge": "docker0",
"Gateway": "172.17.42.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.20",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"LinkLocalIPv6Address": "fe80::42:acff:fe11:14",
"LinkLocalIPv6PrefixLen": 64,
"MacAddress": "02:42:ac:11:00:14",
"PortMapping": null,
"Ports": {
"22/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "2222"
}
],
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "10001"
}
]
}
},
From inside of the container we are able to ping websites:
ping google.com
PING google.com (62.168.125.50) 56(84) bytes of data.
64 bytes from 62.168.125.50: icmp_seq=1 ttl=59 time=0.486 ms
64 bytes from 62.168.125.50: icmp_seq=2 ttl=59 time=0.520 ms
We are also able to get the index page of Discourse with wget (eg. doing a wget localhost from inside of container).
The problem is when we try to reach the container from outside (eg. from the Docker host):
$ wget 172.17.0.20
or
$ wget localhost:10001
does not work and hangs. When we tried sniffing the traffic on docker0 interface we found out that there is a TCP checksum problem:
# tcpdump -i docker0 -vvv
tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:19:11.098052 IP (tos 0x0, ttl 64, id 30762, offset 0, flags [DF], proto TCP (6), length 60)
172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xcc74 (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003252927 ecr 0,nop,wscale 7], length 0
11:19:11.098127 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x26fb), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003252927 ecr 3003252927,nop,wscale 7], length 0
11:19:12.095536 IP (tos 0x0, ttl 64, id 30763, offset 0, flags [DF], proto TCP (6), length 60)
172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xcb7a (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003253177 ecr 0,nop,wscale 7], length 0
11:19:12.095613 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x2601), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253177 ecr 3003252927,nop,wscale 7], length 0
11:19:13.303543 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x24d3), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253479 ecr 3003252927,nop,wscale 7], length 0
11:19:14.099543 IP (tos 0x0, ttl 64, id 30764, offset 0, flags [DF], proto TCP (6), length 60)
172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xc985 (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003253678 ecr 0,nop,wscale 7], length 0
11:19:14.099625 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x240c), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253678 ecr 3003252927,nop,wscale 7], length 0
11:19:15.903555 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51607: Flags [S.], cksum 0x8266 (incorrect -> 0x0eeb), seq 1342518319, ack 2198677968, win 28960, options [mss 1460,sackOK,TS val 3003254129 ecr 3003244349,nop,wscale 7], length 0
Seems like the docker container sends packets that are dropped because of bad checksums (?). We tried to disable any kind of offloading on the vethc356415:
ethtool -K vethc356415 gso off lro off tso off
ethtool --offload vethc356415 tx off rx off
with no results. Right now we are stuck. Is this an issue or a configuration problem ?
gndgnd
All 3 comments
Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues
BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
ENEEDMOREINFO
GordonTheTurtle
on 8 Oct 2015
The issue persisted after upgrade to docker 1.8.2. Issue resolved by changing the host firewall settings.
gndgnd
on 9 Oct 2015
Can you please provide some info on the firewall settings that needed to be changed to resolve this
surild
on 29 Oct 2015
Related issues
kunyan
路
3Comments
noahwhite
路
3Comments
gg7
路
3Comments
cmingxu
路
3Comments
programmerq
路
3Comments
Most helpful comment
Can you please provide some info on the firewall settings that needed to be changed to resolve this