Che: Cannot wget/curl/git/apt-get to internet from within docker container

Created on 30 Jul 2016  Â·  14Comments  Â·  Source: eclipse/che

Cannot wget/curl/git/apt-get to internet from within docker container

Reproduction Steps:

  1. Create a workspace with custom stack. In the dockerfile, there is a "sudo apt-get update" statement
  2. From the console log, I can observe that the request stops at [waiting header]

Expected behavior:
Able to do sudo apt-get update and install software.

Observed behavior:
I can observe from the console log that the apt-get update stops at [waiting header].

I troubleshoot the issue with the following steps. Confirm that able to ping but when doing wget/curl it seems to be able to connect but not getting response.

ubuntu@dev14-04:~$ docker run -itd --name=container1 busybox
545bf50a46d798602e21d04b177ab9500c48af5c8b37b2773204067631507211
ubuntu@dev14-04:~$ docker attach container1
/ # ping -w3 www.google.com
PING www.google.com (202.75.147.25): 56 data bytes
64 bytes from 202.75.147.25: seq=0 ttl=53 time=24.949 ms
64 bytes from 202.75.147.25: seq=1 ttl=53 time=22.862 ms
64 bytes from 202.75.147.25: seq=2 ttl=53 time=22.751 ms

--- www.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 22.751/23.520/24.949 ms
/ # wget http://www.eclipse.org/che/
Connecting to www.eclipse.org (198.41.30.198:80)

In another container with curl 

user@11611da3d377:/projects$ curl -O https://github.com/lhwong/datasciencecoursera/blob/master/HelloWorld.md
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0

I don't have the same issue when running on Windows.

Che version: 14.5.1
OS and version: Ubuntu 16.04
Docker version: 1.12.0
Che install: Docker container, server (zip)

Additional information:

  • Problem started happening recently, didn't happen in an older version of Che: No
  • Problem can be reliably reproduced, doesn't happen randomly: Yes

The problem persists when firewall is disabled.

In container:

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:03
          inet addr:172.17.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:63 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8423 (8.2 KiB)  TX bytes:5336 (5.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The network bridge seems OK:

ubuntu@dev14-04:~$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "6e86e16d0e75de89442fc4dc672928c35cdd1e66d886f8eacb61beb0361f6006"                                                                                                             ,
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "11611da3d3772e15c528d58170bc515f605b33393e4e506f304841fdc1893d08":                                                                                                              {
                "Name": "ubuntu",
                "EndpointID": "7cf5e799e3a2bade5c8b6632cf30f3c7c499d5a60abf6c946                                                                                                             dd2601bdeec3f7b",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "129d1d5af39a1f5d0a823f91d89ae41b48a2ca240b2190616d36dd75c217cd27":                                                                                                              {
                "Name": "container1",
                "EndpointID": "560a488340a965f158809feccbd7343ebe3a80ddcd2e06419                                                                                                             23171322407b1c4",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "545bf50a46d798602e21d04b177ab9500c48af5c8b37b2773204067631507211":                                                                                                              {
                "Name": "container2",
                "EndpointID": "146d32a022c46daebc5cbf286ae29dac47ca615fcc4a11a2b                                                                                                             b405a0b9317ad70",
                "MacAddress": "02:42:ac:11:00:04",
                "IPv4Address": "172.17.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
kinquestion
Source
picture lhwong

Most helpful comment

@lhwong yes, unfortunately, Eugene is correct here. There is something that is preventing your Docker containers from communicating with the host networking appropriately. Do you know if there are any non-standard docker-engine configuration that has been done to this system? You may want to consider adding --net=host to the way that you launched the container.

But unless we can diagnose the root cause of the docker issues - then che is going to be subject to the same issues.

picture TylerJewell on 1 Aug 2016
👍2

All 14 comments

@garagatyi @riuvshin - any ideas?

@lhwong - could this be a DNS issue from within the container? If you try to curl from an IP address any different address? And to confirm - you are not behind any sort of proxy?

TylerJewell picture TylerJewell on 31 Jul 2016

docker run -ti codenvy/ubuntu_jdk8 bash
sudo apt-get update

Try running it manually. I just checked a few base images and found no issues with connectivity.

ghost picture ghost on 31 Jul 2016

No issue with "ping -w3 www.google.com", hence, I eliminate the issue of DNS.

Tried the following

ubuntu@ws:~$ docker run -ti codenvy/ubuntu_jdk8 /bin/bash
user@ca2aee00d5e5:/projects$ sudo apt-get update
0% [Waiting for headers] [Waiting for headers]

Nothing happened after that

Able to do wget from the web server running on host running docker (192.168.1.165) 

user@ca2aee00d5e5:/projects$ sudo wget http://192.168.1.165:8080
--2016-08-01 04:21:42--  http://192.168.1.165:8080/
Connecting to 192.168.1.165:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5 [text/html]
Saving to: ‘index.html’

index.html          100%[===================>]       5  --.-KB/s    in 0s

2016-08-01 04:21:42 (1.06 MB/s) - ‘index.html’ saved [5/5]

but not able to do wget from any address from Internet

user@ca2aee00d5e5:/projects$ wget https://github.com/lhwong/datasciencecoursera/blob/master/HelloWorld.md
--2016-08-01 04:28:37--  https://github.com/lhwong/datasciencecoursera/blob/master/HelloWorld.md
Resolving github.com (github.com)... 192.30.253.113
Connecting to github.com (github.com)|192.30.253.113|:443... connected.

user@ca2aee00d5e5:/projects$ sudo wget http://www.google.com                   
--2016-08-01 04:30:57--  http://www.google.com/
Resolving www.google.com (www.google.com)... 202.75.147.53, 202.75.147.18, 202.75.147.38, ...
Connecting to www.google.com (www.google.com)|202.75.147.53|:80... connected.
HTTP request sent, awaiting response...

I have confirmed that no HTTP_PROXY is set on environment variable and /etc/wgetrc.

lhwong picture lhwong on 1 Aug 2016

@lhwong this is a networking issue on your side - Che is out of equation here. You may try an official Ubuntu image:

docker run -ti ubuntu bash
apt-get update

Have you configured Docker to use proxy?

ghost picture ghost on 1 Aug 2016

There isn't any proxy server in my network and HTTP_PROXY env is not set. From the host, I don't have any problem in running apt-get update.

It seems to be issue of the version of the ubuntu container?

ubuntu@ws:~$ docker run -ti ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
43db9dbdcb30: Already exists
2dc64e8f8d4f: Already exists
670a583e1b50: Already exists
183b0bfcd10e: Already exists
Digest: sha256:c6674c44c6439673bf56536c1a15916639c47ea04c3d6296c5df938add67b54b
Status: Downloaded newer image for ubuntu:latest
root@9f681f26f5a9:/# apt-get update
0% [Waiting for headers]

Nothing happens after this

ubuntu@ws:~$ docker run -ti ubuntu:14.04 /bin/bash
root@d9dd5b1c102b:/# sudo apt-get update
Ign http://archive.ubuntu.com trusty InRelease
12% [Waiting for headers]

Nothing happens after that.

ubuntu@ws:~$ docker run -ti 32bit/ubuntu:14.04 /bin/bash
...
root@35a6986323dd:/# sudo apt-get update
Ign http://extras.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://extras.ubuntu.com trusty Release.gpg [72 B]
Hit http://extras.ubuntu.com trusty Release
Ign http://extras.ubuntu.com trusty Release
Ign http://extras.ubuntu.com trusty/main i386 Packages/DiffIndex
Hit http://extras.ubuntu.com trusty/main i386 Packages
Ign http://extras.ubuntu.com trusty/main Translation-en
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Get:2 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://archive.ubuntu.com trusty-backports InRelease
Ign http://security.ubuntu.com trusty-security Release
Get:3 http://archive.ubuntu.com trusty Release.gpg [933 B]
Ign http://security.ubuntu.com trusty-security/main i386 Packages/DiffIndex
Get:4 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Ign http://security.ubuntu.com trusty-security/restricted i386 Packages/DiffIndex
Get:5 http://archive.ubuntu.com trusty-backports Release.gpg [933 B]
Ign http://security.ubuntu.com trusty-security/universe i386 Packages/DiffIndex
Hit http://archive.ubuntu.com trusty Release
Ign http://security.ubuntu.com trusty-security/multiverse i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates Release
Ign http://archive.ubuntu.com trusty-backports Release
Hit http://archive.ubuntu.com trusty/main i386 Packages
Hit http://archive.ubuntu.com trusty/restricted i386 Packages
Hit http://archive.ubuntu.com trusty/universe i386 Packages
Hit http://archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://archive.ubuntu.com trusty/main Translation-en
Hit http://archive.ubuntu.com trusty/multiverse Translation-en
Hit http://archive.ubuntu.com trusty/restricted Translation-en
Hit http://archive.ubuntu.com trusty/universe Translation-en
Ign http://archive.ubuntu.com trusty-updates/main i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates/restricted i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates/universe i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates/multiverse i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/main i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/restricted i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/universe i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/multiverse i386 Packages/DiffIndex
Get:6 http://archive.ubuntu.com trusty-backports/restricted Translation-en [28 B]
Get:7 http://archive.ubuntu.com trusty-backports/restricted i386 Packages [28 B]
Ign http://security.ubuntu.com trusty-security/main Translation-en
Ign http://security.ubuntu.com trusty-security/multiverse Translation-en
Ign http://security.ubuntu.com trusty-security/restricted Translation-en
Ign http://security.ubuntu.com trusty-security/universe Translation-en
Err http://security.ubuntu.com trusty-security/main i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Err http://security.ubuntu.com trusty-security/restricted i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Err http://security.ubuntu.com trusty-security/universe i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Err http://security.ubuntu.com trusty-security/multiverse i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Ign http://archive.ubuntu.com trusty-updates/main Translation-en
Ign http://archive.ubuntu.com trusty-updates/multiverse Translation-en
Ign http://archive.ubuntu.com trusty-updates/restricted Translation-en
Ign http://archive.ubuntu.com trusty-updates/universe Translation-en
100% [Waiting for headers]

Takes long time to update. More than 2 hours still has not complete on a fast line.

lhwong picture lhwong on 1 Aug 2016

@lhwong this is a local Docker setup/connectivity issue. Right now we're debugging your local system and official ubuntu images, but not Che.

ghost picture ghost on 1 Aug 2016

@lhwong yes, unfortunately, Eugene is correct here. There is something that is preventing your Docker containers from communicating with the host networking appropriately. Do you know if there are any non-standard docker-engine configuration that has been done to this system? You may want to consider adding --net=host to the way that you launched the container.

But unless we can diagnose the root cause of the docker issues - then che is going to be subject to the same issues.

TylerJewell picture TylerJewell on 1 Aug 2016
👍2

@TylerJewell yes. Adding --net=host option solves the connectivity issue, however, I can't find the way to create container the option in che or use a pre-created container as workspace. Is there a way to configure che to create container with the option?

lhwong picture lhwong on 2 Aug 2016

@lhwong - can you provide the syntax that you are using to start / stop Che? We are launching a much simpler syntax this week which has this option built into it. If you are not running Che this way, then we shoud get you on this tactic first.
https://eclipse-che.readme.io/docs/usage-docker

TylerJewell picture TylerJewell on 2 Aug 2016

@TylerJewell I am using che 4.5.1 and running che on Docker-native system. I start che with "./che.sh --remote:192.168.1.150 run".

Do I need to run che in a Docker container in order to configure --net=host?

lhwong picture lhwong on 2 Aug 2016

Please just use the syntax I linked in the previous post. Just type it exactly as it is provided. Should be ok.

TylerJewell picture TylerJewell on 2 Aug 2016

[DOCKER] Ign:1 http://archive.ubuntu.com/ubuntu xenial InRelease
[DOCKER] Ign:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Take really long time for each of the above. I believe still the similar issue.

When running ifconfig within the container created for the workspace launched, I get the following

ubuntu@ws:~$ docker exec -it 841cc4279486 /bin/bash
bash-4.3# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:03
          inet addr:172.17.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:3%32714/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3009 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3303 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:219795 (214.6 KiB)  TX bytes:6088135 (5.8 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1%32714/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

If I run a container with --net=host, I get

ubuntu@ws:~$ docker run -it --net=host busybox
/ # ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:F6:06:20:22
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:f6ff:fe06:2022/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4180 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3596 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6123327 (5.8 MiB)  TX bytes:333721 (325.8 KiB)

ens3      Link encap:Ethernet  HWaddr FA:16:3E:7A:02:A8
          inet addr:10.0.0.15  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe7a:2a8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1454  Metric:1
          RX packets:244136 errors:0 dropped:0 overruns:0 frame:0
          TX packets:125382 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:346661009 (330.6 MiB)  TX bytes:14721880 (14.0 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:164 errors:0 dropped:0 overruns:0 frame:0
          TX packets:164 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:12088 (11.8 KiB)  TX bytes:12088 (11.8 KiB)

veth3523fb7 Link encap:Ethernet  HWaddr 9E:EB:A4:27:3C:DD
          inet6 addr: fe80::9ceb:a4ff:fe27:3cdd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3509 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3174 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6103250 (5.8 MiB)  TX bytes:242913 (237.2 KiB)

veth7b80411 Link encap:Ethernet  HWaddr 46:66:6A:7F:3C:AB
          inet6 addr: fe80::4466:6aff:fe7f:3cab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:50 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4578 (4.4 KiB)  TX bytes:2940 (2.8 KiB)

Hence, I believe the --net=host option (the host networking) is not being applied. Any idea?

lhwong picture lhwong on 4 Aug 2016

@lhwong maybe you should look at iptables and uwf? I believe if you post this to Docker issues you will get a way faster and helpful response.

ghost picture ghost on 4 Aug 2016

iptables and ufw look good. I believe this is issue of connectivity of our network here. Thanks for your help.

lhwong picture lhwong on 11 Aug 2016
Was this page helpful?
0 / 5 - 0 ratings

Related issues

amisevsk picture amisevsk  Â·  3Comments
sudheerherle picture sudheerherle  Â·  3Comments
apupier picture apupier  Â·  3Comments
LaneGeek picture LaneGeek  Â·  3Comments
redeagle84 picture redeagle84  Â·  3Comments