Nvidia-docker: Build fails behind a HTTP proxy

[toships]

OS and docker version below:

toships@server-05:~/Documents/nvidia-docker$ uname -a
Linux server-05 3.16.0-52-generic #71~14.04.1-Ubuntu SMP Fri Oct 23 17:24:53 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
toships@server-05:~/Documents/nvidia-docker$ docker -v
Docker version 1.9.1, build a34a1d5
toships@server-05:~/Documents/nvidia-docker$

Error on install below :

toships@server-05:~/Documents/nvidia-docker$ sudo make install
make -C /home/toships/Documents/nvidia-docker/tools install
make[1]: Entering directory '/home/toships/Documents/nvidia-docker/tools'
Sending build context to Docker daemon 96.77 kB
Step 1 : FROM golang
---> cd6e9b146853
Step 2 : ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c
---> Using cache
---> ab2714b30180
Step 3 : ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be
---> Using cache
---> 1de66c070c4d
Step 4 : ENV NVIDIA_GDK_SUM 1e32e58f69fe29ee67b845233e7aa9347f37994463252bccbc8bfc8a7104ab5a
---> Using cache
---> 697201ceaa3d
Step 5 : RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY && apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub && echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list
---> Using cache
---> cabc7dd1abe4
Step 6 : RUN apt-get update && apt-get install -y --no-install-recommends --force-yes cuda-cudart-dev-6-5=6.5-19 cuda-misc-headers-6-5=6.5-19 && rm -rf /var/lib/apt/lists/*
---> Using cache
---> 784c8f082087
Step 7 : RUN objcopy --redefine-sym memcpy=memcpy@GLIBC_2.2.5 /usr/local/cuda-6.5/lib64/libcudart_static.a
---> Using cache
---> 717306ff1350
Step 8 : RUN wget -O gdk.run -q http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_352_39_gdk_linux.run && echo "$NVIDIA_GDK_SUM gdk.run" | sha256sum -c --strict - && chmod +x gdk.run && ./gdk.run --silent && rm gdk.run
---> Using cache
---> 3535f3a748da
Step 9 : COPY src /go/src
---> Using cache
---> fff8b44018c0
Step 10 : VOLUME /go/bin
---> Using cache
---> f879f9376c93
Step 11 : ENV CGO_CFLAGS "-I /usr/local/cuda-6.5/include -I /usr/include/nvidia/gdk"
---> Using cache
---> 1c8fce5a08a4
Step 12 : ENV CGO_LDFLAGS "-L /usr/local/cuda-6.5/lib64 -L /usr/src/gdk/nvml/lib -ldl -lrt"
---> Using cache
---> 8bdd291703ac
Step 13 : ARG UID
---> Using cache
---> 4f6319b1aa77
Step 14 : RUN useradd --non-unique --uid $UID build
---> Using cache
---> 6a24e18b8818
Step 15 : USER build
---> Using cache
---> 9d6085d78502
Step 16 : CMD go get -v -ldflags="-s" nvidia-docker nvidia-docker-plugin
---> Using cache
---> 4c0f4b8e2793
Successfully built 4c0f4b8e2793
Fetching https://golang.org/x/crypto/ssh?go-get=1
https fetch failed.
import "golang.org/x/crypto/ssh": https fetch: Get https://golang.org/x/crypto/ssh?go-get=1: dial tcp 216.58.192.17:443: i/o timeout
package golang.org/x/crypto/ssh: unrecognized import path "golang.org/x/crypto/ssh"
Fetching https://golang.org/x/crypto/ssh/agent?go-get=1
https fetch failed.
import "golang.org/x/crypto/ssh/agent": https fetch: Get https://golang.org/x/crypto/ssh/agent?go-get=1: dial tcp 216.58.192.49:443: i/o timeout
package golang.org/x/crypto/ssh/agent: unrecognized import path "golang.org/x/crypto/ssh/agent"
Fetching https://golang.org/x/crypto/ssh/terminal?go-get=1
https fetch failed.
import "golang.org/x/crypto/ssh/terminal": https fetch: Get https://golang.org/x/crypto/ssh/terminal?go-get=1: dial tcp 216.58.192.49:443: i/o timeout
package golang.org/x/crypto/ssh/terminal: unrecognized import path "golang.org/x/crypto/ssh/terminal"
github.com/justinas/alice (download)
# cd .; git clone https://github.com/justinas/alice /go/src/github.com/justinas/alice
Cloning into '/go/src/github.com/justinas/alice'...
fatal: unable to access 'https://github.com/justinas/alice/': Failed to connect to github.com port 443: Connection timed out
package github.com/justinas/alice: exit status 128
Fetching https://gopkg.in/tylerb/graceful.v1?go-get=1
https fetch failed.
import "gopkg.in/tylerb/graceful.v1": https fetch: Get https://gopkg.in/tylerb/graceful.v1?go-get=1: dial tcp 107.178.216.236:443: i/o timeout
package gopkg.in/tylerb/graceful.v1: unrecognized import path "gopkg.in/tylerb/graceful.v1"
make[1]: *** [build] Error 1
make[1]: Leaving directory '/home/toships/Documents/nvidia-docker/tools'
make: *** [install] Error 2

I checked "https://golang.org/x/crypto/ssh?go-get=1" and I get a "nothing to see here" message. Is this an issue or am I doing something wrong ? Please help.

[flx42]

Are you getting this error consistently? I just tried on multiple machines and it works fine for me.
Looks like a (temporary?) network issue.

[3XX0]

Yeah looks like your HTTPS traffic is being dropped.

[toships]

ohh thanks .. will look into this and report back .. thanks for quick reply ..

[toships]

Hi, I tried "wget https://www.google.com" and it works out fine .. is there any other way to test this and make sure this is the issue before I go to my network admin ?

[flx42]

Can you successfully git clone projects on GitHub?
To reuse the same example as above:

$ git clone https://github.com/justinas/alice.git

[toships]

yes - just tried it

[toships]

In fact I tried
wget "https://golang.org/x/crypto/ssh?go-get=1"
and got a response - which is also nothing to see here - what I got from the browser ... see below

Proxy request sent, awaiting response... 200 OK
Length: 604 [text/html]
Saving to: ‘ssh?go-get=1.1’

100%[======================================>] 604 --.-K/s in 0s

2016-01-27 11:25:40 (94.3 MB/s) - ‘ssh?go-get=1.1’ saved [604/604]

[3XX0]

Weird, are you behind a proxy ?

[toships]

yes - but the proxy seems to be working fine atleast for the browser and wget on command line ..

[toships]

Kind of digging more into it - this seems to be fetching due to import in remote.go

tools/src/nvidia-docker/remote.go: "golang.org/x/crypto/ssh"
tools/src/nvidia-docker/remote.go: "golang.org/x/crypto/ssh/agent"
tools/src/nvidia-docker/remote.go: "golang.org/x/crypto/ssh/terminal"

May be go does https fetch differently which is being blocked by my proxy? Will look and report but any help is appreciated.

[toships]

as in previous comment - relevant thread ?

https://github.com/termux/termux-packages/issues/7

[3XX0]

Maybe a problem with the Go user agent or http_proxy not being set?

[toships]

Now I can download from a standalone go command using https but the build still fails at the same place.

toships@server-05:~/Documents/nvidia-docker$ go get -d -v golang.org/x/crypto/ssh
Fetching https://golang.org/x/crypto/ssh?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh?go-get=1
get "golang.org/x/crypto/ssh": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
toships@server-05:~/Documents/nvidia-docker$ go get -d -v golang.org/x/crypto/ssh/agent
toships@server-05:~/Documents/nvidia-docker$ go get -d -v golang.org/x/crypto/ssh/terminal

[3XX0]

Ok so it's not a problem with the user-agent.
Do you have the http_proxy or https_proxy environment variable set on your machine ?
If so can you try adding -e http_proxy and -e https_proxy in the build command line

[toships]

Ok fixed the issue - thanks for all your help.

The -e option did not work for me. Came back with an error.
However when the Dockerfile.build file was modified with the bold lines it worked out

FROM golang

ENV http_proxy http://proxy.xxxxxx.com:80/
ENV https_proxy http://proxy.xxxxxx.com:80/

ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c

[3XX0]

Oh right -e is not used at build time, my bad.
We might need to add more build arguments to fix it generically.

[shinfo001]

Hi, In order to avoid a proxy of the problem, it is necessary to fix all of dockerfile.
We have to avoid this problem in the next code.

------ addproxy.sh --------

find ~/nvidia-docker/ -type f -name "Dockerfile*" | while read FILE
do
echo "modify Dockerfile - "${FILE}
sed -i -e "2i env http_proxy=$http_proxy" ${FILE}
sed -i -e "3i env https_proxy=$https_proxy" ${FILE}
done

---

$ sh addproxy.sh

[3XX0]

What about setting it up in /etc/default/docker?

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

[shinfo001]

As it might be expected,The default docker, if we use the proxy behind must have the following it configuration.

[bdalton]

There are three different _scopes_ for environment variables such as http_proxy:

1. In the host, the docker command uses it for things like pushing and pulling images and for searching. For Debian based hosts that use upstart (such as Ubuntu 14.04) this is commonly set in /etc/default/docker. For systemd-based Debian hosts this can be set in /etc/systemd/system/docker.service.d/http-proxy.conf. You can also set it on the commandline. The variable is not visible either when BUILDING or RUNNING a container.
2. In your Dockerfile you can set environment variables. These are visible when the container is RUNNING only. For example, add this to your dockerfile.

ENV http_proxy="http://172.17.42.1:3128"

1. Neither of these two methods are visible when the container is building. With docker 1.9 or newer, you can use --build-arg http_proxy=172.17.42.1:3128. This will set an environment variable when BUILDING only. You need to propagate this with an ENV line for it to be available when running.

Unfortunately, this isn't the only problem. apt-get does not respect the http_proxy or HTTP_PROXY environment variables. The easiest way to set a proxy is by creating /etc/apt/apt.conf.d/30proxy with the contents:

Acquire::http::Proxy "http://172.17.42.1:3128";

Note: 172.17.42.1 reliably points to tho docker host. You can run a forwarding proxy there like cntlm or squid that has your credentials to your corporate proxy. This is undocumented and could potentially change in the future. That is the state of using docker behind a corporate proxy though.

[3XX0]

Closing since Docker now provides --network at build time and we have package repositories