Buildkit: Buildkit does not configure DNS correctly inside container

[abergmeier]

For a project, I enabled DOCKER_BUILDKIT="1".
When building, I get wget: bad address 'my.company.com'.

The offending line is:

RUN wget http://my.company.com/foobar.txt

I am not really sure, why this does not work. Other lines like apk add --no-cache git work fine in contrast.

When experimenting, I get a similar error with:

RUN go mod download

And this works fine both on the host and when not using Buildkit.

[AkihiroSuda]

Docker version?

[tonistiigi]

Any custom dns config?

[abergmeier]

Any custom dns config?

Not that I know of.

[abergmeier]

Docker version?

docker --version
Docker version 18.09.5, build e8ff056

On Ubuntu Bionic

[abergmeier]

Looking at journalctl I see the following entries:

systemd-resolved[1024]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
systemd-resolved[1024]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
systemd-resolved[1024]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
dockerd[2070]: time="2019-05-14T14:24:12Z" level=error msg="fatal error: rpc error: code = Unknown desc = executor failed running [/bin/sh -c wget http://my.company.com/foobar.txt]: exit code: 1"
dockerd[2070]: panic: rpc error: code = Unknown desc = executor failed running [/bin/sh -c wget http://my.company.com/foobar.txt]: exit code: 1
dockerd[2070]: goroutine 1 [running]:
dockerd[2070]: main.main()
dockerd[2070]:         /src/frontend/dockerfile/cmd/dockerfile-frontend/main.go:26 +0x34f

[abergmeier]

cat /etc/resolv.conf on host prints:

nameserver 127.0.0.53
options edns0

so this might be related to https://github.com/docker/libnetwork/issues/1654!?

In the container it prints:

options edns0
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

which seems to miss the local and company DNS Servers.

[tonistiigi]

@abergmeier Just so that everything is clear, can you confirm that you do not have connectivity to 8.8.8.8 nameserver from this machine. In the output you posted about "In the container it prints:", is this from docker run or from RUN cat /etc/resolv.conf in Dockerfile. If not latter can you post that as well and can you confirm that you do have network connectivity from regular containers run with docker run.

[abergmeier]

from RUN cat /etc/resolv.conf in Dockerfile

That.

[tiborvass]

Apologies @abergmeier but we're trying to zero in this issue and I'm a bit confused right now.
For sake of completeness and being extra sure we're not missing anything, do you mind providing inline outputs for the following:

1. ls -l /etc/resolv.conf
2. systemctl status systemd-resolved.service
3. cat /etc/docker/daemon.json | grep -i dns
4. docker run -it --rm alpine cat /etc/resolv.conf
5. echo -e "from alpine\n run cat /etc/resolv.conf" | DOCKER_BUILDKIT=0 docker build --no-cache --progress plain
6. echo -e "from alpine\n run cat /etc/resolv.conf" | DOCKER_BUILDKIT=1 docker build --no-cache --progress plain
7. ps aux | grep dockerd

Thanks!

[abergmeier]

There you go:

1. Output: lrwxrwxrwx 1 root root /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
2. Output:

● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-05-15 09:02:10 CEST; 5min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 1000 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/systemd-resolved.service
           └─1000 /lib/systemd/systemd-resolved

Mai 15 09:02:46 s0121552 systemd-resolved[1000]: Using degraded feature set (UDP) for DNS server 8.8.4.4.
Mai 15 09:02:47 s0121552 systemd-resolved[1000]: Using degraded feature set (UDP) for DNS server 8.8.8.8.
Mai 15 09:02:52 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:02:56 s0121552 systemd-resolved[1000]: Using degraded feature set (TCP) for DNS server 8.8.8.8.
Mai 15 09:02:56 s0121552 systemd-resolved[1000]: Using degraded feature set (TCP) for DNS server 8.8.4.4.
Mai 15 09:03:03 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:03 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:03 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:19 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:19 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

1. Output: cat: /etc/docker/daemon.json: No such file or directory
2. Output:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

1. Output:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

1. Output:

options edns0

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

which seems pretty wrong, especially since our network does not route ipv6!

1. Output: root 2084 1.2 0.3 5676068 123720 ? Ssl 09:02 0:11 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

[abergmeier]

Sadly I cannot seem to override /etc/resolv.conf inside the container, so I cannot work around this :cry:

[thaJeztah]

searching for that error, I came to this page https://www.linode.com/community/questions/17384/error-server-returned-error-nxdomain-mitigating-potential-dns-violation-dve-2018

which indicates this may be a bug in systemd resolvd (just from a quick read)

[abergmeier]

which indicates this may be a bug in systemd resolvd (just from a quick read)

This is probably one factor. On the other hand, Buildkit IMO definitely has a bug where it handles resolv.conf different than legacy Docker (with the latter working fine).

[tiborvass]

@abergmeier Thanks, can you also provide the output for cat /run/systemd/resolve/resolv.conf please ?

[abergmeier]

cat /run/systemd/resolve/resolv.conf outputs:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

[tiborvass]

@abergmeier would you mind trying a dockerd binary I built from https://github.com/moby/moby/pull/39295 ?

Here's a link: https://drive.google.com/file/d/15hJWKtlEUpoqYi7uKsaO20qqtEREZ768/view?usp=sharing
The sha256 is c7ed7bb6011ce6fd3270d69d528a01a18060a30074028ca1dca334299d3b2c6f.

[abergmeier]

Seems to fix the problem :tada:

1. Output:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

[varlucian]

Will the issue be backported to 18.06?
If not, what is the workaround?

[thaJeztah]

18.06 is EOL, so it won't be backported to that version

[varlucian]

Sorry, I meant 18.09 not 18.06