Docker-transmission-openvpn: DNS issues since last build

[h45rd]

Ahoy!

After being away from dockering for a while I have spent the last hour trying to debug why my torrent container is not able to resolve DNS anymore since watchtower updated to the last dev build:

RESOLVE: Cannot resolve host address: my.vpn1.server:8888 (Try again)
RESOLVE: Cannot resolve host address: my.vpn2.server:8888 (Try again)
Could not determine IPv4/IPv6 protocol
SIGUSR1[soft,init_instance] received, process restarting

No configuration on host or container was changed in the meantime. Using an external DNS server via the _--dns_ parameter does not alleviate the issue.

Verified via interactive shell from within the container that indeed DNS resolution via the internal docker DNS server (as I am using user-based networking) is failing.
The internal DNS is reachable from within the container.

After a lot of trial and error I noticed that DNS resolution works if I create the container without the _NET_ADMIN caps_. Of course this is not a workable solution..
Other containers with the same basic layout (user networking, NET_ADMIN caps, etc.) still work fine:

docker run --rm -it --network=mycustomnet --cap-add=NET_ADMIN ubuntu:16.04 /bin/bash
apt update && apt install dnsutils -y
dig google.com
google.com.             190     IN      A       172.217.19.206
;; Query time: 18 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)

Looking at the commit history and the build logs I cannot for the life of me figure out what changed between these two pushes that would cause these issues:

time="2018-07-31T14:55:03Z" level=info msg="Found new haugene/transmission-openvpn:dev image (sha256:96d1574f925b754ff7386000cf5897dd45eb99a391e98e2742485a973432864b)"
time="2018-08-06T13:09:47Z" level=info msg="Found new haugene/transmission-openvpn:dev image (sha256:641840c65bc5764ec8f8ae73710ac1c9944032299c5ea9fc679b496c3d19cee6)" 

Most likely an updated package in the Ubuntu repository. But which one? And why am I apparently the only one affected by this issue?
Any advice / idea how to further debug this? At my wits' end for now and rolled back to using 2.0 for the time being.

Best,
h45rd

[mnkhouri]

@h45rd, I'm also experiencing the same issue, so you're not the only one affected!

RESOLVE: Cannot resolve host address: [address trimmed] (Temporary failure in name resolution)
RESOLVE: Cannot resolve host address: [address trimmed] (Temporary failure in name resolution)
Could not determine IPv4/IPv6 protocol
SIGUSR1[soft,init_instance] received, process restarting

It's interesting that our RESOLVE messages are slightly different in the last parentheses -- not sure if that has any significance. Happy to help debug, but @h45rd has already done everything I can think of :)

Edit: My Docker host is _MacOS 10.13.6 (x86_64)_

[h45rd]

Sorry to say but good to know that the issue is not limited to my specific setup :)

Still.. given the seemingly very limited group of people affected by this issue, something in our host and/or container configuration seems to be rather special.
Maybe we should compare similarities in order to get closer to the problem.
Off the top of your head: do you use any "_special_" docker-related feature/setting/etc. that most regular users probably are not using?

All I can think of in my case right now would be user-defined networking.
My current docker host is on _4.4.0-131-generic # 157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux_

There is one other debugging possibility I can think of: dump, sort and diff the package index of dev 2018-07-31 vs dev 2018-08-06T13 and go from there. If nothing obvious pops up, up/downgrade single packages until the culprit is found.
Won't be getting around to that until the weekend, though.

[ties]

Still.. given the seemingly very limited group of people affected by this issue, something in our host and/or container configuration seems to be rather special.

I ran into this issue as well. The only "custom" part of my config is that the container is not on the default network, but instead is in a custom network containing only the container itself and a reverse proxy server (connected to multiple networks). LOCAL_NETWORK is set and ENABLE_UFW=true.

My current docker host is on 4.17.0-1-amd64 #1 SMP Debian 4.17.8-1 (2018-07-20) x86_64 GNU/Linux

[IvanGolovko3k]

I ran into this issue too. I'm using docker-compose with the docker-compose-armhf.yml. I modified/added a couple of environment variables in the transmission section (see below), everything else is default. I want to run this on a Raspberry Pi 2 with Raspbian 9 (Stretch). I cloned the git repo yesterday and ran docker-compose. The images have been build and all three containers are starting up.

RESOLVE: Cannot resolve host address: my.provider.com:1198 (Temporary failure in name resolution)
RESOLVE: Cannot resolve host address: my.provider.com:1198 (Temporary failure in name resolution)
Could not determine IPv4/IPv6 protocol
SIGUSR1[soft,init_instance] received, process restarting

docker-compose Version 1.22.0, build f46880f
Docker Version: 18.03.1-ce
Linux 4.14.54-v7+ #1126 SMP Wed Jul 11 20:01:03 BST 2018 armv7l GNU/Linux

docker-compose-armhf.yml modifications:

services:
 transmission:
  environment:
    - OPENVPN_PROVIDER=MyProvider
    - OPENVPN_CONFIG=MyProvierConfig
    - OPENVPN_USERNAME=myusername
    - OPENVPN_PASSWORD=mypassword
    - OPENVPN_OPTS=--inactive 3600 --ping 10 --ping-exit 60
    - WEBPROXY_ENABLED=true
    - LOCAL_NETWORK=192.168.0.0/24
    - ENABLE_UFW=true
    - FW_ALLOW_GW_NET=true
    - TRANSMISSION_DHT_ENABLED=false
    - TRANSMISSION_UTP_ENABLED=false
    - TRANSMISSION_PEX_ENABLED=false
    - DROP_DEFAULT_ROUTE=true

Update 1:
I rebuild the images and added iputil-ping to the docker-tranmission-openvpn image for debugging. Pinging 8.8.8.8 and 8.8.4.4 works.

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=10.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=122 time=8.90 ms

Name resolution is still not working.

ping www.google.com
ping: www.google.com: Temporary failure in name resolution

cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0

As mentioed in the following article 127.0.0.11 in the internal docker dns server.
https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/

[mnkhouri]

I think the reason most people aren't seeing this is that the issue is only present with the ENABLE_UFW option.

Here is a minimal docker-compose.yml to reproduce the issue:

version: '2'
services:
    broken:
        image: haugene/transmission-openvpn:2.1
        devices:
            - '/dev/net/tun:/dev/net/tun'
        volumes:
            - '~/temp/test:/data'
        ports:
            - '9094:9091'
        cap_add:
            - NET_ADMIN
        environment:
            - OPENVPN_PROVIDER=PIA
            - OPENVPN_USERNAME=$PIAUSER
            - OPENVPN_PASSWORD=$PIAPASS
            - 'OPENVPN_CONFIG=CA Toronto'
            - ENABLE_UFW=true  # This line breaks v2.1

    working-new:
        image: haugene/transmission-openvpn:2.1
        devices:
            - '/dev/net/tun:/dev/net/tun'
        volumes:
            - '~/temp/test:/data'
        ports:
            - '9095:9091'
        cap_add:
            - NET_ADMIN
        environment:
            - OPENVPN_PROVIDER=PIA
            - OPENVPN_USERNAME=$PIAUSER
            - OPENVPN_PASSWORD=$PIAPASS
            - 'OPENVPN_CONFIG=CA Toronto'
            # - ENABLE_UFW=true # This line breaks v2.1

    working-old:
        image: haugene/transmission-openvpn:2.0 # older version
        devices:
            - '/dev/net/tun:/dev/net/tun'
        volumes:
            - '~/temp/test:/data'
        ports:
            - '9096:9091'
        cap_add:
            - NET_ADMIN
        environment:
            - OPENVPN_PROVIDER=PIA
            - OPENVPN_USERNAME=$PIAUSER
            - OPENVPN_PASSWORD=$PIAPASS
            - 'OPENVPN_CONFIG=CA Toronto'
            - ENABLE_UFW=true  # This line works in v2.0

[haugene]

Hey guys, and thanks for the help with debugging.

I can confirm that I'm seeing the same, no DNS when ENABLE_UFW is true. That made me think of this change as the possible culprit: https://github.com/haugene/docker-transmission-openvpn/pull/559/files/f2a18996493dbbb71a45c73083e56a84b7a9a89c#diff-4f1dc7404e52989eadee99ae6dea355aR106

Also commented in that PR that this possibly should only be done when the special IPTABLES_REJECT parameter is set. Anyways, moved the sed line into the if block of the reject option and that seemed to fix it for me.

Pushing the fix to dev branch for now. @sillyfrog Can you confirm that your Synology fix is working and that you don't have any DNS issues because of it? If not we should maybe remove the line altogether as you mentioned wouldn't interfere with your fix.

Let's verify this on the dev branch and I'll do a merge to master when it's confirmed :)

[haugene]

Dev is building... https://hub.docker.com/r/haugene/transmission-openvpn/builds/

[h45rd]

Arg.. at first suspected exactly that commit but the NET_ADMIN observation threw me off course..

Can confirm that DNS resolution is working again with the current dev branch and ENABLE_UFW set to true.
Thanks haugene and the other contributors to this thread :+1:

[IvanGolovko3k]

I can confirm that the issue is resolved with a fresh build image from the dev branch on Raspbian.

Thank you!!

[sillyfrog]

That line is not doing what it should be... (not the first thing I have found like this with ufw), so I think that completely removing the sed -i -e s/MANAGE_BUILTINS=no/MANAGE_BUILTINS=yes/ /etc/default/ufw line is the way to go (I have tested this just now, and it's OK for the Synology). The idea is it should be keeping the firewall clean and removing all rules - but apparently that's not working as planned.

Did you want me to submit a PR, or are you OK to include it in this one?

Cheers.

[haugene]

@sillyfrog No worries, removed it now. Merging to master and doing a new release as well.

[haugene]

Merged to master and included in the new 2.2 tag as well as lastest and dev tags. Closing this then...