Weave: Add option to specify weave DNS upstream servers

[timcharper]

Presently, weave upstream DNS is configurable only via the hosts /etc/resolv.conf. I'd like to configure weave to be the host's DNS server. Doing so now results in an infinite loop.

As a workaround for not having this option, I've created a systemd script which moves the system /etc/resolv.conf out of the way, and puts in the desired weave configuration, while weave is starting. Then, once started, it puts the files back to its original form.

I found the code which is reading the configuration here:

https://github.com/weaveworks/weave/blob/v1.4.6/nameserver/dns.go#L62

[bboreham]

WeaveDNS is coded to ignore its own address as an upstream server, to break the infinite loop. This is at line 66 of the file you linked. Can you share some details of exactly how you saw this problem?

[timcharper]

Thanks, @bboreham, that helps. My configuration is more complex than I initially described. /etc/resolv.conf actually points the system to use mesos-DNS, which then forwards to weave as an upstream server.

With the information you've revealed to me, I believe the loop is happening because weave-DNS will filter out itself at 172.17.0.1, but won't filter out the host IP. Infinite loop diagnosis is based on the following output:

# dig youtube.com

; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> youtube.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63027
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;youtube.com.           IN  A

;; Query time: 281 msec
;; SERVER: 10.141.141.11#53(10.141.141.11)
;; WHEN: Wed Apr 13 15:54:37 UTC 2016
;; MSG SIZE  rcvd: 29

My issue could be resolved, likely, if I didn't have mesos-DNS forward to weave-DNS, but instead listed them as siblings in /etc/resolv.conf. However, "happens to work" is not the same as "will always work". It is problematic to put heterogeneously-responding DNS servers (as in, they don't respond the same) in /etc/resolv.conf because the behavior for how libc uses multiple nameservers is fundamentally unspecified, and implementations differ between libc implementations. For example, glibc will resolve them in order, however musl-libc resolves them in parallel and takes the first response (even if that response is no result). This behavioral difference can be observed by comparing the output of dig and drill while heterogeneous nameservers are specified in /etc/resolv.conf, where dig will give a consistent response, but drill will give a different response between runs (based on what I presume to be the DNS server that wins the race).

Because libc behavior for multiple nameservers is undefined, I believe I still need weave DNS to use a different set of nameservers than those specified in the host /etc/resolv.conf, and would still stand to benefit from the option to specify it.

[bboreham]

As a work-around, can you try changing your launch command to:

weave launch-router --dns-effective-listen-address <mesos-DNS IP> [args as before]

That should filter out the Mesos DNS instead of Weave DNS.

[timcharper]

Thanks @bboreham, this definitely solves the issue of infinite loop, but it does not solve the issues of heterogeneously listed servers in /etc/resolv.conf.

To explain in an example, given the config:

/etc/resolv.conf

search mesos
nameserver 10.141.141.11

/etc/mesos-dns.conf:

  ...
  "resolvers": [
    "172.17.0.1",
    "8.8.8.8",
    "8.8.4.4"
  ],
  ...

gives me the following resolution pathway:

 /---------\
( DNS query )
 \_________/
      |
      V
+-----------+
| mesos-dns |
+-----------+
      |
      |--------------+-------------+
  (1) |          (2) |         (3) |
      V              V             V
+-----------+   +---------+   +---------+
| weave-dns |   | 8.8.8.8 |   | 8.8.4.4 |
+-----------+   +---------+   +---------+

mesos-dns will reliably resolve in order; however, if weave-dns returns 0 results, then mesos-dns does not continue to try upstream nameservers, and 0 results is returned as the answer.

The resolution pathway I need is as follows:

         /---------\
        ( DNS query )
         \_________/
              |
              V
        +-----------+
        | mesos-dns |
        +-----------+
              |
              V        
        +-----------+  
        | weave-dns |  
        +-----------+  
              |
              |
       +------+------+
       |             |
       V             V
  +---------+   +---------+
  | 8.8.8.8 |   | 8.8.4.4 |
  +---------+   +---------+

[rade]

why not swap weave-dns and mesos-dns?

[timcharper]

I'd thought of that; since weave-dns gets it's upstream DNS servers from /etc/resolv.conf, I would need an /etc/resolv.conf that looks like this:

search mesos
; weave-dns
nameserver 172.17.0.1
; mesos-dns
nameserver 10.141.141.11

and a resolution pathway like this:

 /---------\
( DNS query )
 \_________/
      |
      +--------------+
      |              |
      V              |
+-----------+        |
| weave-dns |        |
+-----+-----+        |
      |              |
      +--------------+
                     |
                     V
               +-----------+
               | mesos-dns |
               +-----------+
                     |
              +------+------+
          (1) |         (2) |
              V             V
         +---------+   +---------+
         | 8.8.8.8 |   | 8.8.4.4 |
         +---------+   +---------+

glibc would handle this well, but musl-libc would not.

[rade]

ok. I guess the phrase is "patches welcome" :)

FWIW, we used to have configurable upstreams in a previous incarnation of weaveDNS.

[timcharper]

I noticed that, @rade ; Understood WRT to patches welcome. I'm surprised no-one else has raised an issue since removing the feature, but I sense I may be a bit of an outlier in wanting to use mesos-dns and weave-dns. I have reasons, but people are probably solving those problems different ways.

I agree that patches welcome is an appropriate response. Thanks for the discussion. If others chime in on this issue, and report similar issues using different libc's, then I'll take a stab at resolving it.

[timcharper]

(actually... now that I think about it, it's problematic even if you don't want to compose two dns servers. The practice of putting heterogeneous DNS servers in /etc/resolv.conf is unsanitary and should be avoided, and in it's current incarnation this is precisely what weave pushes you to do if you want to use weave-dns as a resolver on the host machine).

But, I say this out of clarity, not in anger. I've a work-around so I'll wait until others complain about this to move on putting together a patch. I may be the only one who cares.

[rade]

if you want to use weave-dns as a resolver on the host machine

That is not a very typical use of weaveDNS. But we know there are users with that sort of setup, so I wonder what they are doing. Perhaps the Docker daemon's dns options can be used to control the upstreams of weaveDNS?

We _will_ address this issue ourselves eventually, but that may take a while. Hence the "patches welcome".

[rade]

Perhaps the Docker daemon's dns options can be used to control the upstreams of weaveDNS?

I think that may have been possible only prior to use moving the router into the host netns; looks like these days it is seeing the host's resolv.conf.

[rade]

also, prior to the host netns switch, WEAVE_DOCKER_ARGS="--dns ..." weave launch would have worked.