Caddy: ipv6: does not bind on ipv4 and ipv6 for sites that resolve to both

[wmark]

1. What version of Caddy are you running (caddy -version)?

0.8.3

2. What are you trying to do?

Serve a site that can be accessed using _ipv4_ and _ipv6_.

3. What is your entire Caddyfile?

http://ip6.nl:80 {
  bind ip6.nl
  root /var/www
}

4. How did you run Caddy (give the full command and describe the execution environment)?

ip6.nl resolves to 193.200.132.187 as well as [2a02:2308:10::c:19].

The server has both addresses configured (output of ip addr confirms this).

5. What did you expect to see?

$ netstat -lnp --inet6 | grep -F caddy

tcp6  0  0  2a02:2308:10::c:19:80  :::*  LISTEN  8888/caddy

6. What did you see instead (give full error messages and/or log)?

Caddy does not listen on tcp6, only on tcp (--inet).

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

#!/bin/bash
# Linux

ip -6 addr add 2a02:2308:10::c:19 dev lo
ip -4 addr add 193.200.132.187 dev lo

# or add something like this to your *home router's* nameserver
cat >>/etc/hosts <<<EOF
193.200.132.187 ip6.nl
2a02:2308:10::c:19 ip6.nl
EOF

cat >/tmp/Caddyfile-ds <<EOF
http://ip6.nl:80 {
  bind ip6.nl
  root /var/www
}
EOF

caddy -conf Caddyfile-ds &
sleep 1
netstat -lnpa | grep -F caddy
# tcp6, or only tcp?

# ----
kill $!
sed -i -e '/ip6.nl/d' /etc/hosts
ip -6 addr del 2a02:2308:10::c:19 dev lo
ip -4 addr del 193.200.132.187 dev lo

[mholt]

It listens on whatever address your system resolver returns.

Is it that you want to spawn two listeners for the same site?

[wmark]

The resolver returns two addresses, the ipv4 and the ipv6 one. _Caddy_ goes with the first one (ipv4) and ignores the ipv6-one. (It's really a flaw in Golang.)(Just listening on [::]:80 is a no-go on multi-homed servers, or several sites.)

Without this _Caddy_ is not fully »ipv6 compliant«, and falling behind _Nginx_ or _Apache httpd_.

Yes, I am suggesting using http.Serve with two or more Listener. Due to possible shared state I recommend not just starting two _servers_ in two _goroutines_. (Yes: 1←L, No: L×(1←1))

This would most probably be achieved by an implementation along these lines:

1. If directive bind gets more than one parameter, and/or…
2. If and when a hostname resolves to more than one address, and:
   
   
   
   • for ipv4 addresses: for every such address that is in net.InterfaceAddrs()
   
   
   • for ipv6: for every such address that falls under the configured prefixes
     
     … create a new Listener, or reuse an existing one.
   
   
3. Implement pattern fan in: Another MultiListener encapsultes them all, acts as the sole Listener we then pass to http.* (or any other server).

---

Primer on ops:

• It's a common scenario in datacenters that your server has network devices such as ext0, intra0, storage0 and so on. (It's a good practice to rename a ext0 or en0p2 and the such according to their function.)
• Unless a daemon can bind to a specific network interface (good!) you just avoid 0.0.0.0:PPPP and [::]:PPPP for that reason.
• (Then, it's not uncommon that you have multiple ipv4 addresses assigned to said ext0: For example, a machine-dependent one, and some free-floating ones (like, one for every customer).)
• Instead of several /112 or /128 ipv6 addresses (which is sometimes done for free-floating ones (the largest DC operator in Austria does this)) the server gets an entire ipv6 /64 subnet. Therefore the subnet check in step (2) above.

Websites for ipv4+ipv6:

# zone — ip6.nl

@ IN A 193.200.132.187
@ IN AAAA 2a02:2308:10::c:19
www IN A 193.200.132.187
www IN AAAA 2a02:2308:10::c:19

# nginx.conf

listen 193.200.132.187:80;
listen [2a02:2308:10::c:19]:80 ipv6only=on;

(Yes, one-trick–pony installations can get away with listen [::]:80 ipv6only=off;.)

[mholt]

Looking into this.

[wmark]

That's no feature but a big flaw that severely limits _Caddy_'s use in ipv6 deployments.

Debugging this you could try to enable ipv6 for domains caddyserver.com and matt.life. ;-)

[mholt]

This is not a bug. You can give Caddy an ip6 address to bind to. You want a feature so that caddy can bind to more one interface, and I think it'll be good for the bind directive, but I need to look into it more.

[mholt]

Track https://github.com/golang/go/issues/9334

[Juerd]

Hi, I'm the owner of the ip6.nl domain and I maintain the server to which the associated IP addresses are assigned.

I was surprised to see that these were used in an example, by someone I don't know. For the record, I would like to add to this discussion that ip6.nl does not use Caddy, and that the user wmark is not associated with ip6.nl in any way.

It is generally a bad idea to re-use actual internet domain names and public routable IP addresses in examples or for local tests. This is why the domain name example.org (RFC2606) and the IPv6 prefix 2001:db8::/32 (RFC3849) exist.

wmark, please only use official example domains and IP addresses, or your own.

[rhester72]

I have the same issue/concern as wmark - the ability to listen on distinct specific IPv4 and IPv6 addresses is a must in my environment (where sslh front-ends Caddy on the same host).

[rhester72]

My own thoughts about use cases and potential solutions [heavily edited and revised from the initial posting, that's what I get for writing specs early in the morning]:

It's a given that many people today are front-ending their web server(s) in some way - proxy, load balancer, multiplexer, what have you. It's also a reasonable bet on a SoHo installation that more than one of these functions will be played by the same host, and quite possibly without things like Docker constructs.

In my own case, I am front-ending 80/TCP and 443/TCP with sslh, multiplexing VPN, SSH, and HTTP(S) traffic over those ports (as many scenarios restrict outbound traffic to 80 and 443). I achieve this by the following:

Internet (80/TCP) -> router -> multiplexer (IP A) -> web server:81 (IP B)
Internet (443/TCP) -> router -> multiplexer (IP A) -> web server:444 (IP B)

The exact same construct is used for IPv6 as IPv6, except NAT isn't used - the "public" IPv6 points at the multiplexer, and a "private" IPv6 houses the web server

Therein lie the issues with Caddy:

• Wanting to take over -every- address and interface on ports 80 and 443 make things like same-system front-ending very unnecessarily challenging
• The requirement that Caddy be listening _itself_ on 80 and 443 to make Automatic HTTPS function seem an arbitrary requirement - fine for a test environment or lab, but if you're using a front-end, odds are that you _don't_ want Caddy listening on default ports, and you probably still _do_ want Automatic HTTPS - and there's no reason you shouldn't be able to do that, because Caddy shouldn't see a difference (if the front-end is working properly - and if not, you won't get your cert!)

There are variations on this theme, of course - say I want to multiplex 80 but not 443...this is simply not possible by any means with Caddy as it exists today without far more granular port control.

It gets even more interesting...what if I want to try QUIC? Worse, over IPv6? See where I'm going with this? =)

Port management is a hard requirement for Caddy if it is to be seen/used as more than a toy, and given the amazing things Caddy brings to the table, I'd _really_ like to see it brought out of that realm and thus available to a much wider audience.

Following a model similar to that of nginx, in this case, isn't such a bad idea as it keeps complexity down/abstracted from the user while offering what's needed (I'm just talking about the Caddyfile, not the back-end implementation, for which my Go skills are effectively non-existent, sadly).

Consider the following potential Caddyfile entries (where no port binding implies 80/HTTP and 443/TLS and port bindings to 80 implies HTTP and 443 implies TLS):

• No "bind" directive - work as you do today, binding on :::80 and :::443
• Single bind directive, i.e.

bind 192.168.1.3

Bind on 192.168.1.3:80 as HTTP and 192.168.1.3:443 as TLS

bind [2001:470:8000:2]

Bind on [2001:470:8000:2]:80 as HTTP and [2001:470:8000:2]:443 as TLS

(These two should be able to be combined in a single Caddyfile for dual-stack environments)

• "nossl" bind directive:

bind 192.168.1.3 nossl

Bind only on 192.168.1.3:80 as HTTP only

bind 192.168.1.3:80

Bind only on 192.168.1.3:80 as HTTP only (same meaning as 192.168.1.3 nossl)

bind 192.168.1.3:80 nossl

Bind only on 192.168.1.3:80 as HTTP only (redundant, but valid)

bind 192.168.1.3:443 nossl

Bind only on 192.168.1.3:443 as _HTTP only_ (weird, but valid!)

• "ssl" bind directive:

bind 192.168.1.3
bind 192.168.1.4:444 ssl

Bind on 192.168.1.3:80 as HTTP and TLS and 192.168.1.4:444 as TLS only

bind 192.168.1.5:81 ssl

Bind on 192.168.1.5:81 as TLS only

• More complex binding

bind 192.168.1.5
bind 192.168.1.6:8080
bind 192.168.1.20 ssl

Bind on 192.168.1.5:80 as HTTP and TLS, 192.168.1.6:8080 as HTTP and TLS and 192.168.1.20 as TLS only

• Invalid binding examples

bind 192.168.1.3
bind 192.168.1.3 nossl

How should this be handled? Invalid configuration (since technically the TLS binding is inferred but then explicitly rejected), or just 'merge' the two requests and go with the more-specific (second, HTTP-only) binding?

bind 192.168.1.3 nossl
bind 192.168.1.3 ssl

Similar to the above - invalid, or merge and assume you want HTTP on 80 and SSL on 443?

Given the particularly peculiar nature of QUIC using UDP and thus generally _not_ front-ended, I propose the following additional flag:

• QUIC designation

bind 192.168.1.3
bind 192.168.1.2 quic
bind [2001:470:8000:2] quic

Bind on 192.168.1.3:80 as HTTP and TLS, 192.168.1.2:443 UDP and [2001:470:8000:2]:443 UDP as QUIC

Naturally, I'd assume that:

• The quic directive (and in fact the entire binding) would be ignored if the -quic option isn't passed on the command line, and
• quic would directly imply ssl (and thus mandate TLS)

Bindings should be permitted at the global or vhost level, and where overlap exists, multiple vhosts should be allowed to share the given binding (just as Caddy works today, I believe).

Just my $0.02 cents for how to unambiguously represent various configuration options to the user as exposed through the Caddyfile while offering complete deployment flexibility. Whether or not this is something that can be achieved in Go, however, is a completely different question I am not qualified to answer.

This spec would require the following:

• The restriction of only permitting Automatic HTTPS to be attempted when no ports are specified would have to be removed
• The problem of ACME not functioning properly in a IPv6-only configuration referenced here would have to be solved

Feedback/comments/criticism/tomatoes welcome.

NOTE: I'm thinking of revising this approach somewhat, specifically around quic - such that ssl infers quic, but quic does _not_ infer ssl. This would mean you potentially have UDP listeners where you may not 'want' them, but it would greatly simplify configuration parsing, which I think is a worthwhile tradeoff. I'd also like to further expand on host/port definitions in order to make it clear that this proposal is intended as 100% backwards-compatible with existing behavior, and such that binding would proceed exactly as it does today, just in a 'waterfall' model.

[abh]

I expected this to work, but it only listens on the last 'bind' statement. (I understand it working this way might be required to be able to override a default):

bind 2607:f238:3::222:0
bind 207.171.7.222

so alternatively having this work would be nice:

bind 2607:f238:3::222:0 207.171.7.222

There's no way to use a set of specific IPs and have IPv4 and IPv6 work currently without duplicating the whole configuration.

[hrbrmstr]

will this issue be getting any attention? lack of parity with nginx for just this issue has me a bit confused about all the recent "use caddy!" hype from high profile aggregators and bloggers.

[keks]

I would also love to see this fixed. For now I use socat as a workaround:

sudo socat  tcp6-listen:80,bind=example.com,fork  tcp4-connect:example.com:80  &
sudo socat  tcp6-listen:443,bind=example.com,fork tcp4-connect:example.com:443 &

But seriously, this bug has been open for way too long.

[NoAgendaIT]

We use bind (and have to use bind) with IP-addresses because we're on a multihomed server (multiple IP addresses), and use round-robin DNS which means that resolving our hostname will return IP addresses from more than one server.

So, what we need is to be able to specify multiple IP addresses with bind:

bind 203.0.113.15, 2001:DB8:1234:5678:abcd::15

Or to specify bind more than once:

bind 203.0.113.15
bind 2001:DB8:1234:5678:abcd::15

For now, I will use firejail (or socat, thanks @keks) as a workaround. But @mholt could you please give this issue the attention it deserves?

[mholt]

But @mholt could you please give this issue the attention it deserves?

Eventually, yes - but this issue isn't a priority for me right now.

To anyone tracking this feature request, why not submit a PR? It'd be a great way to get involved and make a first contribution. The other alternative to waiting or doing it yourself is to purchase the Engineering Package, which means that this feature will receive priority. Your time is valuable, so the cost to you is about the same either way.

[rhester72]

So you're now suggesting that for only $9,000 US, we'd _improve_ our chances of getting a very basic and expected feature of a web server in Caddy, despite the fact that (to the best of my knowledge) Go itself still doesn't support multi-binding, in an open-source and community-contributed-and-supported project.

Wow. I have no words. I'm not certain any longer who the target audience for Caddy is, but it would certainly seem that anyone with more complex requirements than a basic, single-homed, IPv4-only development web server isn't part of it.

[mholt]

@rhester72

So you're now suggesting that for only $9,000 US

It's $900 monthly -- roughly the value of one day's consulting/contracting work if you count hours. That's totally appropriate to get a feature like this.

we'd _improve_ our chances

If you go this route, what you're also paying for is a re-adjustment of the authors' priorities. I've already said this is something Caddy will get, it's just a matter of when.

getting a very basic and expected feature of a web server in Caddy despite the fact that (to the best of my knowledge) Go itself still doesn't support multi-binding,

Then it's not "very basic" :wink: There be dragons in network stacks. But this feature is not super hard either. I suspect an extra loop will be needed in the function that groups site configs by listen address to create multiple listeners. Go may not support multi-binding in a single call to Listen() in the way you/we think about it, but there's nothing stopping us from creating multiple listeners each listening on a different bind address.

in an open-source and community-contributed-and-supported project.

Exactly. I would love to have your contributions to make it better (and the only cost then is your time). Honestly, I would prefer this. :+1:

I'm not certain any longer who the target audience for Caddy is

That's why we're in charge of it -- we have the vision and we know where Caddy is going, because I and the other maintainers try to keep close tabs on what the community is doing with it. We can't do everything at once. That's why we appreciate the feedback, the interaction, and your involvement!

[wmark]

FYI, I wouldn't label this as easy/for-beginners: The good implementation will require some knowledge of OS (as well as networking!) fundamentals a beginner won't have. Plus, testing will be tricky.

But I support labelling this as bug and not RFE.

[NoAgendaIT]

@mholt I totally understand, so no arguing from me. It's just that the project I'd like to run caddy for is one I support pro-bono - a small grass-roots, donation supported podcast, so there's no money for things like this. I'll just have to wait until someone else picks up the issue, especially as I am a sysadmin, not a coder.

[mholt]

I think https://github.com/golang/go/issues/9334 is probably the _best_ place to take this issue, as that's really the limitation here. Maybe we can hack around it in Caddy, but the _right_ answer is to improve the standard library (and then everyone else benefits too). However, even if the stdlib does get this feature, what exactly does that entail? Interfaces can go up and come down: does binding a host to "all its interfaces" imply that it binds to new interfaces as they come up? how?

For now, just bind to the empty hostname (which Caddy does by default without the bind directive).

[abh]

For now, just bind to the empty hostname (which Caddy does by default without the bind directive).

I moved on from caddy mostly because of this so I don’t care too much, but just to repeat: the use case here is caddy co-existing with other software that listens on port 80/443 on the same host.

[wmark]

By considering NIC to IP allocations you are inviting friction into your thought process. Before it broadens the scope of this issue (though not orthogonally), assume it's the responsibility of a webserver (any daemon, really) to listen on all IPs you throw at it. (If not all are known in advance, it's the problem of the operator.)

In case you still feel uncomfortable about this, have Caddy display a warning. Not all addresses we listen on are currently assigned to this machine. Caddy might have been started before DHCP kicked in, or this could be a more sophisticated environment.

Anyway, the implementation will look like this:

1. Either the operator provides them using directive bind as usual, or gather all applicable IP addresses (of any family) explicitly.
2. For every of the above addresses create a new Listener.
   Once (1)–(4) is working, use IP_FREEBIND here, to cover your concerns from before.
3. Feed those Listeners to a Multiplexer, which implements net.Listener.
4. Use this Multiplexer(Listener, Listener, …) henceforth.