Caddy: Could not start HTTPS server for challenge, bind: address already in use

[bluecmd]

1. What version of Caddy are you running (caddy -version)?

Caddy 0.9.3

2. What are you trying to do?

Using Caddy to on-demand request SSL certificates.

3. What is your entire Caddyfile?

redacted.tld {
  bind 9.9.9.9
  # Caddy doesn't support binding two addresses :-(
  # bind [redacted]
  # Skip authentication for public services
  proxy / [::1]:8080 {
    transparent
  }
}

cloud.redacted.tld *.cloud.redacted.tld *.*.*.*.cloud.redacted.tld {
  bind 9.9.9.9
  # Caddy doesn't support binding two addresses :-(
  # bind [redacted]
  # Authenticate through oauth2_proxy and then go to NGINX
  proxy / [::1]:8081 {
    transparent
  }
  tls {
    max_certs 100
  }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

$ grep -vE '^;' /lib/systemd/system/caddy.service
[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network.target
Wants=network.target

[Service]
Restart=on-failure

User=www-data
Group=www-data

ExecStart=/usr/local/bin/caddy -log stderr -agree=true -conf=/etc/Caddyfile -root=/var/tmp -email [email protected]
ExecReload=/bin/kill -USR1 $MAINPID

PrivateTmp=true

CapabilityBoundingSet=CAP_NET_BIND_SERVICE
LimitNOFILE=8192


[Install]
WantedBy=multi-user.target

5. What did you expect to see?

I expect any request to the matched vhosts to have an SSL certificate generated and used to serve the traffic.

6. What did you see instead (give full error messages and/or log)?

Some domains work, until it doesn't. After some point no more domains succeed. It's almost as if it's the Let's Encrypt limit, but as the error doesn't even mention that I'm guessing it's not.

When trying to connect to a domain that has no previous certificate Chrome reports:

This site can’t provide a secure connection

foobar.cloud.redacted.tld sent an invalid response.
Try running Windows Network Diagnostics.
ERR_SSL_PROTOCOL_ERROR

Caddy logs:

Nov 12 15:40:34 border caddy[30222]: 2016/11/12 15:40:34 [INFO] Obtaining new certificate for foobar.cloud.redacted.tld
Nov 12 15:40:34 border caddy[30222]: 2016/11/12 15:40:34 [INFO][foobar.cloud.redacted.tld] acme: Obtaining bundled SAN certificate
Nov 12 15:40:35 border caddy[30222]: 2016/11/12 15:40:35 [INFO][foobar.cloud.redacted.tld] acme: Trying to solve TLS-SNI-01
Nov 12 15:40:35 border caddy[30222]: 2016/11/12 15:40:35 http: TLS handshake error from 1.2.3.4:24707: [foobar.cloud.redacted.tld] failed to get certificate: [foobar.cloud.redacted.tld] error presenting token: Could not start HTTPS server for challenge -> listen tcp 9.9.9.9:443: bind: address already in use

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

Not sure.

Maybe related to the "bind" statement as nobody else seems to have reported the issue. When testing now I have nothing else running on port 443 (on any IP).

I'm quite stumped on why this happens, but happy to help to test any configuration or provide non-redacted logs and config in a non-public forum.

[dfiel]

Same issue here:

Caddyfile:

 *.*, *.*.* {

         proxy / 10.250.1.3 {
             transparent
             websocket
         }
         tls {
            max_certs 500
         }
 }

 104.XXX.72.6:80 {
    redir https://XYZ.com
 }

`

Log:

2016/11/16 15:48:09 [INFO] Obtaining new certificate for invoice.dfiel.com
2016/11/16 15:48:10 [INFO][invoice.dfiel.com] acme: Obtaining bundled SAN certif icate
2016/11/16 15:48:10 [INFO][invoice.dfiel.com] acme: Trying to solve TLS-SNI-01
2016/11/16 15:48:10 http: TLS handshake error from 100.34.228.142:35352: [invoic e.dfiel.com] failed to get certificate: [invoice.dfiel.com] error presenting tok en: Could not start HTTPS server for challenge -> listen tcp :443: bind: address already in use
2016/11/16 15:48:11 [WARNING] Stapling OCSP: no OCSP stapling for [www.dfiel.com ]: ocsp: error from server: unauthorized

What could the issue be?

[tgmedia-nz]

Same here, same version and scenario - any ideas?

[dfiel]

This is a pretty bad issue @mholt, any ideas?

[tobya]

Does your site work when you already have a TLS certificate . eg is this a problem with certificate creation or (as the error would suggest) a more general problem with caddy not being able to bind to 443?

[dfiel]

@tobya I only have sites with Let's Encrypt. Sites that already have certificates load just fine, I can browse them at :443, but sites that need to renew their certs have issues. It seems to be a different site every time

[mholt]

Hi, sorry, I'm not ignoring this -- just been really busy. Any collaborators want to take this? (@tobya maybe?) First step would be to take the above Caddyfile and try to reproduce it (be sure to use Let's Encrypt's staging endpoint!!!).

[elcore]

Hi,

I am really busy right now 😢 , but I´d like to point you all in the "right" direction 😄 !

I "guess" it has something to do with the TLS-SNI-01 challenge:

https://github.com/xenolf/lego/blob/master/acme/tls_sni_challenge_server.go#L41-L44

s.listener, err = tls.Listen("tcp", net.JoinHostPort(s.iface, s.port), tlsConf)

https://github.com/golang/go/blob/master/src/crypto/tls/tls.go#L70-L71

// Listen creates a TLS listener accepting connections on the
// given network address using net.Listen.

There is no such issue with the HTTP-01 challenge!

2016/11/23 23:48:27 [INFO][1.caddy.domain.com] acme: Trying to solve TLS-SNI-01
2016/11/23 23:48:28 http: TLS handshake error from XXX.XXX.XXX.XXX:34980: [1.caddy.domain.com] failed to get certificate: [1.caddy.domain.com] error presenting token: Could not start HTTPS server for challenge -> listen tcp :443: bind: address already in use

2016/11/23 23:50:40 [INFO] Obtaining new certificate for 2.caddy.domain.com
2016/11/23 23:50:41 [INFO][2.caddy.domain.com] acme: Obtaining bundled SAN certificate
2016/11/23 23:50:41 [INFO][2.caddy.domain.com] acme: Could not find solver for: dns-01
2016/11/23 23:50:41 [INFO][2.caddy.domain.com] acme: Trying to solve HTTP-01
2016/11/23 23:50:41 [INFO][2.caddy.domain.com] Served key authentication
2016/11/23 23:50:42 [INFO][2.caddy.domain.com] The server validated our request

[mholt]

Caddy should be able to know when the port is in use, though, and tell lego to use an alternate port. It does in most cases, but I wonder if the problem here is how we handle (or don't handle) the wildcards in the hostname.

[bluecmd]

Confirmed to fix the issue for me :)

[elcore]

Hello @bluecmd,

perfect :smile:

[mholt]

I can't reproduce this even with the above Caddyfiles... very odd. I'm using my "spare" domain in these tests, and the only change I've made is to disable the HTTP challenge to ensure that I use the TLS-SNI challenge as your logs indicate (also I don't go 5 subdomains deep with the wildcards, just one, in my Caddyfile):

https://mydomain.com
2016/11/26 14:30:14 https://mydomain.com
https://*.mydomain.com
2016/11/26 14:30:14 https://*.mydomain.com
http://mydomain.com
2016/11/26 14:30:14 http://mydomain.com
http://*.mydomain.com
2016/11/26 14:30:14 http://*.mydomain.com
2016/11/26 14:30:21 [INFO] Obtaining new certificate for cloud.mydomain.com
2016/11/26 14:30:22 [INFO][cloud.mydomain.com] acme: Obtaining bundled SAN certificate
2016/11/26 14:30:22 [INFO][cloud.mydomain.com] acme: Could not find solver for: dns-01
2016/11/26 14:30:22 [INFO][cloud.mydomain.com] acme: Could not find solver for: http-01
2016/11/26 14:30:22 [INFO][cloud.mydomain.com] acme: Trying to solve TLS-SNI-01
2016/11/26 14:30:23 [INFO][cloud.mydomain.com] The server validated our request
2016/11/26 14:30:23 [INFO][cloud.mydomain.com] acme: Validations succeeded; requesting certificates
2016/11/26 14:30:24 [INFO] acme: Requesting issuer cert from https://acme-staging.api.letsencrypt.org/acme/issuer-cert
2016/11/26 14:30:24 [INFO][cloud.mydomain.com] Server responded with a certificate.

My Caddyfile (taken from @bluecmd's):

mydomain.com {
  bind 192.168.1.3
  # Caddy doesn't support binding two addresses :-(
  # bind [redacted]
  # Skip authentication for public services
  proxy / [::1]:8080 {
    transparent
  }
}

*.mydomain.com {
  bind 192.168.1.3
  # Caddy doesn't support binding two addresses :-(
  # bind [redacted]
  # Authenticate through oauth2_proxy and then go to NGINX
  proxy / [::1]:8081 {
    transparent
  }
  tls {
    max_certs 2
  }
}

Can you provide more details to ensure the behavior you're describing can be reproduced?

[dfiel]

I've found this is a random issue, only affecting some domains, and then after a restart, different domains don't work. It's not easily reproducible, maybe someone else can provide more details

[elcore]

Hello @mholt,

Should we give it a try and push it to master? -- It increases coverage by 1.5% 😎

Confirmed to fix the issue for me :)

P.S.: bind 0.0.0.0 -> listen tcp 0.0.0.0:443: bind: address already in use

--- FAIL: TestListenerAddrEqual (0.00s)
        caddy_test.go:95: Test 3 (0.0.0.0:8825 == [::]:8825): expected true but was false
FAIL
FAIL    github.com/mholt/caddy  0.026s

[mholt]

P.S.: bind 0.0.0.0 -> listen tcp 0.0.0.0:443: bind: address already in use

BINGO. That does it.

Your fix seems to work for me too. Thanks!

[tgmedia-nz]

confirming that it fixed it for me, too! Awesome work guys! :)