Caddy: Caddy ignores -https-port and breaks custom port mapping with recent change

Created on 24 Dec 2018  Â·  58Comments  Â·  Source: caddyserver/caddy

1. What version of Caddy are you using (caddy -version)?

broken (latest master):
0b83014ff81b68b8fde21521662339396c277ab8

a working revision:
4f5df39bdd9ce05146da14bb60f5a17a163d5262

2. What are you trying to do?

Proper security by explicit port forwarding and explicitly limited capabilities.

3. What is your entire Caddyfile?

http://example.com:8080 {
  redir https://asdas.net{uri}
}

https://example.comt:8443 {
  tls [email protected]
  limits 1mb
  log stdout
  errors stdout
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy -agree -http-port 8080 -https-port 8443 -conf /path/to/config/Caddyfile -log stdout
  • external port mapping from external ports 80 and 443 to internal 8080 and 8443 by SDN.

6. What did you expect to see?

Older versions including the "working" revision respect the flags and are able to request and and renew LE/ACME certificates fine with the config. At least the version defined as "broken" tries to bind to port 443 ignoring the given CLI flag.

bug

Most helpful comment

All 58 comments

I am also experiencing the same issue with the Caddyfile below. When I only have one directive everything works as expected but as soon as I add a second directive Caddy tries to bind to port 443 even though the -https-port 8443 command line flag is set.

https://one.example.com {
  tls {email}
  ...
}

https://two.example.com {
  tls {email}
  ...
}

Huh, I've been using those flags in developing all the latest commits and haven't had any issues. @rmoriz is the broken commit you linked to the first one where it broke, or is it the only one you've tried? I ask because I haven't seen this myself, despite constant use of those flags (with multiple sites), on any of the latest commits.

I forgot to add to my original message, the console output I get when starting Caddy is:

2019/01/22 22:02:16 [INFO] [{domain}] acme: Trying to solve TLS-ALPN-01
2019/01/22 22:02:17 [{domain}] failed to obtain certificate: acme: Error -> One or more domains had a problem:
[{domain}] [{domain}] acme: error presenting token: presenting with standard provider server: could not start HTTPS server for challenge -> listen tcp :443: bind: permission denied

@oscartbeaumont Thanks. Which tag or commit are you using?

@mholt just the (then) current version and the last one we used before (and was working for us). Breaking commit must have happened somewhere in between.

@rmoriz Would you mind trying the most recent commit - the latest on master? A lot of changes have happened in those code paths recently and I don't know if I remember having that problem, but worth checking if it's fixed for you now, about a month later.

I am using the latest tag (v0.11.2) but I had the same problem on the master branch.
EDIT: I just tested the master branch and can confirm the bug still exists

@mholt still broken with a7aeb979be5a59ae82fac0783fe54b83c7fc57fd

Activating privacy features... 2019/01/22 22:21:35 [INFO] acme: Registering account for [email protected]
2019/01/22 22:21:35 [INFO] [asdas.net] acme: Obtaining bundled SAN certificate
2019/01/22 22:21:36 [INFO] [asdas.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/vIAvxtzyJitkj_Q5etotyov7FPVt4LNwmq-nUMJUBYA
2019/01/22 22:21:36 [INFO] [asdas.net] acme: use tls-alpn-01 solver
2019/01/22 22:21:36 [INFO] [asdas.net] acme: Trying to solve TLS-ALPN-01
2019/01/22 22:21:36 [asdas.net] failed to obtain certificate: acme: Error -> One or more domains had a problem:
[asdas.net] [asdas.net] acme: error presenting token: presenting with standard provider server: could not start HTTPS server for challenge -> listen tcp :443: bind: permission denied

Well, I can't reproduce this, not with my Caddyfile, not with the one you provided. I even added log lines to print the ports being used, and the flags are being honored.

To further debug this, you'll have to do some investigation on your own, I'm afraid.

I was just trying to reproduce the bug and I managed to make it work. It must be something about the way @rmoriz and I are building our Docker containers that is causing the issue. You can see how I set up the system here. Now i'm going to step through my Docker container and try and see what is causing the issue.

@whitestrake is helping to debug this, siince he has Docker. (Thank you!)

Make sure to use build.go to build Caddy, instead of running go build directly, since you won't get any useful version information with caddy -version unless you use build.go.

Did some quick testing and replicated the issue quite easily with the provided repo rmoriz/caddy_example_issue_2407. It doesn't seem related to the Docker environment itself, because I can pull the built binary out of the container and run it on the host with the same problem.

Next step to check is whether building it outside of the Docker environment using the same steps as the Dockerfile produces the same results. Since @mholt can build that revision without this issue appearing, we need to narrow down whether it's an environment issue (i.e. Docker ?? or Alpine 3.8 or Golang 1.11) or a problem with the steps used to build it.

@whitestrake or @rmoriz Can you test adding the -email command line flag to the Caddy startup command and see if it fixes the problem?

using a poor man's bisect approach (see https://github.com/rmoriz/caddy_example_issue_2407/blob/master/bisect.sh ) it looks like it started with https://github.com/mholt/caddy/commit/d1171af6792437a258c16b975678b3da6a5fca5e

@oscartbeaumont does not change anything.

It _started_ at that commit? So everything after that commit, including that one itself, exhibits the behavior you're reporting? But _before_ that commit, things work, with no other changes?

Bizarre. That little change should have no effect on the ports that are used...

@mholt you should become familiar with docker so you could just replicate that case with a single command ;-)

Probably, but that'll take more time I don't have. I reallllly gotta get my thesis done by February.

Since you've got it all set up, you might as well add some log.Print statements and see where the port logic goes wrong. Look in the vendored client.go (in certmagic) for starters.

@oscartbeaumont: adding an -email flag with a valid email address did not solve the issue when building revision a7aeb97.

I have started poking around in the code and found that the change made in d1171af should not be affecting the issue because the code in the if statement was never called. I believe this is an issue with certmagic because I can reproduce the issue with this small go program that uses Certmagic. (It is possible I am completely wrong, so if someone else has time it would probably be good for them to have a look)

I got my demo working by adding the AltTLSALPNPort value to the certmagic.Config as shown here. This same value is set in Caddy but somewhere in between it being set here and it being used by Certmagic here the value is changing. If I add the line c.TLS.Manager.AltTLSALPNPort = 8443 just before the ObtainCert is run everything works as expected. I am struggling to work out what specifically is changing it, any ideas @mholt? I saw something like this line which would cause an issue like this but that line isn't being executed with my config.

Maybe my git bisec has an off-by-one error. The commit before https://github.com/mholt/caddy/commit/d1171af6792437a258c16b975678b3da6a5fca5e was https://github.com/mholt/caddy/commit/598de9e6d990619e4a325e166825e3d3c365823c which involves certmagic. Sorry for that.

If you bisect those two commit what is the result? It might help in finding where the AltTLSALPNPort is being modified in the code.

Well…

➜  caddy_example_issue_2407 git:(master) export caddy_revision=d1171af6792437a258c16b975678b3da6a5fca5e 
➜  caddy_example_issue_2407 git:(master) docker-compose up --build --force-recreate                    
Building proxy
Step 1/17 : FROM golang:1.11-alpine3.8
 ---> f56365ec0638
Step 2/17 : ARG caddy_revision
 ---> Using cache
 ---> d062015bd4db
Step 3/17 : RUN apk add --no-cache git tar curl      && mkdir -p /go/src/github.com/mholt      && cd /go/src/github.com/mholt && git clone https://github.com/mholt/caddy
 ---> Using cache
 ---> a78665328da1
Step 4/17 : RUN cd /go/src/github.com/mholt/caddy && git checkout ${caddy_revision} -b deployment      && go get -u -d github.com/caddyserver/builds
 ---> Using cache
 ---> 4524e0e2abe8
Step 5/17 : WORKDIR /go/src/github.com/mholt/caddy/caddy
 ---> Using cache
 ---> 93ddfca9ac59
Step 6/17 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o caddy  && ./caddy -version
 ---> Using cache
 ---> 6a97f0b3cd47
Step 7/17 : FROM alpine:3.8
 ---> 3f53bb00af94
Step 8/17 : ARG caddy_revision
 ---> Using cache
 ---> 613153c3456c
Step 9/17 : COPY --from=0 /go/src/github.com/mholt/caddy/caddy/caddy /usr/bin/caddy
 ---> Using cache
 ---> 67a07b449ceb
Step 10/17 : RUN apk add --no-cache ca-certificates curl   && chmod 0755 /usr/bin/caddy   && /usr/bin/caddy -version   && addgroup -g 82 -S www-data   && adduser -u 82 -D -S -G www-data www-data
 ---> Using cache
 ---> cd998f609f73
Step 11/17 : ENV CADDYPATH=/config
 ---> Using cache
 ---> 1e8f8887b258
Step 12/17 : RUN mkdir -p /config/tls && chown -R www-data:www-data /config     && /usr/bin/caddy -version
 ---> Using cache
 ---> 055923f5f404
Step 13/17 : ADD files/Caddyfile /config/Caddyfile
 ---> Using cache
 ---> 4824880c8dde
Step 14/17 : USER www-data
 ---> Using cache
 ---> 7675af2d34ce
Step 15/17 : WORKDIR /config
 ---> Using cache
 ---> 7f18d8da76b0
Step 16/17 : ENTRYPOINT ["/usr/bin/caddy"]
 ---> Using cache
 ---> ce14f5175569
Step 17/17 : CMD ["-ca", "https://acme-staging-v02.api.letsencrypt.org/directory", "-agree", "-http-port", "8080", "-https-port", "8443", "-conf", "/config/Caddyfile", "-log", "stdout"]
 ---> Using cache
 ---> cfadc08756eb
Successfully built cfadc08756eb
Successfully tagged caddy_example_issue_2407_proxy:latest
Recreating caddy_example_issue_2407_proxy_1 ... done
Attaching to caddy_example_issue_2407_proxy_1
proxy_1  | 2019/01/26 14:52:45 [ERROR] Loading persistent UUID: open /config/uuid: permission denied
proxy_1  | 2019/01/26 14:52:45 [ERROR] Persisting instance UUID: open /config/uuid: permission denied
proxy_1  | Activating privacy features... 
proxy_1  | 
proxy_1  | Your sites will be served over HTTPS automatically using Let's Encrypt.
proxy_1  | By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
proxy_1  |   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
proxy_1  | Please enter your email address to signify agreement and to be notified
proxy_1  | in case of issues. You can leave it blank, but we don't recommend it.
proxy_1  |   Email address: 2019/01/26 14:52:47 [INFO] [asdas.net] acme: Obtaining bundled SAN certificate
proxy_1  | 2019/01/26 14:52:47 [INFO] [asdas.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/5vvVYSQEN4L-1sb2TTZBKBvGy5R6MEReGvk9pFanapQ
proxy_1  | 2019/01/26 14:52:47 [INFO] [asdas.net] acme: use tls-alpn-01 solver
proxy_1  | 2019/01/26 14:52:47 [INFO] [asdas.net] acme: Trying to solve TLS-ALPN-01
proxy_1  | 2019/01/26 14:52:47 [asdas.net] failed to obtain certificate: acme: Error -> One or more domains had a problem:
proxy_1  | [asdas.net] [asdas.net] acme: error presenting token: presenting with standard provider server: could not start HTTPS server for challenge -> listen tcp :443: bind: permission denied
caddy_example_issue_2407_proxy_1 exited with code 1
➜  caddy_example_issue_2407 git:(master) 





➜  caddy_example_issue_2407 git:(master) export caddy_revision=598de9e6d990619e4a325e166825e3d3c365823c
➜  caddy_example_issue_2407 git:(master) docker-compose up --build --force-recreate                    
Building proxy
Step 1/17 : FROM golang:1.11-alpine3.8
 ---> f56365ec0638
Step 2/17 : ARG caddy_revision
 ---> Using cache
 ---> d062015bd4db
Step 3/17 : RUN apk add --no-cache git tar curl      && mkdir -p /go/src/github.com/mholt      && cd /go/src/github.com/mholt && git clone https://github.com/mholt/caddy
 ---> Using cache
 ---> 7a96d3422a71
Step 4/17 : RUN cd /go/src/github.com/mholt/caddy && git checkout ${caddy_revision} -b deployment      && go get -u -d github.com/caddyserver/builds
 ---> Using cache
 ---> e887ce670885
Step 5/17 : WORKDIR /go/src/github.com/mholt/caddy/caddy
 ---> Using cache
 ---> 066d6838b384
Step 6/17 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o caddy  && ./caddy -version
 ---> Using cache
 ---> ea8fa785412a
Step 7/17 : FROM alpine:3.8
 ---> 3f53bb00af94
Step 8/17 : ARG caddy_revision
 ---> Using cache
 ---> 613153c3456c
Step 9/17 : COPY --from=0 /go/src/github.com/mholt/caddy/caddy/caddy /usr/bin/caddy
 ---> Using cache
 ---> 518550dc1b85
Step 10/17 : RUN apk add --no-cache ca-certificates curl   && chmod 0755 /usr/bin/caddy   && /usr/bin/caddy -version   && addgroup -g 82 -S www-data   && adduser -u 82 -D -S -G www-data www-data
 ---> Using cache
 ---> 12a67d37bff7
Step 11/17 : ENV CADDYPATH=/config
 ---> Using cache
 ---> 5d90dc232c2c
Step 12/17 : RUN mkdir -p /config/tls && chown -R www-data:www-data /config     && /usr/bin/caddy -version
 ---> Using cache
 ---> bf5ef687af2e
Step 13/17 : ADD files/Caddyfile /config/Caddyfile
 ---> Using cache
 ---> 926de41f09fb
Step 14/17 : USER www-data
 ---> Using cache
 ---> 3f9b454b19d9
Step 15/17 : WORKDIR /config
 ---> Using cache
 ---> 6acd70cccf62
Step 16/17 : ENTRYPOINT ["/usr/bin/caddy"]
 ---> Using cache
 ---> b257bf13c92a
Step 17/17 : CMD ["-ca", "https://acme-staging-v02.api.letsencrypt.org/directory", "-agree", "-http-port", "8080", "-https-port", "8443", "-conf", "/config/Caddyfile", "-log", "stdout"]
 ---> Using cache
 ---> 532a6a181caf
Successfully built 532a6a181caf
Successfully tagged caddy_example_issue_2407_proxy:latest
Recreating caddy_example_issue_2407_proxy_1 ... done
Attaching to caddy_example_issue_2407_proxy_1
proxy_1  | 2019/01/26 14:53:14 [ERROR] Loading persistent UUID: open /config/uuid: permission denied
proxy_1  | 2019/01/26 14:53:14 [ERROR] Persisting instance UUID: open /config/uuid: permission denied
proxy_1  | Activating privacy features... 
proxy_1  | 
proxy_1  | Your sites will be served over HTTPS automatically using Let's Encrypt.
proxy_1  | By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
proxy_1  |   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
proxy_1  | Please enter your email address to signify agreement and to be notified
proxy_1  | in case of issues. You can leave it blank, but we don't recommend it.
proxy_1  |   Email address: 2019/01/26 14:53:16 [INFO] [asdas.net] acme: Obtaining bundled SAN certificate
proxy_1  | 2019/01/26 14:53:17 [INFO] [asdas.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/XLZWTOvzP3VzgQt0MHKpZhI8bEHZTGzXJc5apRbCNag
proxy_1  | 2019/01/26 14:53:17 [INFO] [asdas.net] acme: use tls-alpn-01 solver
proxy_1  | 2019/01/26 14:53:17 [INFO] [asdas.net] acme: Trying to solve TLS-ALPN-01
proxy_1  | 2019/01/26 14:53:22 [asdas.net] failed to obtain certificate: acme: Error -> One or more domains had a problem:
proxy_1  | [asdas.net] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challenge, url: 
caddy_example_issue_2407_proxy_1 exited with code 1

Excellent work investigating so far!

That has led me to a place where it may be possible the AltTLSALPNPort setting is getting lost, at least if the tls directive is used at all:

https://sourcegraph.com/github.com/mholt/caddy@fdec3c68f0ba761cff340c78138c5093598b28a5/-/blob/caddytls/setup.go#L67 - this is called when Caddyfile directives are being executed. Directives are not executed until after InspectServerBlocks, where the value is initially set. Thus, perhaps the value is being lost there.

However, I haven't confirmed this, and remember that we still have the weird problems trying to reproduce this from inside a container or outside (i.e. I do not experience the bug building on my Mac, as I have been using these flags for quite some time in dev, successfully), which I'm not sure if this explains it.

Keep in mind that this is conditional upon the tls directive being present.

Hope this helps, let me know if you can pinpoint it!

Edit: Stupid "close and comment" button is so close to "comment", sorry

So you are correct with the cause being that line. I am creating a Pull Request but is there a reason that line was there in the first place, It just seems useless since the config.Manager is already set.

Not that I can remember... I'm interested to see the pull request, and for a confirmation or test that it fixes your use case (with Docker) as well as not breaking mine (which already works). Have we figured out why it only doesn't work when built inside Docker?

I just noticed that it sets the cache so it required. I am just going to have it parse the existing config through.

Not sure why it only doesn't work in Docker. The other thing that is weird is that the HTTP port is working even though both ports are attached to the same config struct.

@mholt what environment are you using because I am having the same problem on my Movjave MacOS machine. I don't think this is a Docker environment related issue, just that Docker is a use case for the non-default HTTP and https ports. Also, do you have any clue why this is only a problem with the HTTPS port?

I'm on Mojave as well.

Is the only difference the fact that the Caddyfile in the container used the tls directive and the Caddyfile outside the container does not? (Remember, we have determined that the presence of the tls directive triggers this bug.) I'm still not clear on your exact Caddyfiles in each scenario.

Also, do you have any clue why this is only a problem with the HTTPS port?

Nope. Maybe we don't fully understand this yet, then.

https://git.example.com {
  tls [email protected] {
    ca https://acme-staging-v02.api.letsencrypt.org/directory
  }
  status 500 /
}

https://jenkins.example.com {
  tls [email protected] {
    ca https://acme-staging-v02.api.letsencrypt.org/directory
  }
  status 500 /
}

It was happening without the Lets Encrypt staging environment but it enabled just not to hit rate limits.

If I change my Caddyfile to be:

https://git.example.com {
  status 500 /
}

https://jenkins.example.com {
  status 500 /
}

and parse the -email and -ca when I start the server, I am still getting the same error.

Then perhaps something else is going wrong. Does your PR fix the issue for you? And if so, why? (EDIT: actually, we should check if the tls directive is getting injected before executing directives, if it is missing. I can't remember but that might be happening)

My understanding is that the port is being lost in between it being set and used by Certmagic. My PR parses the existing value of the port through when the cache is added to Certmagic. What is still confusing me is how the HTTP port works but HTTPS doesn't and why your environment works and mine doesn't.

Can you clarify whether your PR fixes the issue for you, too?

We will probably need to understand these things. :) As far as I understand, there are a few people now who can replicate both scenarios. You, @whitestrake, and @rmoriz I believe.

Sorry, that was a bit unclear. Yes it does fix the issue. I will upload a binary later with my change and @whitestrake and @rmoriz can test it.

I have compiled the binaries for my PR. They can be downloaded from here or you can compile them yourself from the source here. I have also included my Caddyfiles (with only the domains & emails changed) and the commands I used to start Caddy. I have verified the Linux binary (In Docker) (I have not tested the Mac binary. My Mac is having unrelated problems). When running Caddy I had my router route the WAN port 80 & 443 to 8080 & 8443 respectively on the device running the code and the desired output is being able to navigate to both domains defined in the Caddyfile over HTTPS and see an HTTP 500 status page (This is done via the status directive in the Caddyfile, and is not something being broken). The browser gives me an HTTPS warnings due to the use of the staging Let's Encrypt CA but that is irrelevant to this issue.

looks good. I've added a branch to my test case using your commit https://github.com/rmoriz/caddy_example_issue_2407/tree/fix_attempt1

@oscartbeaumont How do we know that the HTTP port is carried over properly? Have you seen it try the HTTP challenge using the alternate HTTP port then?

I have actually have not seen it work with the HTTP port now that I think about it because it has only been using the HTTPS challenge. Sorry, that was an incorrect assumption. Did my fix work for you @rmoriz?

@oscartbeaumont yes. So far it does work. However the re-deployed instance uses the existing LE certificate and the renew will happen in a couple of weeks.

Is it possible for you to create a container and test the issuing of certificates. You can use the Let's Encypt staging environment as show in my Caddyfiles here?

@oscartbeaumont I just configured another, new subdomain. LE works:

https://test-subdomain.asdas.net/

http://asdas.net:8080 {
  redir https://asdas.net{uri}
}

https://asdas.net:8443 {
  tls [email protected]
  limits 1mb
  log stdout
  errors stdout
}

http://test-subdomain.asdas.net:8080 {
  redir https://test-subdomain.asdas.net{uri}
}

https://test-subdomain.asdas.net:8443 {
  tls [email protected]
  limits 1mb
  log stdout
  errors stdout
  status 418 /
}

You can force the HTTP-01 challenge by using the -disable-tls-alpn-challenge flag, and that can help confirm that the -http-port flag is broken without the fix as well.

Okay, so far this looks like good news. I think everything is in order, except: do we understand why it works built outside of Docker but not inside?

I don't think that is true. For me it was not working outside of Docker and my change fixed that as well.

All of my reported results in this issue were from caddy built and ran within docker.

Maybe you have cap_net_bind_service set by default on your system. I that case you won't get the permission denied error (nonetheless caddy still using the wrong/default ports)

Correct me if I'm wrong but shouldn't that cause the certificate issuing to fail because it's listening on a different port than what is exposed to the internet but it is something still worth checking.

In both cases LE connects to port 80/443. In the working case, a firewall/load balancer/external tool maps the ports, in the broken state, caddy binds to the ports itself (if it runs as root or got the sys cap).

When I did my earlier testing, my finding was that the binary built inside Docker would fail in this manner regardless of whether the binary was executed in a container or directly on the host (I specifically extracted the binary from the container and tried it on the host, to the same effect).

Binaries built outside of Docker, including official binaries from the Caddy website, did not seem to exhibit the issue.

If everyone could please use #2452 and help test it, and report back, that would be great.

Great! We'll hopefully merge that very soon. Thanks.

Likewise worked for me.

Hello! I think the issue is still alive.

root@mattermost:~# cat kwm*

https://example.test.com:8443 {
    errors stderr
    log stdout

    tls [email protected]

    root ./www

    # kwmserverd API v1 websocket
    proxy /api/v1/websocket 127.0.0.1:8778 {
        websocket
    }

    # kwmserverd API v1 (full, including admin)
    proxy /api/v1 127.0.0.1:8778

    # kwmserverd API v2
    proxy /api/kwm/v2/ 127.0.0.1:8778 {
        websocket
    }
}
root@mattermost:~# caddy -conf kwm*
Activating privacy features... done.
2019/02/14 14:01:21 Listen: listen tcp :80: bind: address already in use

root@mattermost:~# caddy -version
Caddy 0.11.3 (non-commercial use only)

@s1lviu that's a different issue. This was specifically related to the -https-port CLI flag.

For Let's Encrypt certificate issuance to work, port 80 needs to be available. See here: https://caddyserver.com/docs/automatic-https

@s1lviu If you cannot bind to port 80 and do your own mapping to be reachable on port 80 (using external, docker, iptables methods), you'll have to specify a non-tls part like in my example in this issues' opening post https://github.com/mholt/caddy/issues/2407#issue-393792153 (and start caddy with the right -http-port that you are mapping port 80 to.)

Was this page helpful?
0 / 5 - 0 ratings

Related issues

wallacyyy picture wallacyyy  Â·  53Comments

jpoehls picture jpoehls  Â·  54Comments

roblabla picture roblabla  Â·  45Comments

miekg picture miekg  Â·  95Comments

vicanso picture vicanso  Â·  57Comments