Caddy: HTTP port 80 not listening despite Caddyfile global option `auto_https disable_redirects`

[polarathene]

I have been using Caddy locally on port 80 fine, I wanted to try HTTPS locally which afaik requires using local certs, this works but for some reason continues to redirect to HTTPS despite the auto_https disable_redirects global option to prevent that.

Config (Caddyfile):

{
  local_certs
  auto_https disable_redirects
}
localhost

root * /usr/share/caddy

file_server

I run this via the official DockerHub image like so:

docker run -p 8000:80 -p 443:443 -v $PWD/public:/usr/share/caddy/ -v $PWD/caddy/data:/data -v $PWD/caddy/Caddyfile:/etc/caddy/Caddyfile caddy

Log output confirms that it's only listening on HTTPS:

2020/08/05 02:24:21 [INFO][cache:0xc00012a1e0] Started certificate maintenance routine
{"level":"info","ts":1596594261.8360162,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}

Related logs following:

{"level":"info","ts":1596594261.8360522,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["localhost"]}
{"level":"warn","ts":1596594261.878241,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/08/05 02:24:21 define JAVA_HOME environment variable to use the Java trust
2020/08/05 02:24:21 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/08/05 02:24:21 certificate installed properly in linux trusts
{"level":"info","ts":1596594261.9039123,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1596594261.9040866,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
{"level":"info","ts":1596594261.9041824,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1596594261.90419,"msg":"serving initial configuration"}
2020/08/05 02:24:21 [INFO][localhost] Obtain certificate; acquiring lock...
2020/08/05 02:24:21 [INFO][localhost] Obtain: Lock acquired; proceeding...
2020/08/05 02:24:21 [INFO][localhost] Certificate obtained successfully
2020/08/05 02:24:21 [INFO][localhost] Obtain: Releasing lock
2020/08/05 02:24:21 [WARNING] Stapling OCSP: no OCSP stapling for [localhost]: no OCSP server specified in certificate

So naturally..

curl localhost:8000
curl: (56) Recv failure: Connection reset by peer

Am I doing something wrong or unsupported here or is this a bug?

[polarathene]

With my working HTTP config prior to using HTTPS, my global config was auto_https off and instead of localhost I had :80.

It seems for just HTTP that localhost:80 works, but not 127.0.0.1:80(that's ok for not working due to running within Docker iirc) or 0.0.0.0:80(which seems a bit odd?). Just using localhost with HTTPS disabled doesn't seem to work, so perhaps I need to explicitly add port 80 to the config still?

[francislavoie]

This is working as expected. Caddy defaults to only listening on port 443 (HTTPS) unless automatic HTTPS rules allow for setting up the HTTP redirect.

If you want to explicitly serve HTTP alongside HTTPS without the redirect, you need to tell Caddy to explicitly do so.

Here's what your Caddyfile above adapts to, and as you can see, it only listens on :443.

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "localhost"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "vars",
                          "root": "/usr/share/caddy"
                        },
                        {
                          "handler": "file_server",
                          "hide": [
                            "Caddyfile"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "automatic_https": {
            "disable_redirects": true
          }
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuer": {
              "module": "internal"
            }
          }
        ]
      }
    }
  }
}

The solution would be to use a Caddyfile like this:

{
    local_certs
    auto_https disable_redirects
}

http://localhost, https://localhost {
    root * /usr/share/caddy

    file_server
}

I recommend taking a look at the caddy adapt --pretty output after doing so to compare. You'll notice Caddy expands your config into two distinct server blocks because http and https have slightly different semantics internally.

For next time, please ask your usage questions on the Caddy community forums. We prefer to keep the GitHub issue board for bugs and feature requests. Don't forget to fill out the thread template so we can help you!

[polarathene]

We prefer to keep the GitHub issue board for bugs and feature requests.

@francislavoie it definitely felt like a bug. I had looked over docs regarding this global config option, there is no mention that disable_redirects requires manually specifying both http and https protocols, if no explicit port is provided then the hostname should be sufficient(that's the point of the default HTTP/HTTPS port configs right?)

I can raise this as a feature request, or as a documentation issue?

https://caddyserver.com/docs/automatic-https#effects

The default port (if any) is changed to the HTTPS port 443
HTTP is redirected to HTTPS (this uses HTTP port 80)

As it states after that, explicit config won't override, so if I provide :80 or localhost:80 HTTPS will be available but no redirect, that's fine. As quoted text clarifies, and I have confirmed, providing an address of localhost will redirect HTTP to HTTPS, or I can access HTTPS directly. Thus, proper UX with disable_redirects is to implicitly support both HTTP and HTTPS right...?

Unclear if the auto_https is meant to avoid redirect for addresses that specifically use the HTTP configured port as you still have docs that state this:

https://caddyserver.com/docs/automatic-https#overview

Caddy keeps all certificates renewed, and redirects HTTP (default port 80) to HTTPS (default port 443) automatically.

There is another section which also points out that explicit port for server address negates the features: https://caddyserver.com/docs/automatic-https#activation

Listening exclusively on the HTTP port

That gotcha isn't touched on what is probably a more commonly discovered doc regarding HTTPS support: https://caddyserver.com/docs/quick-starts/https

Understandably because in most situations it'd be under implicit config for a domain with no explicit ports, and rare that someone setting up Caddy for such would want HTTP accessible without a redirect (I was trying to verify/test a browser behaviour that differs between HTTP and HTTPS).

---

While this is an unlikely use-case in production (I'm only doing it for local development / testing), it'd be nice if this statement would be true for disable_redirects for Automatic HTTPS:

You won't have to know or do anything else about it. It should "just work"!

---

Summary / TL;DR

• disable_redirects should be compatible with a single hostname address without needing to explicitly add protocols or ports, just like the default automatic https feature works implicitly.
• Docs take a bit more digging to grok why this feature requires being more explicit, and why port 80 (when matching the default HTTP port config) doesn't automatically forward. Could mostly be avoided by proper implicit support, or earlier clarification about explicit ports breaking the feature(and regular redirects too).

[polarathene]

Don't forget to fill out the thread template so we can help you!

Was this regarding the community forum? If it was about the preceding github issues purpose, opening an issue provides no issue template selection, just blank.

[francislavoie]

Was this regarding the community forum? If it was about the preceding github issues purpose, opening an issue provides no issue template selection, just blank.

Yes.

If you disable redirects, it turns off this part:

• HTTP is redirected to HTTPS (this uses HTTP port 80)

Which means it won't listen on port 80 at all (except for the ACME HTTP challenge but that's separate from what Caddy will serve). That's the intended behaviour. Caddy is _secure by default_ which mean it won't serve HTTP implicitly. If you want your site served over HTTP (which is unsecured), you need to configure it as such.

You're right though, the Automatic HTTPS doc doesn't specifically mention what happens when you disable redirects. That could be clarified. That page was written before that option existed.

[polarathene]

Caddy is secure by default which mean it won't serve HTTP implicitly.

Right.. but it doesn't use disable_redirects by default.

I think the confusion is resolved with the default port being changed to HTTPS instead of HTTP, such that only one port was configured for serving my content from the address, the redirection on port HTTP/80 was implicit and unrelated to that.

Not sure why, but I was under the impression that disabling redirects would implicitly use both the HTTP and HTTPS ports for an address that provided none, or that if I explicitly provide a port, especially one that matches the default HTTP configured port, that automatic HTTPS would still be supported, just not redirect from the HTTP port.

Instead it was effectively the same as auto_https off, the logged output about only listening on HTTP port could have better pointed out that explicit port declaration conflicted with auto_https disable_redirects being able to provide HTTPS implicitly.

I possibly assumed HTTP was still implicitly listened with this feature, with the redirect only serving content from HTTPS, but that auto_https off allowed for HTTP default, that auto_https disable_redirects only disabled redirects but allowed for both HTTP/HTTPS as-is... if that makes sense.

With other servers/proxies similar features I think I recall having the redirect as a feature you enable/enforce, and that with Caddy all that was different was a smarter, more secure default choice to redirect by default. Other new Caddy users may also have that assumption and end up confused like I did.

You're right though, the Automatic HTTPS doc doesn't specifically mention what happens when you disable redirects. That could be clarified.

Probably would be helpful :sweat_smile: Perhaps a clear table to visualize the impact of what's happening? How about something like this:

| auto_https | HTTP | HTTPS |
|-------------------|-------------|-------------|
| default | redirects | connects |
| disable_redirects | unreachable | connects |
| off | connects | unreachable |

NOTE: Explicit ports for the address opt-out of automatic https and is equivalent to auto_https off
NOTE: If using disable_redirects, to make HTTP reachable, you need to explicitly prefix your address with both protocols, http://localhost, https://localhost {} see Addresses section for configuring multiple addresses.
NOTE: The default implicit port for an address is the HTTPS port, auto_https off changes that to the HTTP port.

[francislavoie]

Instead it was effectively the same as auto_https off, the logged output about only listening on HTTP port could have better pointed out that explicit port declaration conflicted with auto_https disable_redirects being able to provide HTTPS implicitly.

Where do you see that? The logs say that for HTTPS, not HTTP. To quote your logs:

{"level":"info","ts":1596594261.8360162,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}

With other servers/proxies similar features I think I recall having the redirect as a feature you enable/enforce, and that with Caddy all that was different was a smarter, more secure default choice to redirect by default. Other new Caddy users may also have that assumption and end up confused like I did.

Yeah, generally we ask users to throw away their assumptions about other servers, because Caddy takes a distinctly different approach with many things, to have better defaults and more reasonable assumptions (or lack of assumptions where appropriate in comparison to other servers).

That table looks pretty good. If you want to propose the change on the docs, go for it and we can work to refine it from there. Make sure to keep in mind that technically JSON is the primary configuration language while Caddyfile is the user-friendly layer on top, so the Automatic HTTPS docs page should not assume the user is using the Caddyfile config.

[polarathene]

Where do you see that? The logs say that for HTTPS, not HTTP. To quote your logs

It's not in the logs I provided because that was for the config with address localhost, thus the implicit port to listen on is HTTPS and since HTTP isn't being redirected it's no longer reachable.

If I listen to localhost:80, as I did with with auto_https off, then auto_https disable_redirects will not enable HTTPS as I had assumed, and the warning in logs state only listening to HTTP port not HTTPS. Thus in that case it's like auto_https off, the log output just doesn't clarify that HTTPS isn't being listened to due to auto_https not being valid config with an explicit port.

It kind of tries to communicate the issue, but I didn't know at the time it was because I had provided an explicit port. Probably better to use a level of warning when auto_https is enabled, stating that auto_https is ignored for the server due to explicitly configured port.

{"level":"info","ts":1596600528.7456336,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}

I'm not experienced with Go, so not something I'd like to tackle a PR for, should I raise an issue about it?

---

Yeah, generally we ask users to throw away their assumptions about other servers, because Caddy takes a distinctly different approach with many things

That's fair, although for users like me, it helps when the docs aren't assuming things like certain users giving Caddy a try might be, it's been a common occurrence for me with Caddy's docs, which usually cover what I need to know somewhere but sometimes that info seems scattered/buried (as in not obvious where it is or covered on a long doc page filled with paragraphs).

I'll chalk it up to my learning style not being a good match and just general learning curve or prior bias assumptions.

---

If you want to propose the change on the docs, go for it and we can work to refine it from there.

Will do :+1:

Make sure to keep in mind that technically JSON is the primary configuration language while Caddyfile is the user-friendly layer on top

As a new user and with the docs content, Caddyfile has been easier to approach. Perhaps if in future the docs can have code snippets managed like Traefik docs with tabbed snippets for different config examples, it'd be different. The Caddyfile examples do have less noise, while also letting me use comments to experiment with. JSON makes more sense once I'm not experimenting / learning as much.

so the Automatic HTTPS docs page should not assume the user is using the Caddyfile config.

I'd need to go through the docs more and try the JSON config out I guess as I saw it's JSON config page for automatic HTTPS and it has different settings that afaik are missing in the Caddyfile, and at a glance wasn't entirely clear to me how it was meant to be used.

[francislavoie]

If I listen to localhost:80, as I did with with auto_https off, then auto_https disable_redirects will not enable HTTPS as I had assumed, and the warning in logs state only listening to HTTP port not HTTPS. Thus in that case it's like auto_https off

Well not exactly, there's a distinction between auto_https off and auto HTTPS just not being activated due to it not matching the rules. The former will turn it off globally (when considering the Caddyfile - JSON allows disabling per server) whereas the later is only for that one server. auto_https disable_redirects won't force activate auto HTTPS on anything that doesn't match the rules in the first place, so it has no effect (which is different than explicitly disabling the feature).

Probably better to use a level of warning when auto_https is enabled, stating that auto_https is ignored for the server due to explicitly configured port.

I don't think we can call it a warning if it could literally be the intended configuration by the user. Because then we'd get complaints saying "why is it warning when it's doing exactly what I want?" etc. info is correct here. It's just a "hey user, FYI, this-"

I'd need to go through the docs more and try the JSON config out I guess as I saw it's JSON config page for automatic HTTPS and it has different settings that afaik are missing in the Caddyfile, and at a glance wasn't entirely clear to me how it was meant to be used.

I'm not sure how to answer this because it's not specific enough of a question, but I'd gladly answer if you can clarify the question.