Caddy: v2: HTTP-only wildcard sites still redirect to HTTPS if another site is configured with automatic HTTPS

[lukasbestle]

Caddy version: v2.0.0-beta9.0.20200401064340-809e72792c50 h1:r0vSakzB/zmOVhKZINtlCNi1NfaagKG2NP7oAZUSjL4=

Caddyfile:

tls.test {
   tls internal

   respond "HTTPS"
}

http://* {
  respond "HTTP"
}

Expected behavior: Requests to http://tls.test should be automatically redirected to https://tls.test, while requests to any other site that matches the second site block should be served via HTTP only and not redirect to HTTPS.

Actual behavior: Redirection occurs. If you comment out the first site block, it works again. So the issue only occurs if there is also a site listening on HTTPS.

[Henrocker]

@lukasbestle Can you reproduce this behaviour with latest beta 20? Your version is beta 9.

[lukasbestle]

@Henrocker No, it's a build of commit 809e72792c50, which is more recent than beta 20.

[Henrocker]

@lukasbestle I'm a little bit confused because your Caddy version clearly states that you're on beta 9:

Caddy version: v2.0.0-beta9.0.20200401064340-809e72792c50 h1:r0vSakzB/zmOVhKZINtlCNi1NfaagKG2NP7oAZUSjL4=

Can you download one of the attached precompiled versions of Caddy from https://github.com/caddyserver/caddy/releases/tag/v2.0.0-beta.20?

[lukasbestle]

I don't blame you, I was confused as well. But that seems to be because go build's version detection is broken.

Take a look at the part behind the last dash: There's the commit hash 809e72792c50. If you don't believe me, try building Caddy yourself from any recent commit hash of the master branch.

[mholt]

That's a sorting bug in Go modules. I think it will get fixed in rc1 soon. But the commit follows the tag which is correct.

[Henrocker]

@mholt Ahh okay, got it.

[mholt]

@lukasbestle

http://*

That matches only single-label domain names like localhost or foobar, etc. Similarly, http://*.example.com matches only foo.example.com (etc.) but not foo.bar.example.com.

To catch all hostnames, omit the *.

Edit: to clarify, I verified this by using your config first and trying it out :) Removing the * works as expected.

[lukasbestle]

Ah, thanks!

I tried it with the following configuration:

tls.test {
   tls internal

   respond "HTTPS"
}

http:// {
  respond "HTTP"
}

but now the second host overrides the first one completely. I don't get a redirect from http://tls.test to https://tls.test anymore. Caddy throws the warning user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects, but I'd still expect the first matching matcher to match its requests.

What I want to achieve is basically this: I have a dynamic list of hosts that are imported from different files. Every request to a host that was not defined should be responded with an error message.

[mholt]

but now the second host overrides the first one completely. I don't get a redirect from http://tls.test to https://tls.test anymore. Caddy throws the warning user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects, but I'd still expect the first matching matcher to match its requests.

Well, this does make sense though, because you're saying that all HTTP requests should be answered with respond "HTTP" -- the redirects _are_ actually encoded into the internal config, but are never being reached because your HTTP config is a catch-all. We absolutely never, ever must override explicit user configuration, because setting explicit configuration is how the user overrides the implicit configuration.

[lukasbestle]

Hm, I would have expected that sites are matched top-down, meaning the first site that matches a request will serve it. So in this case the protocol-less tls.test host should match both HTTP as well as HTTPS requests.

What would my config need to look like to make it work like I need?

[mholt]

I realize that, but this is how auto-HTTPS works (greatly simplified version): Domain names are extracted from the config's host matchers. These are then used to create redirect routes in a server on the HTTP port. From the docs, this is how routes work:

Routes are executed sequentially. First a route's matchers are evaluated, then its grouping. If it matches and has not been mutually-excluded by its grouping, then its handlers are executed sequentially. The sequence of invoked handlers comprises a compiled middleware chain that flows from each matching route and its handlers to the next.

Because auto HTTPS is implicit behavior, we always _append_ the redirect routes to any explicitly-configured routes on that server. As far as I know, it's practically infeasible to insert the implicit routes higher in the route list while guaranteeing that it won't interfere with the user's configuration.

Besides, here the user has specified that all HTTP sites should be handled in a certain way (respond "HTTP"). If it was filtered by some sort of host matcher, you'd get the behavior you want, since all other domains wouldn't match your explicit catch-all configuration. In other words, don't use a catch-all HTTP site, specify the names you want to capture.

Why are you even configuring sites on the HTTP port anyway?

[lukasbestle]

That all makes sense and I get why it works like it does. Thanks for the explanations!

Why are you even configuring sites on the HTTP port anyway?

The only route I have configured explicitly on the HTTP port is the fallback route. There's no way around that – I can't have HTTPS on an arbitrary number of domains, otherwise Caddy would have to generate certificates like crazy. Of course it would be great if Caddy would be able to serve HTTPS for the fallback, but that's just not feasible. Which is why I configured the fallback route to only listen to HTTP requests (which is already better than nothing).

Basically the use-case is the following: I have a dev server that will serve projects in the local network. A local DNS server ensures that all requests to *.test will be routed to that local dev server. Caddy should now do this for each request, in order:

1. A specific list of hosts is configured manually, i.e. I have a site for each project. The hosts are configured by domain (for example with project.test, *.project.test). If the request matches those domains, the route should be executed (which means: redirect from HTTP to HTTPS and then serve whatever is configured for the site).
2. If the request does not match any site, there should be a simple error response.

How can I achieve this?

[francislavoie]

I think the fix would be to make a site block that does the redirect itself instead of relying on the auto-HTTPS redirect.

http://tls.test {
    redir https://{hostport}{uri}
}

[mholt]

@francislavoie Yep, that's what I'm thinking too, from what i said earlier:

If it was filtered by some sort of host matcher, you'd get the behavior you want, since all other domains wouldn't match your explicit catch-all configuration. In other words, don't use a catch-all HTTP site, specify the names you want to capture.

And of course you can combine all your known hostnames into that same block.

To be fair, this is a bit of an edge case I think.

[lukasbestle]

Thanks for the idea, but unfortunately it's not going to work for me this way as the hosts that are defined all come from individual files that are imported with import /path/to/vhosts/*.conf. So I don't have a global list of hosts for which I could define such a site block. I could add one in each separate file, but that's a bit too much of a hack TBH. Looks like I will need to go without a fallback route for now...

I understand that you can't implement every feature request, but TBH it surprises me that fallback routes are an edge case. What do you guys do with requests to non-existing sites?

[mholt]

I think we mean that each host can just make their own redirect in their own .conf files?

[mholt]

Can you post your full config? That will help understand what you're trying to do better.

[lukasbestle]

Yes, but:

I could add one in each separate file, but that's a bit too much of a hack TBH.

I'm not necessarily the one who writes those conf files and I don't want to put a big "leave this here as is for technical reasons" disclaimer around it.

[lukasbestle]

Sure:

Caddyfile:

# Basic vhost settings
(basic) {
  tls internal
  encode gzip
}
(file_server) {
  import basic
  file_server
}

# PHP vhost settings
(php-72) {
  import file_server
  php_fastcgi unix//usr/local/var/run/php-fpm.72.sock
}
(php-73) {
  import file_server
  php_fastcgi unix//usr/local/var/run/php-fpm.73.sock
}
(php-74) {
  import file_server
  php_fastcgi unix//usr/local/var/run/php-fpm.74.sock
}
(php-latest) {
  import php-74
}

# Custom vhosts
import /path/to/vhosts/*.conf

Example vhost:

  example.test,   example.10.1.3.1.xip.io,
*.example.test, *.example.10.1.3.1.xip.io {
  root * /path/to/project
  import php-latest
}

The xip.io hostnames are just there for testing because the local DNS server is not properly configured yet. But I hope you get the idea.

[francislavoie]

So what's the purpose of having this at all?

http:// {
  respond "HTTP"
}

Edit: Another idea, if you know in some backend DB the list of domains you have configured to have https, you could do something like:

http:// {
  reverse_proxy * your-backend:8080
}

and then have your backend do the Location: redirect itself to https if configured. Kinda goofy, but delegates the work away from the Caddyfile.

[mholt]

but TBH it surprises me that fallback routes are an edge case. What

I didn't say fallback routes were an edge case, I said that trying to use implicit behavior while overriding it at the same time is an edge case.

I suppose we could implement _one_ exception to the append-only behavior of our implicit redirect routes. If and only if ("AND" the below conditions together):

• There is only 1 route in the plaintext HTTP server
• That singular route has NO matchers
• The implicit redirect route has at least 1 matcher

... we could probably prepend the implicit redirect route in that case.

Alternate, possibly looser, conditions:

• The LAST route in the plaintext HTTP server has NO matchers
• The implicit redirect route has at least 1 matcher

then we could could insert the implicit redirect route before the last one.

[lukasbestle]

Or what about something like this:

<fallback> {
  respond "This is a fallback route"
}

This would be as explicit as it gets. The syntax is just an example and could be whatever you think is suitable.

[mholt]

Er, what does <fallback> mean? Fallback to what?

Let's go with one of my proposals. Will either of those work for you?

[lukasbestle]

Fallback route if no other route has matched the request. Meaning: If Caddy receives a request it cannot respond to otherwise, this route should be used. That‘s the feature I actually need.

Your suggestions might work (I don‘t know the Caddy internals enough to judge that in advance), but why change the internal logic if there is a way the user can explicitly tell Caddy what they want? :)

[mholt]

Fallback route if no other route has matched the request. Meaning: If Caddy receives a request it cannot respond to otherwise, this route should be used.

That's precisely what :80 or http:// do.

It just so happens that it overwrites any implicit logic.

but why change the internal logic if there is a way the user can explicitly tell Caddy what they want? :)

I mean, yes, that's what we've been saying -- you can always do the HTTP->HTTPS redirects yourself. The auto redirects are just an added convenience for 99.99% of sites.

[mholt]

Actually, my proposal is worse than what we have now. Consider this site block:

http:// {
    header Foo Bar
}

Currently, this allows you to set headers, even for the auto-HTTPS redirects. In other words, you can already define a "fallback" route in this manner, it's just a question of whether the fallback route terminates the handler chain.

Yours happens to terminate the handler chain, so no future handlers (such as the implicit HTTP->HTTPS redirects) will be invoked.

So, Caddy already supports fallback routes in the way you want by doing what you're doing with the http:// site block.

My proposal is worse because it assumes that all fallback routes terminate the handler chain by responding to the request, and my proposal would make it impossible to add middleware to the redirects, because it would prepend the redirects (which terminate the chain) rather than append them (and thus invoke the other, user-defined middleware first).

I've been experimenting today with this and that's the conclusion.

I'm still not sure what you want to do (@francislavoie asked "So what's the purpose of having this at all?" but I don't really see where you answered that).

What about:

https://example.test, http://example.test {
    @http {
        protocol http
     }
     redir @http https://{hostport}{uri}
     # other site config here
}

and just do the redirects yourself? Then you can still have your fallback plaintext HTTP site AND redirects. This could probably be expressed in the JSON more succinctly but that's the trade-offs of using the Caddyfile.

I'm just not sure of a good way to satisfy both requirements (either adding middleware to the redirect routes, or overriding the redirects entirely) by changing the internals.

[mholt]

For posterity, in case we ever revisit this issue, the code I used to insert the routes before the user's explicit catch-all route was:

if len(srv.Routes) > 0 && len(srv.Routes[len(srv.Routes)-1].MatcherSets) == 0 {
    // insert our routes just before the last one
    last := len(srv.Routes) - 1
    srv.Routes = append(srv.Routes[:last], append(routes, srv.Routes[last])...)
    srv.Routes = append(srv.Routes, catchAllRedirRoute())
    continue redirServersLoop
}
srv.Routes = append(srv.Routes, routes...)
srv.Routes = append(srv.Routes, catchAllRedirRoute())
continue redirServersLoop

where catchAllRedirRoute() just returns the catch-all redirect route that is necessary to support redirects for yet-unknown hostnames via on-demand TLS (and which always goes at the end -- whereas, the routes in routes all have Host matchers on them).

[lukasbestle]

That's precisely what :80 or http:// do.
It just so happens that it overwrites any implicit logic.

So it's not precisely what :80 or http:// do.

<fallback> (or whatever it would get called) would be the fallback below any other route, including the implicit HTTP->HTTPS redirection logic.

I think both :80/http:// as well as <fallback> have their use-case, it's only a different one:

• If I create a route for <fallback>, I only want Caddy to respond to requests where no other route matches (including any of the implicit routes) – as a user I don't even know at this point how the "magic" HTTP->HTTPS redirection works and I don't want anything about it to change. So basically this would be a feature for the 99.9 % of users who "just" need a fallback route.
• If I create a route for :80/http://, I want explicit control over everything, including the magic HTTP->HTTPS redirect, which means that in this case I would still have to configure the redirect myself if I want one. :80/http:// stays the "advanced mode" for the 0.01 % of use-cases.

By having a separate <fallback> route feature (again, the naming is completely up to you) next to :80/http://, the user can explicitly tell Caddy whether they want to keep the implicit redirection or not, without Caddy needing any heuristics that mess with the handler chain as you are describing.

I'm still not sure what you want to do (@francislavoie asked "So what's the purpose of having this at all?" but I don't really see where you answered that).

My use-case fits the 99.9 % use-case I described above for <fallback>. For me, the fact that the HTTP->HTTPS redirection is disabled if I configure a fallback route is an unwanted side-effect, not a feature. So with the current implementation, I would have to manually re-enable something that I didn't even want to disable in the first place.

I really just want a fallback for requests that didn't match any site, nothing more, nothing less. I have described my use-case in detail in https://github.com/caddyserver/caddy/issues/3212#issuecomment-607441373. If anything is unclear, please let me know which specific part of my use-case you need more info on.

---

BTW: I learned a lot about how Caddy matches sites from our discussion. It would be great if this information was in the docs, maybe even as a separate page.

The Caddyfile docs tell you that you can put your site address into the config, but they don't explain anywhere what the syntax is (explicit protocol definition, what is matched by wildcards and what not, ports etc.) and they also don't explain in which order sites are matched (including that Caddy provides implicit routes, what those routes do, what they are overridden by and where in the chain the implicit HTTP->HTTPS redirection happens).

Some parts are documented deep inside the "JSON config structure" docs, but if you don't use JSON configuration, you won't even know where to look in the JSON tree. A separate page or a section in the Caddyfile tutorial would help people understand how Caddy site matching works in a broader context.

[mholt]

@lukasbestle

So it's not precisely what :80 or http:// do.

Well, it _is_ what they do, but your definition of "fallback" includes implicit/hidden behavior, whereas Caddy's does not: explicit config should always be able to override something that is hidden. I suppose we could consider a new config property in the JSON where you can describe routes that are specifically inserted _after_ the auto-HTTPS redirects, but... redirects are simple enough to configure explicitly anyway.

One problem with this discussion's trajectory is that it's rooted in the Caddyfile, but the auto-HTTPS logic happens independent of config adapter: it is implemented at the core of the HTTP server, no matter how the config is represented; it all boils down to unmarshaling JSON and setting up from that. So discussing solutions revolving around the Caddyfile can't be productive.

What you're trying to do might not be able to be expressed in the Caddyfile, anyway. Explicit redirects (as opposed to torquing the implicit redirects) are simple enough to construct with the JSON. Let me demonstrate.

I took the liberty of spending a few minutes and crafting a JSON(C) config from scratch that _I think_ does what you want, and even commented it for you (forgive GitHub's ludicrous tab width of 8, it's actually very digestible at 2 or 4):

{
    "apps": {
        "http": {
            "servers": {
                "issue3212_tada": {
                    "listen": [
                        ":80",
                        ":443"
                    ],
                    "routes": [
                        {
                            // this first route is for all recognized hostname
                            "match": [
                                {
                                    "host": [
                                        // put all your recognized hostnames in this list
                                        "tls.test"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    // all recognized hostnames will be served by this subroute
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            // if HTTP, redirect to HTTPS
                                            "match": [
                                                {
                                                    "protocol": "http"
                                                }
                                            ],
                                            "handle": [
                                                {
                                                    "handler": "static_response",
                                                    "status_code": "308",
                                                    "headers": {
                                                        "Location": [
                                                            "https://{http.request.hostport}{http.request.uri}"
                                                        ]
                                                    }
                                                }
                                            ],
                                            "terminal": true
                                        },
                                        {
                                            "handle": [
                                                {
                                                    "handler": "static_response",
                                                    "body": "🎉 HTTPS"
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "terminal": true
                        },
                        {
                            // this route is for all requests with unrecognized hostnames
                            "match": [
                                {
                                    // this matcher might not even be required,
                                    // depending on your setup and preferences
                                    "protocol": "http"
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "static_response",
                                    "body": "Fallback 👍"
                                }
                            ]
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        // can specify "subjects" here if you only want some names issued internal certs
                        "issuer": {
                            "module": "internal"
                        }
                    }
                ]
            }
        }
    }
}

I think this is quite an elegant configuration: each hostname only has to be specified once. If I'm wrong, maybe this config needs to be tweaked a bit, but that's certainly close to the right idea, I imagine? (I haven't tested it.)

Please let me know how far this gets you.

What you wanted to do goes against the grain of the Caddyfile's benefits. It's designed for common configs of simpler sites. It always has been, that's why the Caddyfile is so easy to use. But as you can see, such an elegant config as I provided here with JSON cannot be expressed in the Caddyfile (as far as I know). That's a limitation of the Caddyfile, and that's just how it is.

The Caddyfile docs tell you that you can put your site address into the config, but they don't explain anywhere what the syntax is (explicit protocol definition, what is matched by wildcards and what not, ports etc.)

Sure they do: https://caddyserver.com/docs/caddyfile/concepts#addresses

and they also don't explain in which order sites are matched

It's first matching site block. I guess I can add a sentence that says that.

(including that Caddy provides implicit routes, what those routes do, what they are overridden by and where in the chain the implicit HTTP->HTTPS redirection happens).

That's more or less explained here: https://caddyserver.com/docs/automatic-https -- including a simple phrase that says, "Automatic HTTPS never overrides explicit configuration." IMO that is pretty clear.

I'm glad you learned a lot though. :)

[lukasbestle]

Thanks for your detailed reply and the example configuration.

Unfortunately the example configuration doesn't work for me, but for a different reason as you might expect: The end result of Caddy serving the correct responses is achieved, but with a JSON configuration of 92 lines. A human-readable Caddyfile with the same configuration would look like this:

tls.test {
   tls internal

   respond "🎉 HTTPS"
}

<fallback> {
  respond "Fallback 👍"
}

And it's not just about the lines of code either. Caddyfile is an amazing syntax. Not only for simple sites, but also for pretty detailed setups.

Your JSON configuration misses a very important part of my setup: The wildcard import from a vhosts directory. It's incredibly amazing that I can create a new file in this directory with just a few lines of Caddyfile syntax, reload the Caddy config and boom – done! I don't have to write my own tool that manages a database of sites and produces JSON config. I can just write each vhost config by hand or write a simple shell script that generates one with a bit of string concatenation and that's it.

I think you should keep in mind that there are users who have use-cases that depend on the simplicity and awesomeness of Caddyfile. Pointing them to "this is all easily possible with JSON" won't cut it unfortunately.

It's true that Caddy features don't just need to be supported by Caddyfile but also by the core and the JSON config language, but I don't see why a <fallback> feature could not be supported by all layers of Caddy. Either by the Caddyfile adapter generating an appropriate configuration dynamically or by the core itself.

The same applies to all Caddy features: If there is a way to make features available from Caddyfile, more people will benefit from those features for their sites that are generally simple but just need one or two "advanced" features. My <fallback> proposal is an example for a feature that would fit with the simplicity of Caddyfile while empowering users.

It looks like you are not interested in this feature and that's fine. After all it's your decision which features you want to implement. So I'm not going to insist on this any more – it would be a waste of our both's time. But please take away one important lesson from this: Caddyfile is awesome and probably one of the main reasons Caddy is as popular as it is. I wouldn't have made the switch to Caddy if JSON config was the only option. I love that JSON config exists and how flexible it makes Caddy, but it's just not suited for every project.

---

Regarding docs:

Sure they do: https://caddyserver.com/docs/caddyfile/concepts#addresses

I know that section, but it doesn't answer any of the questions I listed:

• It shows examples, but doesn't explain the syntax of addresses.
• It also doesn't explain how wildcards work, it doesn't even show that you can use them inside domains or that they don't match dots in domains.

It's first matching site block. I guess I can add a sentence that says that.

That's not the whole story though. Once the automatic HTTPS redirect is involved, a block that is further down can override a block that is further up, even if it would actually match the request (as we've seen from this issue).

That's more or less explained here: https://caddyserver.com/docs/automatic-https -- including a simple phrase that says, "Automatic HTTPS never overrides explicit configuration." IMO that is pretty clear.

That does not explain what explicit configuration means (= under which specific conditions will a route disable Automatic HTTPS?).

Also: The sentence is about Automatic HTTPS in general, but what about the redirection in particular? That's not mentioned at all at the moment, which made me open this issue in the first place because I didn't know that the redirect is implemented using an implicit route. At which point the docs again don't tell you in which cases the implicit route gets overridden etc. etc.

[francislavoie]

I had an idea that might help solve this.

From my understanding, all that's needed to rebuild a new redir directive in the catch-all is the list of the hosts that should be redirected to, right?

I think during the auto HTTPS phase, we could set some placeholder in the replacer like {http.autohttps.redir_hosts} that contains an array of all the hosts that were determined to need an HTTP->HTTPS redirect. Then, if users are overriding the fallback route with http://, then they could do something like this:

http:// {
  @shouldRedir {
    expression {host} in {http.autohttps.redir_hosts}
  }
  redir @shouldRedir https://{host}{uri} 308

  respond "HTTP"
}

This would use a CEL expression matcher to determine whether the redirect should happen for any given host, then do the redirect if so.

This should work just fine in the JSON as well, for the same reasons.

@mholt do you think this is viable?

[francislavoie]

@lukasbestle I have a POC in https://github.com/caddyserver/caddy/pull/3246 that you can try out and see if this does what you want. You can find build artifacts here: https://github.com/caddyserver/caddy/actions/runs/73972069

Edit: Nevermind, it won't work yet 😜

Edit2: Alright I think I have it working now! https://github.com/caddyserver/caddy/actions/runs/74092434

[lukasbestle]

I think this is a good solution from the technical perspective, but I still think that it is confusing for end-users who don't know about 100 % of the inner-workings of Caddy.

As I have explained above, users won't even know from the current docs that the automatic HTTP->HTTPS redirect is implemented using an implicit route and that the redirect is therefore overridden by a custom route on the HTTP port.

In addition to more docs on this as I have explained above, maybe there could be a Caddyfile shortcut on top of #3246:

http:// {
  redir_https

  respond "HTTP"
}

[francislavoie]

A shortcut like that is possible, but I don't think it's warranted unless we see that this usecase is actually common, which I don't think it will be, frankly.

We definitely would document this as an example somewhere (likely on the auto HTTPS page) and if anyone needs help with it, they can just ask us on the forums.

[carlwgeorge]

I have a similar use case. In the Fedora RPM package for Caddy, I'm shipping a welcome page and a default configuration similar to this:

http:// {
    root * /usr/share/caddy
    file_server
}
import Caddyfile.d/*.caddyfile

This allows a user to install the package, start the daemon, and have a working web server. In Caddy v1, adding additional blocks under Caddyfile.d would work as expected. But in Caddy v2, those additional blocks do not have HTTP to HTTPS redirection, as described in https://github.com/caddyserver/caddy/issues/3212#issuecomment-607405018. My expectation was that the hostname was matched before the protocol, but after reading through this issue I can see that is not the case.

Could a handler be added above any "catch-all" (http:// or :<http port>) that does the redirects? I think that would allow the desired behavior.

[mholt]

@carlwgeorge

My expectation was that the hostname was matched before the protocol

What do you mean by this exactly? The protocol is HTTP; or are you referring to the distinction between HTTP and HTTP delivered over TLS?

Could a handler be added above any "catch-all" (http:// or :) that does the redirects? I think that would allow the desired behavior.

As I mentioned above, we need to first have this discussion in JSON, not Caddyfile, since the auto redirects have nothing to do with the Caddyfile config adapter.

As it stands, though, adding explicit redirects is very easy; any reason that will not work?

[francislavoie]

I'll reopen this discussion because I think there's enough valid complaints about this for us to find an appropriate solution.

The HTTP->HTTPS redirect part of the Auto HTTPS logic currently appends the route that does the redirect to the existing list of routes for whatever server listens on the HTTP port. This means that any routes that don't have host matchers will take precedence over the redirect routes.

Instead, I think we could be a bit smarter about it and insert the redirect routes just before the first route found in the HTTP server that does not have a host matcher.

For example with this Caddyfile:

http:// {
        respond "Foo"
}

http://foo.com {
        respond "Baz"
}

abc.com {
        respond "Bar"
}

We get the following JSON. I added comments explaining where routes are currently being added, and where I think we should have them added ideally:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "foo.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Baz",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            },
    // this is where I think the redirect routes should be inserted
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Foo",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
    // this is where the redirect routes are currently being inserted
          ],
          "automatic_https": {
            "skip": [
              "foo.com"
            ]
          }
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Bar",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

When this thread was first written, the auto_https disable_redirects global option wasn't available, but now that this is an option, there's recourse for those who need full control over HTTP routes.

The tricky bit with the implementation is that we would need to read the raw matcher sets which are json.RawMessage types to find whether they have host matchers. I'm not sure what the best approach for that would be.

[carlwgeorge]

What do you mean by this exactly? The protocol is HTTP; or are you referring to the distinction between HTTP and HTTP delivered over TLS?

Yes, by protocol I meant http:// or https://.

As it stands, though, adding explicit redirects is very easy; any reason that will not work?

Yes, it's easy to accomplish with explicit redirects. But it's even easier for it to be the automatic behavior, as it was in v1.