Caddy: v2: forward loop

[fmarcia]

I use a configuration to get a unique favicon.ico which avoids handling it in every web app:

{
    admin off
    email [email protected]
}

(common) {
    header -server
    reverse_proxy /favicon.ico https://www.mydomain.tld
}

app1.mydomain.tld {
    import common
    reverse_proxy 127.0.0.1:3000
}

app2.mydomain.tld {
    import common
    reverse_proxy 127.0.0.1:5002
}

www.mydomain.tld,
app3.mydomain.tld {
    header -server
    file_server
    root * /srv/http/www
}

The reverse_proxy directive of (common) causes a never ending redirection. Here is a line of log, beautified for readability ({ip_address} stands for my real IP address):

{
    "level": "info",
    "ts": 1587401926.2366347,
    "logger": "http.log.access.log1",
    "msg": "handled request",
    "request": {
        "method": "GET",
        "uri": "/favicon.ico",
        "proto": "HTTP/2.0",
        "remote_addr": "{ip_address}:38678",
        "host": "app1.mydomain.tld",
        "headers": {
            "Accept-Language": ["fr,en-US;q=0.7,en;q=0.3"],
            "X-Forwarded-For": ["{ip_address}, {ip_address}, {ip_address}, (and 1682 more times)"],
            "Accept-Encoding": ["gzip, deflate, br"],
            "Cache-Control": ["no-cache"],
            "Accept": ["image/webp,*/*"],
            "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0"],
            "Pragma": ["no-cache"],
            "Cookie": ["gid=5BFNOM11U4OPkip9UhHjS1Vh1uPcjILXIINMuUW1Rf81Q3LGkNjkioYuyvF7zNzZ"],
            "Te": ["trailers"]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "ciphersuite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "www.mydomain.tld"
        }
    },
    "common_log": "{ip_address} - - [20/Apr/2020:18:58:46 +0200] \"GET /favicon.ico HTTP/2.0\" 0 0",
    "latency": 3.228133459,
    "size": 0,
    "status": 0,
    "resp_headers": {
        "Server": ["Caddy"]
    }
}

This was working fine in v1 with this configuration:

proxy /favicon.ico https://www.mydomain.tld
proxy / localhost:5002

I'm not sure if it is a bug or a misconfiguration that should be handled somehow.

[francislavoie]

That's very interesting!

I wonder if this would work as an alternative for now (I would argue this is better if you're running it on the same Caddy instance anyways):

(common) {
    header -server
    route /favicon.ico {
        root * /srv/http/www
        file_server
    }
}

[francislavoie]

I wonder if adding debug to global options would reveal more about what's going on. Could you try that and report back what the logs say?

[mholt]

Posting the full logs would be helpful, too! (minus thousands of repetitions due to the bug -- but I want to see the debug logs, as well as some representative access logs)

Edit: Wow, after a looking closer, that is really weird. I wonder what is going on.

[mholt]

Ahhhh I bet I know why!

By default, the reverse proxy sends all incoming headers through to the upstream (except for certain hop-by-hop headers), including the Host header.

So the favicon's upstream request has a host of app1.mydomain.tld.

If you set the upstream header to be the value of the upstream host instead, that would do it. There's an example for this: https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#examples

I bet that will fix it, so I don't think this is a bug.

I wonder if the behavior of the Host header is unintuitive as-is or if we should keep it. This behavior is _obvious_ when looking at the adapted JSON, but when a full URL is used as is acceptable with the Caddyfile adapter, does the user expect that the Host header might get transformed implicitly? (Edit: on second thought, I don't think changing this behavior is a good idea; best to leave it simple across the board I think.)

[francislavoie]

Yeah I don't think we should change anything - I think you're right that the Host header was the issue (that was my first thought too but I didn't connect all of it together in my mind) 👍

[fmarcia]

Using

    reverse_proxy /favicon.ico https://www.mydomain.tld {
        header_up Host {http.reverse_proxy.upstream.hostport}
    }

solved the problem:

{
    "level": "info",
    "ts": 1587407205.4912179,
    "logger": "http.log.access",
    "msg": "handled request",
    "request": {
        "method": "GET",
        "uri": "/favicon.ico",
        "proto": "HTTP/2.0",
        "remote_addr": "{ip_address}:56632",
        "host": "app1.mydomain.tld",
        "headers": {
            "Cookie": ["sid=G0aJVT9lGcQgi1v0oC0t285gUnnNWDSCuh8jTFVNwCJWeIxcDjXJLMudXKt6GS3I"],
            "Upgrade-Insecure-Requests": ["1"],
            "Te": ["trailers"],
            "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0"],
            "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],
            "Accept-Language": ["fr,en-US;q=0.7,en;q=0.3"],
            "Accept-Encoding": ["gzip, deflate, br"]
        },
        "tls": {
            "resumed": true,
            "version": 772,
            "ciphersuite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "app1.mydomain.tld"
        }
    },
    "common_log": "{ip_address} - - [20/Apr/2020:20:26:45 +0200] \"GET /favicon.ico HTTP/2.0\" 200 15406",
    "latency": 0.004201423,
    "size": 15406,
    "status": 200,
    "resp_headers": {
        "Content-Length": ["15406"],
        "Date": ["Mon, 20 Apr 2020 18:26:45 GMT"],
        "Accept-Ranges": ["bytes"],
        "Etag": ["\"q93n76bvy\""],
        "Last-Modified": ["Mon, 20 Apr 2020 18:25:54 GMT"]
    }
}

Thank you both.

[francislavoie]

Just out of curiosity, any reason you don't use this approach?

[fmarcia]

Well, I didn't think about it! But I'm going to use it. It's more "straight".

Thanks again.

[mholt]

Ah good I'm glad that was all there was to it 😅 otherwise I would have found myself a lot more stressed today. Thanks for following up!