Caddy: v2: reverse_proxy "missing port in address"

[owenconti]

1. Which version of Caddy are you using (caddy -version)?

512b004332ebf6dfa3fd14269de3cb0031233e34

2. What are you trying to do?

Trying to reverse proxy all requests on a subdomain to an external domain:

link.owenconti.com/XXXXX -> https://redirects.ohseemedia.com/follow

3. What is your Caddyfile?

link.owenconti.com {
        reverse_proxy * {
                to https://redirects.ohseemedia.com/follow/
                transport http {
                        tls
                }
        }
}

v1 file (which works):

link.owenconti.com {
    proxy / https://redirects.ohseemedia.com/follow/
}

4. How did you run Caddy (give the full command and describe the execution environment)?

Ubuntu, caddy start

7. What did you see instead (give full error messages and/or log)?

Caddy starts fine but when trying to serve a request:

    ERROR   http.log.error  making dial info: upstream https://redirects.ohseemedia.com/follow/: invalid dial address https://redirects.ohseemedia.com/follow/: address /redirects.ohseemedia.com/follow/: missing port in address  {"request": {"method": "GET", "uri": "/github", "proto": "HTTP/2.0", "remote_addr": "XXXXX", "host": "link.owenconti.com", "headers": {"Sec-Fetch-Site": ["none"], "Sec-Fetch-Mode": ["navigate"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-US,en;q=0.9"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"]}, "tls": {"resumed": false, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "link.owenconti.com"}}}

8. Why is this a bug, and how do you think this should be fixed?

Works fine in v1, can't get it working in v2

9. What are you doing to work around the problem in the meantime?

I'm stuck

Bonus: What do you use Caddy for? Why did you choose Caddy?

Trying to use it over nginx, having troubles :)

[mholt]

Right now in Caddy 2, proxy upstreams have to be network addresses; not URLS.

So:

to https://redirects.ohseemedia.com/follow/

becomes

to redirects.ohseemedia.com:443

However, it's possible we will make this work more like it does in v1.

[owenconti]

@mholt thanks for the quick response!

So that means the proxy upstream has to be to a root path? ie: to redirects.ohseemedia.com:443 worked, but to redirects.ohseemedia.com/follow:443 failed with an "unknown network" error.

I was also trying to use either the redir or rewrite directives, but neither seemed to match the request, and just returned a blank page:

rewrite / https://redirects.ohseemedia.com/follow

rewrite * https://redirects.ohseemedia.com/follow

[mholt]

Caddy 2 network addresses are in this form: network/host:port - hence it thinks redirects.ohseemedia.com is a network type, which it's not.

To rewrite the URL, use the rewrite directive:

rewrite * /follow{uri}

That prepends the URI with /follow. Is that what you're trying to do?

[owenconti]

Caddy 2 network addresses are in this form: network/host:port - hence it thinks redirects.ohseemedia.com is a network type, which it's not.

Okay, that makes sense.

Sorry, I shouldn't have tried using a rewrite, that wouldn't be of use for my use case. Essentially what I am doing is proxying all my requests through an ec2 instance, and then the ec2 instance either redirects or (ideally) proxies to a lambda function (https://redirects.ohseemedia.com/follow is where the lambda function currently sits).

The lambda function looks for two custom headers when receiving a request, so my ec2 instance needs to set these headers before sending the request to the lambda function.

I am able to set the headers with the reverse_proxy approach, the downside being that the lambda function has to reside at the root path.

Otherwise, I am able to send a redir to the lambda function, but I can't figure out how to set the headers with a redir:

link.owenconti.com {
        redir * https://redirects.ohseemedia.com/follow {
                header X-Custom-Header 12345
        }
}

link.owenconti.com {
        redir * https://redirects.ohseemedia.com/follow {
                header_up X-Custom-Header 12345
        }
}

[owenconti]

The more I think about it, the more I believe a redirect won't be sufficient for my use case long term.

So I guess in summary, I'm wondering if the ability to reverse proxy to an external nested path will be coming to v2? ie, from v1: proxy / https://redirects.ohseemedia.com/follow/

[mholt]

Yep, we're working on it. I'm actually discussing some last-minute changes to the Caddyfile with @whitestrake to make this possible.

[owenconti]

Okay awesome!! I'll revert to v1 in the meantime, thanks for the info.

[mholt]

You can still do this in v2, just is different for right now. Try the rewrite, I don't know why you want a redirect unless you want the client to make a new request.

[owenconti]

Ideally the client doesn't have to make the new request.

Could you confirm what the syntax would be using rewrite? According to the docs, one of these should work:

rewrite * https://redirects.ohseemedia.com/follow
rewrite / https://redirects.ohseemedia.com/follow

But for both of them Caddy returns a blank page (no match against the matcher pattern)

[mholt]

Did you try what I posted above? rewrite * /follow{uri}

[owenconti]

rewrite * /follow{uri} results in a rehandle loop because /follow doesn't exist on the same host that Caddy is on.

The /follow endpoint is on a completely separate host, which is why I'm trying to rewrite or proxy to https://redirects.ohseemedia.com

[mholt]

So then combine that with your proxy, but make sure it matches requests to /follow only:

reverse_proxy /follow {
     to redirects.ohseemedia.com:443
     transport http {
          tls
     }
}

[owenconti]

Hmm I must be missing something. We're back to the original error now:

making dial info: upstream https://redirects.ohseemedia.com/follow/: invalid dial address https://redirects.ohseemedia.com/follow/: address /redirects.ohseemedia.com/follow/: missing port in address

Caddyfile:

        reverse_proxy /follow {
                to https://redirects.ohseemedia.com/follow/
                transport http {
                        tls
                }
        }
        rewrite * /follow{uri}

[mholt]

Your Caddyfile is not what I posted above, take another look. :wink:

Oh, and you might need

matcher follow {
   not {
       path /follow
   }
}
rewrite match:follow /follow{uri}
reverse_proxy /follow ...

until our improvements to the v2 Caddyfile are complete.

[owenconti]

Okay, hang on. First, I hate to take your time on this, so I'm more than happy to drop this and resort to v1 until the v2 Caddyfile improvements are made.

Can we take a simpler example? Let's say we want to proxy all client request to https://proxy.host through https://google.com. With v1, that would be:

proxy.host {
    proxy / https://google.com
}

How can I replicate that ^ with v2?

[mholt]

Sorry for the confusion. (This just happens to be one thing that's in motion while we're in beta!)

In v2, that Caddyfile is:

proxy.host {
    proxy google.com:443 {
        transport http {
            tls
        }
}

My hope is that before we leave beta, it will become:

proxy.host {
    proxy https://google.com
}

Right now, the Caddyfile isn't smart enough to figure out that a destination starting with https:// is the same as:

• HTTP transport
• with TLS
• on port 443

[owenconti]

Okay, cool. So that works fine. Now if we take it one step further, v1 version:

proxy.host {
    proxy / https://google.com/search
}

If I'm understanding everything correctly, that can't be done currently with v2 because you can't define the /search path anywhere.

Is that a Caddyfile limit? Would I be able to configure a proxy to https://google.com/search via JSON right now in v2?

[mholt]

What that is _actually_ doing is rewriting the request URI: it's prepending the path with /search. This is arguably not the proxy's job. We have a rewrite directive to do that.

Is that a Caddyfile limit? Would I be able to configure a proxy to https://google.com/search via JSON right now in v2?

Yes and yes. You don't even need to use JSON, you can use the Caddyfile, but you have to use the one I showed you above. It's a little awkward for now until the improvements we're planning are finished.

[owenconti]

So I believe I'm getting Caddy v2 to do what I want via:

        matcher follow {
                not {
                        path /follow
                }
        }
        rewrite match:follow /follow
        reverse_proxy /follow ohseemedia.com:443 {
                transport http {
                        tls
                }
        }

Note that I'm proxying to ohseemedia.com instead of redirects.ohseemedia.com just for testing purposes. ohseemedia.com is a Ghost blog running on a separate ec2 instance. redirects.ohseemedia.com is a AWS lambda behind API Gateway.

With v1, I could proxy to the lambda without issue via:

proxy / https://redirects.ohseemedia.com/follow/

However, with v2, API Gateway returns a 404 response.

I'm wondering if v2 sets/sends different headers than v1? Also, is there a way to enable logging so I can inspect the Caddy to proxy server request to try to debug the issue?

[mholt]

Also, is there a way to enable logging so I can inspect the Caddy to proxy server request to try to debug the issue?

Yep! But we haven't exposed logging config in the Caddyfile quite yet (we will soon). For now, if you drop down to the JSON, you can enable logging at the debug level, and you'll get the exact request that is sent to the upstream: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#logging

[owenconti]

Okay, after a lot of trial and error, I finally got it to work. The lambda checks to ensure the host of the incoming request is "redirects.ohseemedia.com".

When using v1's proxy feature, the host is set correctly. When using v2's proxy, the host is not set.

I was able to manually set the Host header to "redirects.ohseemedia.com" in the reverse_proxy.

I'm not sure if there's a bug or if its working as intended with the v2 proxy?

Either way, thanks @mholt for all of your help, you went much further than you needed to!

[mholt]

Ah, nice find! II will look into that soon, to make sure that the Host header is set in v2.

What should it be set to by default? Should it be set to the incoming Host header value or to the upstream host that it is being proxied to?

[owenconti]

I'm not sure what the right answer is, but I believe v1 was doing "to the upstream host that it is being proxied to"

[owenconti]

One more question actually, is there a way to maintain the origin URI when using rewrite?

So right now I have: rewrite match:follow /follow{uri}

But when that's passed to the reverse proxy, {uri} becomes /follow{uri} (which makes sense). Is there any way to "set" or "store" the original URI value?

[mholt]

I'm not sure what the right answer is, but I believe v1 was doing "to the upstream host that it is being proxied to"

Correct, I just don't know if that's the best default behavior.

Is there any way to "set" or "store" the original URI value?

It is actually set, but I forgot to document it. Use {http.request.orig_uri} instead of {http.request.uri}. (Not sure I love the name "orig_uri" so that might change, but there you have it for now.)

[owenconti]

It is actually set, but I forgot to document it. Use {http.request.orig_uri} instead of {http.request.uri}. (Not sure I love the name "orig_uri" so that might change, but there you have it for now.)

Thanks, that worked!

FWIW, it returned an empty string when used with the caddyfile, but worked correctly with JSON.

[mholt]

What returned an empty string, exactly?

[owenconti]

Correct, I just don't know if that's the best default behavior.

HTTP/1.1 says the proxy should set Host to the service its requesting:

An HTTP/1.1 proxy MUST ensure that any request message it forwards does contain an appropriate Host header field that identifies the service being requested by the proxy.

https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23

When a proxy receives a request with an absolute-form of request-target, the proxy MUST ignore the received Host header field (if any) and instead replace it with the host information of the request-target. A proxy that forwards such a request MUST generate a new Host field-value based on the received request-target rather than forward the received Host field-value.

https://tools.ietf.org/html/rfc7230#section-5.4

---

It looks like Host is replaced with :authority in http2. Hope that helps.

https://tools.ietf.org/html/rfc7540#section-8.1.2.3

[owenconti]

What returned an empty string, exactly?

Sorry, {http.request.orig_uri} was empty, but only when used via the caddyfile:

        reverse_proxy /follow redirects.ohseemedia.com:443 {
                header_up X-Custom-Header-Uri {uri}
                header_up X-Custom-Header-OrigUri {orig_uri}
                header_up Host redirects.ohseemedia.com
                header_up -X-Forwarded-For
                transport http {
                        tls
                }
        }

X-Custom-Header-OrigUri is sent as empty

[mholt]

Ah, thanks for clarifying. That's because {orig_uri} is not yet a supported shorthand placeholder (I haven't settled on the name for sure yet), so you have to use {http.request.orig_uri} still. They are not the same thing unless they are hard-coded to be the same thing.

[owenconti]

Oh derp, that's my bad. I had just assumed the shorthand was syntactical sugar for the long form.

{http.request.orig_uri} works perfectly in a caddyfile!

[mholt]

In the latest v2 beta (15), you can do:

reverse_proxy https://example.com

to reverse proxy to an HTTPS backend. However, path elements are not supported since that implies a rewrite, which cannot (yet?) be done in the proxy between choosing a backend and proxying the request.