Can I use Caddy to load balance / rotate across a bunch of proxies that I use?
Essentially, I would like the following curl request:
curl -x localhost:2015 http://www.google.com
to use my set of proxies, rotated behind localhost:2015, which is my Caddy instance.
manojlds
All 16 comments
The proxy directive does it.
Your Caddyfile:
localhost:2015
proxy / yourproxyone yourproxytwo yourproxythree
arthurwhite
on 28 Mar 2016
@arthurwhite that's something I tried before asking this. It doesn't work, and in my case I get a 400 Bad Request from proxy servers I am using.
My Caddyfile is exactly what you mentioned, and I did a
curl -Lvx localhost:2015 http://www.google.com
My working haproxy configuration:
frontend rotatingproxies
bind *:3000
default_backend rotateproxy
backend rotateproxy
server proxy1 yourproxyone:80
balance roundrobin
And doing a similar curl works as expected:
curl -Lvx localhost:3000 http://www.google.com
In Caddy, only _websocket_ and _browse_ middlewares can potentially send a _400 Bad Request_ code.
So I suppose it comes from your proxies.
Did you enable error logs in your Caddyfile to see that clearly?
localhost:2015
proxy / yourproxyone yourproxytwo yourproxythree
errors proxieserrors.log
@manojlds I don't really understand what the -x flag does under the hood, but yeah, enable error logging as Arthur suggests and let's see if we can figure this out. (Caddy isn't meant to be a "forward" proxy but I wonder if it's not that different.)
mholt
on 29 Mar 2016
Caddy isn't meant to be a "forward" proxy […].
That's the answer!
As described in RFC 2616, Section 5.1.2, a proxy needs to receive an absolute URI in the HTTP request line. That's what the -x flag does.
But the Caddy's _proxy_ middleware does nothing with the original *http.Request.RequestURI field.
It makes a new(http.Request) instead of modifying the original: middleware/proxy/reverseproxy.go#L163.
So while curl -x localhost:2015 http://www.google.com sends
> GET http://www.google.com/ HTTP/1.1
@manojlds's proxies receive
> GET / HTTP/1.1
It's clearly a "Bad Request" for a proxy.
arthurwhite
on 29 Mar 2016
@mholt Based on @arthurwhite 's response, Caddy doesn't seem to support forward proxying scenarios. I guess this can be a useful feature to have in Caddy. We can keep this issue open? I am just beginning to learn Go, but myself or someone else can probably submit a PR for this.
manojlds
on 30 Mar 2016
Mmm, it was a premature conclusion…
The next line copies the RequestURI. :sweat_smile:
The problem is somewhere else.
Can you post your error log please?
arthurwhite
on 30 Mar 2016
@arthurwhite There are no error logs. The 400 does come from my proxy severs.
Access logs:
::1 - [30/Mar/2016:12:01:56 +0530] "GET / HTTP/1.1" 400 12
Using some public proxies:
Direct:
curl -Lvx 52.36.247.10:8083 http://www.google.com
Via Caddy. Caddyfile:
:2015
log stdout
errors stdout
proxy / 52.36.247.10:8083
Request:
curl -Lvx localhost:2015 http://www.google.com
Curl verbose output:
* Rebuilt URL to: http://www.google.com/
* Trying ::1...
* Connected to localhost (::1) port 2015 (#0)
> GET http://www.google.com/ HTTP/1.1
> Host: www.google.com
> User-Agent: curl/7.43.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 400 Bad Request
< Content-Language: en
< Content-Length: 3161
< Content-Type: text/html
< Date: Wed, 30 Mar 2016 11:59:22 GMT
< Mime-Version: 1.0
< Server: Caddy
< Server: squid/3.3.8
< Vary: Accept-Language
< Via: 1.1 ip-172-31-22-69 (squid/3.3.8)
< X-Cache: MISS from ip-172-31-22-69
< X-Cache-Lookup: NONE from ip-172-31-22-69:8083
< X-Squid-Error: ERR_INVALID_URL 0
Does this help @arthurwhite ?
manojlds
on 30 Mar 2016
And it is problem between sending / and http://www.google.com in the request. The proxy receives / instead of a absolute url and says invalid url.
manojlds
on 30 Mar 2016
Would it be as simple as changing the Request with our custom Director?
mholt
on 30 Mar 2016
No, we see that the request structure remains correct (the RequestURI value is kept) all along.
Instead, it's the http.DefaultTransport (used in this case) who seems to ignore the RequestURI.
arthurwhite
on 30 Mar 2016
I believe this will work if your Caddyfile is setup like so:
proxy / proxyone proxytwo proxythree {
policy round_robin
proxy_header Host {host}
}
My guess is, as the screenshot in @manojlds shows, Caddy isn't forwarding the Host header. Because of this, Host header may get sent as "proxythree", which will confuse the final destination server if they are using virtual hosts (which almost every host does).
nemosupremo
on 23 Apr 2016
Actually I may be wrong, reading this thread a little closer, it looks like you need golang's http to not strip out the hostname when making the GET request.
To make this work you would need to edit reverseproxy.go#L229 and change .Write() to .WriteProxy() ((*Request) WriteProxy).
This seems like a pretty special/limited case, that AFAICT serves one use case (load balancing amongst proxies) so I'm not sure if its worth baking into Caddy - however the "solution" is pretty easy.
nemosupremo
on 23 Apr 2016
Since this is not likely to work its way into Caddy's HTTP server in the foreseeable future, I'm going to close this. Good discussion!
mholt
on 13 Aug 2016
Hey folks,
Accidentally found this issue, and would like to let you know that http.forwardproxy plugin has been supported in Caddy for almost a year.
Source code and documentation
sergeyfrolov
on 27 Jun 2018
Related issues
jgsqware
·
3Comments
xfzka
·
3Comments
mikolysz
·
3Comments
ericmdantas
·
3Comments
la0wei
·
3Comments
Most helpful comment
Actually I may be wrong, reading this thread a little closer, it looks like you need golang's http to not strip out the hostname when making the GET request.
To make this work you would need to edit reverseproxy.go#L229 and change
.Write()to.WriteProxy()((*Request) WriteProxy).This seems like a pretty special/limited case, that AFAICT serves one use case (load balancing amongst proxies) so I'm not sure if its worth baking into Caddy - however the "solution" is pretty easy.