Caddy: H2 and H2C bi-direction streaming reverse proxy need non-nil req.body and flush headers before copyResponse
I found that haproxy is capable of proxying h2 clients to h2c upstream servers, but caddy's not working even with merged h2c branch #3289. For instance, client and server I used were v2ray(h2 outbound and h2c inbound).
After some digging and testing, I found the cause and have made caddy work in this scenario.
First, HTTP2 allows bi-direction streaming, and not using Transfer-Encoding and Content-Length headers for this. Thus we probably need to keep req.Body valid before invoking RoundTrip to upstream.
https://github.com/caddyserver/caddy/blob/7a99835dab64f7864186185761bbf5194216f8b6/modules/caddyhttp/reverseproxy/reverseproxy.go#L424
req.Body = originBody
start := time.Now()
res, err := h.Transport.RoundTrip(req)
Second, in order to signal h2 clients begin sending streaming body, we need to flush response headers to client before copyResponse from res.Body to http.ResponseWriter. And use -1 as flushInterval for http2 connections.
rw.WriteHeader(res.StatusCode)
if h2 {
if wf, ok := rw.(http.Flusher); ok {
wf.Flush()
}
err = h.copyResponse(rw, res.Body, -1)
} else {
err = h.copyResponse(rw, res.Body, h.flushInterval(req, res))
}
The config file I used:
{
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"my.org"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/usr/share/caddy"
}
]
},
{
"handle": [
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.request.host}"
],
"X-Forwarded-For": [
"{http.request.remote}"
],
"X-Forwarded-Port": [
"{server_port}"
],
"X-Forwarded-Proto": [
"http"
],
"X-Real-Ip": [
"{http.request.remote}"
]
}
}
},
"transport": {
"protocol": "http",
"compression": false,
"versions": [
"h2c",
"2"
]
},
"upstreams": [
{
"dial": "localhost:54321"
}
]
}
],
"match": [
{
"path": [
"/toV2ray"
]
}
]
},
{
"handle": [
{
"handler": "file_server",
"hide": [
"/etc/caddy/Caddyfile"
]
}
]
}
]
}
],
"terminal": true
}
]
}
}
},
"tls": {
"automation": {
"policies": [
{
"issuer": {
"email": "[email protected]",
"module": "acme"
}
}
]
}
}
}
}
masknu
All 10 comments
Interesting -- can you explain a little more? Is this a bug report or feature request? And did you make it work for you by removing the req.Body = nil line? What was the whole, resulting diff in the code? How can we reproduce the problem and test the patch? Thanks.
mholt
on 7 Jul 2020
Yes, this is a bug report,. Caddy is unable to do h2 streaming for apps which need to get a header response before streaming content. And HAProxy is able to do that out of box.
I can send a PR to show the diff I made.
I didn't remove the req.body=nil line. In stead, I save the body to the request context, and reassign it to the req before the RoundTrip call if the req's MajorProto is 2.
masknu
on 7 Jul 2020
@masknu Okay, gotcha. Yes, if you could submit a patch then I will be happy to review it! :+1:
mholt
on 7 Jul 2020
Sorry about the req.body part, after second confirm, it's not an issue.
The only issue is we need to flush the res.header under certain condition(proto==2 and contentlength == -1).
masknu
on 8 Jul 2020
To reproduce the problem:
- Prepare local machine C, and an internet accessible server S.
- Copy server-config.json to S, replace "YOUR.DOMAIN" by your domain.
- Copy client-config.json to C, replace "YOUR.DOMAIN" by your domain.
- Copy caddy.json to S, replace "YOUR.DOMAIN" by your domain, replace "[email protected]" by your email address.
- Download v2ray (version 4.25.1) from v2ray and put it on C and S.
Start v2ray at S with server-config.json by command:
./v2ray -config server-config.jsonStart caddy with caddy.json at step 4.
Start v2ray at C with server-config.json by command:
./v2ray -config client-config.jsonNow, if Caddy behaves like HAProxy does with same configuration, we should be able to wget any internet URL at C by command:
http_proxy=http://localhost:8080 https_proxy=http://localhost:8080 wget https://github.com/caddyserver/caddy/releases/download/v2.1.1/caddy_2.1.1_windows_amd64.zip
Below are the config files mentioned:
server-config.json:
{
"log": {
"loglevel": "debug"
},
"inbounds": [
{
"port": 54321,
"listen": "127.0.0.1",
"protocol": "vmess",
"settings": {
"clients": [
{
"id": "4c7c55bc-5d50-41cc-8f32-a5d205770524",
"level": 1,
"alterId": 64
}
]
},
"streamSettings": {
"network": "h2",
"security": "none",
"httpSettings": {
"host": [
"YOUR.DOMAIN"
],
"path": "/tov2ray"
}
}
}
],
"outbound": {
"protocol": "freedom",
"settings": {}
},
"outboundDetour": [
{
"protocol": "blackhole",
"settings": {},
"tag": "blocked"
}
],
"routing": {
"strategy": "rules",
"settings": {
"rules": [
{
"type": "field",
"ip": ["geoip:private"],
"outboundTag": "blocked"
}
]
}
}
}
client-config.json:
{
"log": {
"loglevel": "debug"
},
"inbounds": [
{
"listen": "0.0.0.0",
"protocol": "http",
"port": 8080,
"settings": {}
}
],
"outbounds": [
{
"tag": "proxy",
"protocol": "vmess",
"settings": {
"vnext": [
{
"address": "YOUR.DOMAIN",
"port": 443,
"users": [
{
"id": "4c7c55bc-5d50-41cc-8f32-a5d205770524",
"alterId": 64
}
]
}
]
},
"streamSettings": {
"network": "h2",
"security": "tls",
"httpSettings": {
"host": [
"YOUR.DOMAIN"
],
"path": "/tov2ray"
},
"sockopt": {
"mark": 255
}
}
},
{
"tag": "direct",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": 255
}
}
},
{
"tag": "block",
"protocol": "blackhole",
"settings": {
"response": {
"type": "http"
}
}
},
{
"tag": "dns-out",
"protocol": "dns",
"streamSettings": {
"sockopt": {
"mark": 255
}
}
}
],
"dns": {
"hosts": {},
"servers": [
{
"address": "223.5.5.5",
"port": 53,
"domains": [
"geosite:cn",
"ntp.org",
"YOUR.DOMAIN"
]
},
"8.8.8.8",
"1.1.1.1",
"114.114.114.114"
]
},
"routing": {
"domainStrategy": "IPOnDemand",
"rules": [
{
"type": "field",
"ip": [
"223.5.5.5",
"114.114.114.114"
],
"outboundTag": "direct"
},
{
"type": "field",
"ip": [
"8.8.8.8",
"1.1.1.1"
],
"outboundTag": "proxy"
},
{
"type": "chinasites",
"outboundTag": "direct"
},
{
"type": "field",
"ip": [
"0.0.0.0/8",
"10.0.0.0/8",
"100.64.0.0/10",
"127.0.0.0/8",
"169.254.0.0/16",
"172.16.0.0/12",
"192.0.0.0/24",
"192.0.2.0/24",
"192.168.0.0/16",
"198.18.0.0/15",
"198.51.100.0/24",
"203.0.113.0/24",
"::1/128",
"fc00::/7",
"fe80::/10"
],
"outboundTag": "direct"
},
{
"type": "field",
"domain": [
"geosite:category-ads-all"
],
"outboundTag": "block"
},
{
"type": "field",
"protocol": [
"bittorrent"
],
"outboundTag": "direct"
},
{
"type": "chinaip",
"outboundTag": "direct"
},
{
"type": "field",
"ip": [
"geoip:private",
"geoip:cn"
],
"outboundTag": "direct"
}
]
}
}
caddy.json:
{
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"YOUR.DOMAIN"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/usr/share/caddy"
}
]
},
{
"handle": [
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.request.host}"
],
"X-Forwarded-For": [
"{http.request.remote}"
],
"X-Forwarded-Port": [
"{server_port}"
],
"X-Forwarded-Proto": [
"http"
],
"X-Real-Ip": [
"{http.request.remote}"
]
}
}
},
"transport": {
"protocol": "http",
"compression": false,
"versions": [
"h2c",
"2"
]
},
"upstreams": [
{
"dial": "localhost:54321"
}
]
}
],
"match": [
{
"path": [
"/tov2ray"
]
}
]
},
{
"handle": [
{
"handler": "file_server",
"hide": [
"/etc/caddy/Caddyfile"
]
}
]
}
]
}
],
"terminal": true
}
]
}
}
},
"tls": {
"automation": {
"policies": [
{
"issuer": {
"email": "[email protected]",
"module": "acme"
}
}
]
}
}
}
}
@masknu Hello Kevin, does caddy v2 work with v2ray (h2c)? if yes, could you share your configurations? Thank you very much.
The following Caddyfile does NOT works for me.
reverse_proxy /{H2PATH} {
to localhost:{PORT}
header_up Host {http.request.host}
header_up X-Forwarded-For {http.request.remote}
header_up X-Forwarded-Port {server_port}
header_up X-Forwarded-Proto "http"
header_up X-Real-Ip {http.request.remote}
transport http {
# compression false
versions h2c 2
}
}
choicky
on 28 Jul 2020
Yes, it works well.
Everything looks fine with your config. If I were you, I would prefix localhost:{PORT} with http:// and move it to the front:
reverse_proxy /{H2PATH} http://localhost:{PORT}{
transport http {
versions h2c 2
}
}
And the server-config.json mentioned above is pretty much what my v2ray server config file looks like. The important part is "streamSettings"."security":
"streamSettings": {
"network": "h2",
"security": "none",
"httpSettings": {
"host": [
"YOUR.DOMAIN"
],
"path": "/tov2ray"
}
}
@masknu Thanks for your reply.
However, my config still does not work.
Could you please send a copy of your config files to my email? It's [email protected] .
choicky
on 28 Jul 2020
@choicky Here is a set of complete config files:
https://gist.github.com/hang333/047ecf4c8d7d2868f1ce142d713a3520
from v2ray/discussion#759 .
masknu
on 30 Jul 2020
@masknu Thank you very much. It works for me.
By the way, the ws configure could be simplified as reverse_proxy /{wsPath} localhost:{Port} It seems that the @v2ray_websocket is not required.
choicky
on 30 Jul 2020
Related issues
xfzka
路
3Comments
PhilmacFLy
路
3Comments
whs
路
3Comments
mschneider82
路
3Comments
treviser
路
3Comments
Most helpful comment
To reproduce the problem:
Start v2ray at S with server-config.json by command:
Start caddy with caddy.json at step 4.
Start v2ray at C with server-config.json by command:
Now, if Caddy behaves like HAProxy does with same configuration, we should be able to wget any internet URL at C by command:
Below are the config files mentioned:
server-config.json:
client-config.json:
caddy.json: