Caddy: http: invalid Read on closed Body with v1.0.1

[orditeck]

Hi,

I'm trying to proxy Proxmox.
sub.domain.com should proxy https://192.168.1.10:8006.
When I try to login through Proxmox's UI, I get a 502 error. It searched on Google and it looks like it may be a bug in Go... so I don't really know what to do here. Any help would be greatly appreciated!

When entering a bad user/pass combo, I get a 400, which is to be expected.
When entering valid user/pass combo, I get a 502. I should get a 200.

1. Which version of Caddy are you using (caddy -version)?

Caddy v1.0.1 (h1:oor6ep+8NoJOabpFXhvjqjfeldtw1XSzfISVrbfqTKo=)

2. What are you trying to do?

Proxy Proxmox proxy / https://192.168.1.10:8006

3. What is your Caddyfile?

sub.domain.com {
    import ../../config/tls.conf

    tls {
        dns cloudflare
        alpn http/1.1
    }

    import ../../config/auth.conf
    gzip
    header / {
        X-Frame-Options "SAMEORIGIN"
    }
    proxy / https://192.168.1.10:8006 {
        transparent
        insecure_skip_verify
        websocket
        timeout 3600s
        try_duration 60s
        max_fails 5
        try_interval 60s
        header_downstream -X-Frame-Options
    }    
}

4. How did you run Caddy (give the full command and describe the execution environment)?

curl https://getcaddy.com | bash -s personal hook.service,http.awses,http.awslambda,http.cache,http.cors,http.datadog,http.expires,http.filter,http.forwardproxy,http.geoip,http.git,http.jwt,http.login,http.mailout,http.minify,http.nobots,http.permission,http.ratelimit,http.realip,http.s3browser,http.supervisor,http.torproxy,http.webdav,supervisor,tls.dns.cloudflare

5. Please paste any relevant HTTP request(s) here.

curl 'https://sub.domain.com/api2/extjs/access/ticket' -H 'origin: https://sub.domain.com' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9,fr;q=0.8' -H 'csrfpreventiontoken: null' -H 'x-requested-with: XMLHttpRequest' -H 'pragma: no-cache' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' -H 'accept: */*' -H 'cache-control: no-cache' -H 'authority: sub.domaine.com' -H 'referer: https://sub.domain.com/' --data 'username=root&password=removed&realm=pam' --compressed

6. What did you expect to see?

No error.

7. What did you see instead (give full error messages and/or log)?

Jul 22 14:24:04 caddy caddy[3929]: 22/Jul/2019:14:24:04 +0000 [ERROR 502 /api2/extjs/access/ticket] http: invalid Read on closed Body
Jul 22 14:26:31 caddy caddy[3929]: 22/Jul/2019:14:26:31 +0000 [ERROR 502 /api2/extjs/access/ticket] http: invalid Read on closed Body

[mholt]

Huh, weird. Do the requests hang at all? Is the upstream actually up and healthy and connecting okay? I suspect Caddy is not buffering the request body in this case because you only have one upstream defined.

If you remove the try_duration does the error go away?

[orditeck]

If you remove the try_duration does the error go away?

The error is now:

Jul 22 19:40:47 caddy caddy[4103]: 22/Jul/2019:19:40:47 +0000 [ERROR 502 /api2/extjs/access/ticket] EOF

[mholt]

I suspect something is wrong with the upstream then, it is just closing the connection it looks like.

[orditeck]

Seems like it. Just checked on Proxmox's side:

Jul 22 14:44:33 elite pveproxy[16989]: to many http header lines
Jul 22 14:44:38 elite pveproxy[16989]: problem with client 192.168.1.53; Connection timed out

[orditeck]

Thanks a lot for your help!

[orditeck]

For anyone having the same issue, I ended up fixing it ("temporarily") by setting $limit_max_headers to 60 instead of 30 in /usr/share/perl5/PVE/APIServer/AnyEvent.pm on the Proxmox server...

Then pvedaemon restart && pveproxy restart

Final proxy looks like this (with the console working):

    proxy / https://192.168.1.10:8006 {
        transparent
        insecure_skip_verify
        websocket
        timeout 3600s
        max_fails 5
        try_interval 60s
    }  

[mholt]

Nice, thanks for following-up with a fix/workaround.