Victoriametrics: vmauth - read error during body copy - Grafana dashboards broken

[chrisfw]

Describe the bug
After updating vmauth to latest version due to #533 we are now experiencing a problem wherein the dashboards are broken for another person logged in as admin, but not for me which is very odd. Note that we are port forwarding from our local windows machines over a putty ssh connection.
Local port 3059 forwards to 127.0.0.1:3000 (remote grafana)
Grafana logs show this error returned by the server:

t=2020-06-05T08:58:15-0500 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/2/api/v1/query_range remote_addr=127.0.0.1 referer="http://127.0.0.1:3059/d/SED6KxFZk/redacted-production-environment-status?inspect=4&tab=error&orgId=1&refresh=10s" error="httputil: ReverseProxy read error during body copy: unexpected EOF"

vmauth logs on the server show the corresponding error:

Jun  5 09:58:15 oci-ssi-mon vmauth-prod: 2020-06-05T13:58:15.570Z#011error#011net/http/httputil/reverseproxy.go:443#011httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:54354->127.0.0.1:8430: use of closed network connection
Jun  5 09:58:55 oci-ssi-mon vmauth-prod: 2020-06-05T13:58:55.351Z#011error#011net/http/httputil/reverseproxy.go:443#011http: proxy error: context canceled

To Reproduce
Use the Prometheus datasource with TLS certificates pointing to a vmauth backend

Expected behavior
vmauth does not generate a read error (for some users) and prevent Grafana dashboards from working.

Screenshots

Version
The line returned when passing --version command line flag to binary. For example:

victoria-metrics-20200601-100434-tags-v1.36.3-0-g32652485e
vmauth-20200601-100445-tags-v1.36.3-0-g32652485e

[chrisfw]

@valyala Could this also be related to gzipping in vmauth? I found this:

Today I found https://github.com/containous/traefik/issues/2903 23 which became stale but is exactly my problem.

The OP was told 2:

It seems that this is a backend issue caused when requesting gzip encoding.

Excerpted from here
which refers to traefik issue

[chrisfw]

@valyala I was able to confirm this is related to gzipping in vmauth. I disabled httpCompression in vmauth via the -http.disableResponseCompression flag and it corrected the problem for the user whose dashboards were broken.

[valyala]

@chrisfw , could you provide request headers from both correct and broken dashboards when Grafana sends requests to vmauth? These headers can be obtained from Network panel in Chrome dev tools - see https://developers.google.com/web/tools/chrome-devtools/network for details.

Additionally, it would be great to know Operating System and web browser version where dashboards are broken.

[valyala]

It looks like I nailed down the issue and fixed it in the commit 69a647b0d2b875a14dba48b2183c15532c4d2d11 . @chrisfw , could you try building vmauth and other VictoriaMetrics components from the lastest commit in master or cluster branch and check whether the issue is fixed?

See build instructions:

• how to build vmauth
• how to build single-node VictoriaMetrics
• how to build cluster components of VictoriaMetrics

[chrisfw]

@valyala, it looks like you built a release with the fix in it? Can I just use that instead?
Sorry, I didn't have a chance to get you the headers as I had work commitments.

[valyala]

it looks like you built a release with the fix in it? Can I just use that instead?

Yes, you can try components from v1.37.0 release.

[chrisfw]

Thanks a lot! I will let you know the outcome.

[chrisfw]

@valyala We just completed testing and the issue appears to be resolved. Thank you very much!