Victoriametrics: [vmagent] Remote Write to Grafana Cloud Prometheus Compatible DB Fails

[chrisfw]

Describe the bug
Remote write to grafana cloud prometheus compatible tsdb for a simple prometheus.yml scraping a netdata prometheus endpoint and internal vmagent endpoint fails, yet the remote write works fine for prometheus

To Reproduce
May need to use grafana cloud prometheus tsdb

Expected behavior
Successful remote writes

Version
The line returned when passing --version command line flag to binary. For example:

$ ./vmagent-prod --version
vmagent-20200401-202747-tags-v1.34.6-0-gb38d048d

$ ./prometheus --version
prometheus, version 2.17.1 (branch: HEAD, revision: ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)
  build user:       root@806b02dfe114
  build date:       20200326-16:18:19
  go version:       go1.13.9

Used command-line flags
Full command line
./vmagent-prod -promscrape.config ./prometheus.yml -remoteWrite.url https://prometheus-us-central1.grafana.net/api/prom/push -remoteWrite.basicAuth.username [REMOVED] -remoteWrite.basicAuth.password [REMOVED] -remoteWrite.maxDiskUsagePerURL 805306368

Log output
`` 2020-04-03T19:48:40.125Z info VictoriaMetrics/lib/promscrape/scraper.go:123 starting 2 scrapers forstatic_config` targets
2020-04-03T19:48:41.267Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 2.000 seconds
2020-04-03T19:48:43.409Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 4.000 seconds
2020-04-03T19:48:47.570Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 8.000 seconds
2020-04-03T19:48:55.761Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 16.000 seconds
2020-04-03T19:49:11.980Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 32.000 seconds

2020-04-03T19:49:44.152Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 60.000 seconds
2020-04-03T19:50:44.349Z error VictoriaMetrics/app/vmagent/remotewrite/client.go:257 unexpected status code received after sending a block with size 637 bytes to "https://prometheus-us-central1.grafana.net/api/prom/push": 400; response body="snappy: corrupt inputn"; re-sending the block in 60.000 seconds```

[valyala]

The issue should be fixed in the commit 895cadfae7b850582aa23064c1ecbe4dc38ffaa4 .

@chrisfw , could you build vmagent from this commit according to these docs and see whether it can write data to Grafana Cloud?

[chrisfw]

Thank you for fixing this so quickly! I will try to test it tomorrow. I’ll let you know.

Regards,
Chris

On Apr 4, 2020, at 9:27 AM, Aliaksandr Valialkin [e-mail] wrote:


The issue should be fixed in the commit 895cadf .

@chrisfw , could you build vmagent from this commit according to these docs and see whether it can write data to Grafana Cloud?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[valyala]

The bugfix is available in v1.34.7

[chrisfw]

Awesome, thanks. I will try to test it today or tomorrow at the latest and
let you know.

Regards,
Chris

On Sun, Apr 5, 2020 at 8:16 AM Aliaksandr Valialkin <
[email protected]> wrote:

The bugfix is available in v1.34.7
https://github.com/VictoriaMetrics/VictoriaMetrics/releases

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399#issuecomment-609407004,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOZ7PNZYEKTM4UYMNVJIMTRLBZBLANCNFSM4L4NFCXQ
.

[chrisfw]

I tested v1.34.7 and it appears to be working fine as I can retrieve my metrics from the Grafana instance utilizing the hosted Prometheus compatible DB.

Thanks very much for your quick action to resolve this issue.