Grafana-zabbix: Parse error Invalid JSON. An error occurred on the server while parsing the JSON text.

[mplm17]

Describe the bug
Zabbix API cannot be accessed by v4.0.0-alpha4 plugin, while v3.12.2 and older versions could. When saving config of zabbix datasource, the following message appears:

Parse error Invalid JSON. An error occurred on the server while parsing the JSON text.

Expected behavior
Datasource configuration should be saved correctly. From grafana server, the following command returns a normal response:
curl -k -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"apiinfo.version","params":[],"id":1}' https://zabbix/api_jsonrpc.php
returns:
{"jsonrpc":"2.0","result":"5.0.1","id":1}

Screenshots
Here is my datasource conf, which works properly in stable versions:

The error message that I get when I click on 'Save & Test' button:

Software versions

| Grafana | Zabbix | Grafana-Zabbix Plugin |
| ------- | ------ | --------------------- |
| 7.1.1 | 5.0.1 | 4.0.0-alpha4 |

[alexanderzobnin]

Hi! Could you show grafana-server logs?

[mplm17]

Hi,

Yes sure, here are some logs:

t=2020-07-27T14:38:10+0200 lvl=eror msg="Request Completed" logger=context userId=13 orgId=1 uname=xyz method=POST path=/api/datasources/5/resources/zabbix-api status=500 remote_addr=x.x.x.x time_ms=214 size=132 referer="https://grafana.x.x/dashboard/new?editPanel=2&orgId=1"
t=2020-07-27T14:38:10+0200 lvl=eror msg="Zabbix authentication error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T14:38:10+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T14:38:10+0200 lvl=eror msg="Request Completed" logger=context userId=13 orgId=1 uname=xyz method=POST path=/api/datasources/5/resources/zabbix-api status=500 remote_addr=x.x.x.x time_ms=180 size=132 referer="https://grafana.x.x/dashboard/new?editPanel=2&orgId=1"
t=2020-07-27T14:44:35+0200 lvl=eror msg="Zabbix authentication error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T14:44:35+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T14:44:35+0200 lvl=eror msg="Request Completed" logger=context userId=13 orgId=1 uname=xyz method=POST path=/api/datasources/5/resources/zabbix-api status=500 remote_addr=x.x.x.x time_ms=136 size=132 referer="https://grafana.x.x/dashboard/new?editPanel=2&orgId=1"
t=2020-07-27T15:02:21+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T15:02:21+0200 lvl=eror msg="Request Completed" logger=context userId=13 orgId=1 uname=xyz method=POST path=/api/datasources/1/resources/zabbix-api status=500 remote_addr=x.x.x.x time_ms=132 size=132 referer="https://grafana.x.x/dashboard/new?editPanel=2&orgId=1"
t=2020-07-27T15:02:21+0200 lvl=eror msg="Zabbix authentication error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T15:02:21+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T15:02:21+0200 lvl=eror msg="Request Completed" logger=context userId=13 orgId=1 uname=xyz method=POST path=/api/datasources/1/resources/zabbix-api status=500 remote_addr=x.x.x.x time_ms=124 size=132 referer="https://grafana.x.x/dashboard/new?editPanel=2&orgId=1"
t=2020-07-27T15:02:25+0200 lvl=eror msg="Zabbix authentication error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T15:02:25+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-27T15:02:25+0200 lvl=eror msg="Request Completed" logger=context userId=13 orgId=1 uname=xyz method=POST path=/api/datasources/1/resources/zabbix-api status=500 remote_addr=x.x.x.x time_ms=116 size=132 referer="https://grafana.x.x/dashboard/new?editPanel=2&orgId=1"

[th0rtz]

Same behavior for me.

[alexanderzobnin]

Ok, thanks for the details, I'll try to reproduce this.

[alexanderzobnin]

Could you also enable debug logging in grafana and post logs? It's interesting, what's an api method returns error.

[th0rtz]

I don't have much info in the debug log

t=2020-07-28T14:32:38+0200 lvl=dbug msg="Received command to update data source" logger=datasources url=https://zabbix.lan/api_jsonrpc.php
t=2020-07-28T14:32:38+0200 lvl=dbug msg="Applying default URL parsing for this data source type" logger=datasource type=alexanderzobnin-zabbix-datasource url=https://zabbix.lan/api_jsonrpc.php
t=2020-07-28T14:32:39+0200 lvl=dbug msg="Data source settings changed" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource id=9.000 name=Zabbix org=1.000
t=2020-07-28T14:32:39+0200 lvl=dbug msg="Initializing data source" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource name=Zabbix org=1.000 id=9.000
t=2020-07-28T14:32:39+0200 lvl=dbug msg="Invoke Zabbix API request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource ds=Zabbix method=apiinfo.version
t=2020-07-28T14:32:39+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-28T14:32:39+0200 lvl=eror msg="Request Completed" logger=context userId=3 orgId=1 uname=usergrafana method=POST path=/api/datasources/9/resources/zabbix-api status=500 remote_addr="" time_ms=139 size=132 referer=https://grafana.lan/datasources/edit/9/
t=2020-07-28T14:32:39+0200 lvl=dbug msg="Invoke Zabbix API request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource ds=Zabbix method=apiinfo.version
t=2020-07-28T14:32:39+0200 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="Parse error Invalid JSON. An error occurred on the server while parsing the JSON text."
t=2020-07-28T14:32:39+0200 lvl=eror msg="Request Completed" logger=context userId=3 orgId=1 uname=usergrafana method=POST path=/api/datasources/9/resources/zabbix-api status=500 remote_addr="" time_ms=85 size=132 referer=https://grafana.lan/datasources/edit/9/

[th0rtz]

@alexanderzobnin what do you need for debuging more ?

[mplm17]

Hi,

Here is the post datas:

And the response:

[alexanderzobnin]

I've got managed to replicate this in some cases, but still not sure why that happens. Give me a bit more time to figure it out.

[alexanderzobnin]

Do you have any proxy behind your Zabbix? I see this if I query zabbix.org:

# curl -k -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":10}' http://www.zabbix.org/zabbix/api_jsonrpc.php
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://zabbix.org/zabbix/api_jsonrpc.php">here</a>.</p>
<hr>
<address>Apache/2.2.29 (Linux/SUSE) Server at www.zabbix.org Port 80</address>
</body></html>

If I add -L flag to follow redirects, then I get the same error:

# curl -k -L -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":10}' http://www.zabbix.org/zabbix/api_jsonrpc.php
{"jsonrpc":"2.0","error":{"code":-32600,"message":"Invalid Request.","data":"The received JSON is not a valid JSON-RPC Request."},"id":null}

But if I query https://zabbix.org/zabbix/api_jsonrpc.php directly, it works:

# curl -k -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":2}' https://zabbix.org/zabbix/api_jsonrpc.php
{"jsonrpc":"2.0","result":"4.2.0","id":2}

So it looks like proxy redirects query with changes that leads to wrong request.

[mplm17]

Hi,

No, I don't have any proxy behind zabbix.

I've tried the same command as yours from grafana server, with the same url setted in datasource configuration:

curl -k -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":10}' https://zabbix.mydomain/api_jsonrpc.php

And it works:

{"jsonrpc":"2.0","result":"5.0.1","id":10}

[alexanderzobnin]

That's strange. Could you check it against a different Zabbix server?

[mplm17]

With zabbix.org it seems working:

So the problem is related to my zabbix server? I will upgrade to 5.0.3, maybe this is a bug in 5.0.1...

[mplm17]

Same issue after upgrading in 5.0.3...

[th0rtz]

My zabbix server is on 4.2.6 and i have the same issue.

[alexanderzobnin]

What's an OS you're running Grafana on, BTW?

[alexanderzobnin]

Also, it might be related to SSL, so try to set Skip TLS Verify option.

[alexanderzobnin]

Found this in PyZabbix issues (related to SSL): https://github.com/lukecyca/pyzabbix/issues/44#issuecomment-57240255

[mplm17]

My Grafana is running on CentOS 7.8.

Same error when enabling Skip TLS Verify option. But btw, this option doesn't seem to work, cause if I change the url by setting the IP instead of domain name, it returns the following error even if Skip TLS Verify option is enabled...

Post https://192.168.x.y/api_jsonrpc.php: x509: cannot validate certificate for 192.168.x.y because it doesn't contain any IP SANs

Same error if I put an other domain name (which is an alias of the long DN):

Post https://zabbix/api_jsonrpc.php: x509: certificate is valid for zabbix.mydomain, not zabbix

So it seems like Skip TLS Verify option is ignored

[mplm17]

I'm currently going further into debugging. I found a way to log every requests on zabbix api. I can see that requests coming from grafana to zabbix api have no data:

ACCEPT-ENCODING: gzip
CONTENT-TYPE: application/json
TRANSFER-ENCODING: chunked
USER-AGENT: Go-http-client/1.1
HOST: zabbix.mydomain

while requests that come from another app have datas:

CONTENT-LENGTH: 277
CONTENT-TYPE: application/json
ACCEPT: */*
HOST: zabbix.mydomain

{"jsonrpc":"2.0","method":"trigger.get","params":{"monitored":true,"only_true":1,"min_severity":1,"sortfield":["lastchange"],"sortorder":"DESC","withLastEventUnacknowledged":true,"expandDescription":true,"selectHosts":["name"]},"id":1,"auth":"***"}

Headers are also not the same... Maybe this could help?

Edit: I upgraded the plugin in 4.0.0 (latest).

[alexanderzobnin]

That looks strange. Maybe it's related to the go-http-client. I'll check it out as well.

[Shmakovm]

Same issue after upgrading in 4.0.0, zabbix 5.0.3 and grafana 7.1.5 (Centos 7.8). On version 3.12.3, there was no error.

[alexanderzobnin]

@Shmakovm do you use https on Zabbix server?

[Shmakovm]

@Shmakovm do you use https on Zabbix server?

Yes, I use https with an internal CA certificate.
It's the same thing I used with the 3.12.3 plugin version and everything worked.

[alexanderzobnin]

@Shmakovm did you use Skip TLS Verify option with 3.12? I found it doesn't work with 4.0 because I have to implement it at the backend side.

[Prefix]

Tested with few Zabbix instances.
1st instance v4 Zabbix (no proxy, no ssl) - Works on Grafana
2nd instance v5 Zabbix troublesome (has Zabbix proxy but it's unrelated to frontend, with proper SSL wildcard certificate (bought one)) - Errors like for everyone
3rd one is Zabbix Appliance v5 straight from Zabbix website, no SSL there is no issues. - Works on Grafana

I tried non-working one with HTTP protocol as well (turned off https, seem that didn't fixed the issue).
SSL certificate is properly configurated (public key contains root public certificates as well)
Skip TLS Verify doesn't help to fix the issue.

Also I checked Zabbix logs it said 200 HTTP codes (httpd logs), not 500 like grafana website says (chrome developer javascript errors).

[Prefix]

https://i.imgur.com/Ky0Ji9W.jpg

[alexanderzobnin]

Yes, looks like it's not related to the SSL cert verification. I tried to test it against Zabbix 5.0.2 from official docker container and added self-signed cert. I get this error when testing data source:

Post "https://localhost:8189/api_jsonrpc.php": x509: certificate signed by unknown authority

If I add InsecureSkipVerify to the http client config, then it works.

So since it works with 3.12 (where grafana send the query), but doesn't work with 4.0 (where plugin backend sends request), there should be some difference between how grafana performs a query and how the plugin do the same.

But I still can't replicate this bug to figure it out, so any details about your environment would be super helpful.

[Shmakovm]

@Shmakovm did you use Skip TLS Verify option with 3.12? I found it doesn't work with 4.0 because I have to implement it at the backend side.

No, I didn't.

for example how curl works.

[mplm17]

How have you add the InsecureSkipVerify option? Maybe we could try the same?

Concerning my environment:

Grafana (7.1.5) is running on:
Centos 7.8
local CA is trusted

Zabbix (5.0.3) is running on:
Centos 7.8
httpd 2.4.6
HTTP certificate signed by a local CA

If you need any other info, just tell me ;)

[alexanderzobnin]

@mplm17 I did it in the source code :) I will fix HTTP options support anyway. If someone can collect traces of HTTP requests that Grafana backend sends to Zabbix, it would be nice.

[Prefix]

I tried v6.6.1 (21bf8b71bc) grafana it works fine. It says Zabbix API version: 5.0.3. Grafana Zabbix Plugin version 3.10.5
At least I'm quite sure it's something wrong from plugin code side.

[achupahin]

@alexanderzobnin If someone can collect traces of HTTP requests that Grafana backend sends to Zabbix, it would be nice.
Zabbix 5.0.3 Grafana 7.1.5 Plugin 4.0.0
Console output
nwrdomvs01.zip
nwrdomvs01-1598601861971.log

[alexanderzobnin]

@achupahin that's browser logs. But I need logs of requests that backend plugin sends. It's not so easy to collect, you need something like tcpdump or wireshark.

[alexanderzobnin]

@Prefix yes, it's definitely plugin issue.

[mplm17]

@achupahin that's browser logs. But I need logs of requests that backend plugin sends. It's not so easy to collect, you need something like tcpdump or wireshark.

I'm trying to do something like that, but it's not easy due to the SSL layer...

[achupahin]

@alexanderzobnin I can try do this - if somebody helps with tcpdump filters.

[Nehemoth]

@alexanderzobnin Maybe it has something to do with this
https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/390016-zabbix-api-always-returns-invalid-json?p=390080#post390080

[mplm17]

@alexanderzobnin Here it is:

Here are 2 others examples that are working good, so we can see the differences:

-with postman:

-from another zabbix api client:

With these outputs, I tried to reproduce the problem on postman by setting the same headers that are set in grafana requests. It seems that if "Content-Length" header is set, it works properly, but if it's not set, we have the parse error...

Hope this will help

[alexanderzobnin]

@mplm17 yes, this super helpful, thanks a lot!

[alexanderzobnin]

Ok, I manage to replicate this bug. It's related to request body. It seems, in some cases it's not closed properly, so Content-Lenght header is empty and Zabbix server can't read request properly.
https://github.com/alexanderzobnin/grafana-zabbix/blob/5ec8fc885b378d448c3d678dbd1f1c2134c15e43/pkg/zabbixapi/zabbix_api.go#L112-L116

[alexanderzobnin]

Ok, tried to solve it. Could anyone test it out and confirm that bug fixed? Here's a built package for all platforms. Just unpack it into grafana plugins folder.

grafana-zabbix-8f70a0c.zip

[achupahin]

@alexanderzobnin Yes, it's working.

[EdersonDeves]

it worked for me, too.

[mobarrio]

Ok, tried to solve it. Could anyone test it out and confirm that bug fixed? Here's a built package for all platforms. Just unpack it into grafana plugins folder.

grafana-zabbix-8f70a0c.zip

Work Zabbix 5.0.2 Grafana 7.1.5

[mplm17]

Ok, tried to solve it. Could anyone test it out and confirm that bug fixed? Here's a built package for all platforms. Just unpack it into grafana plugins folder.

grafana-zabbix-8f70a0c.zip

Thank you very much! It's working for me too ;)

[alexanderzobnin]

Ok, so closing this. Fix will be available in upcoming release soon.