Graylog2-server: API Connection Problems in Graylog 2.0.0

[lasdem]

Problem description

When opening the graylog web interface, we get a login popup (probably for the api) and the login site for graylog.
After Login some features do not work:

1. Incoming and Outgoing Events per Seconds Metrics
2. System Information in the Nodes view
3. Details page of a node
4. and more

ad 1.

Request URL:https://graylog.example.org/api/cluster/metrics/multiple
Request Method:POST
Status Code:200 OK
Payload
{metrics: ["org.graylog2.throughput.input.1-sec-rate", "org.graylog2.throughput.output.1-sec-rate"]}
Response
{"8ae4b7c4-a094-4eed-aaac-11690abc2874":null}

Unhandled rejection TypeError: Cannot read property 'metrics' of null
at eval (webpack:///./src/stores/metrics/MetricsStore.js?:108:25)
at Array.forEach (native)
at eval (webpack:///./src/stores/metrics/MetricsStore.js?:105:29)
at tryCatcher (webpack:///./~/bluebird/js/browser/bluebird.js?:4589:23)
at Promise._settlePromiseFromHandler (webpack:///./~/bluebird/js/browser/bluebird.js?:2695:31)
at Promise._settlePromiseAt (webpack:///./~/bluebird/js/browser/bluebird.js?:2769:18)
at Promise._settlePromises (webpack:///./~/bluebird/js/browser/bluebird.js?:2885:14)
at Async._drainQueue (webpack:///./~/bluebird/js/browser/bluebird.js?:175:16)
at Async._drainQueues (webpack:///./~/bluebird/js/browser/bluebird.js?:185:10)
at Async.drainQueues (webpack:///./~/bluebird/js/browser/bluebird.js?:67:14)

Other requests that fail:

Request URL:https://graylog.example.org/api/system/indexer/cluster/health
Request Method:GET
Status Code:500 Internal Server Error

Request URL:https://graylog.example.org/api/system/indexer/cluster/name
Request Method:GET
Status Code:500 Internal Server Error

Request URL:https://graylog.example.org/api/plugins/org.graylog.plugins.usagestatistics/config
Request Method:GET
Status Code:404 Not Found

Request URL:https://graylog.example.org/api/plugins/org.graylog.plugins.usagestatistics/opt-out
Request Method:GET
Status Code:404 Not Found

Steps to reproduce the problem

1. Upgraded Graylog from 1.3 to 2.0
2. added /api to the Reverse Proxy to go to port 12900 on the graylog server

3. changed /etc/graylog/server/server.conf

   rest_listen_uri = http://0.0.0.0:12900/api/
   rest_transport_uri = https://graylog.example.org:443/api/
   rest_enable_cors = true
   web_listen_uri = http://0.0.0.0:9000/

4. direct the browser to https://graylog.example.org
   

   Environment

• Graylog Version: 2.0.0 Release 6
• Elasticsearch Version: 2.3.2
• MongoDB Version: 3.2.6 Release 1.el6
• Operating System: RHEL 6.7
• Browser version:
  
  
  
  • Chrome 50.0.2661.94 m
  
  
  • Firefox 38.7.0 ESR
  
  
  • IE 11.0.9600.18230
  
  

[joschi]

@lasdem Your configuration settings look a little bit strange.

Can the Graylog nodes reach each other using the rest_transport_uri?
Is the reverse proxy also covering the web interface?
Could you share the configuration of your reverse proxy?
Are there any error messages in the logs of your Graylog server nodes?

[lasdem]

Yes the Proxy is handling TLS for us and is used for both the web and the api.
At the moment we only have the master node running, but the other nodes should be able to reach the rest_transport_uri.
Some calls do work, so I do not think the reverse proxy is the problem.
Also I used a local reverse proxy (nginx) and I am now using the corporate proxy (apache) and both had the same issue. I asked my colleague in the proxy team to send me the configuration and I will post it later today.

API Calls that work:
https://graylog.example.org/api/system/
https://graylog.example.org/api/system/notifications
https://graylog.example.org/api/system/cluster/nodes
https://graylog.example.org/api/streams

fyi I replaced the actual domain name with example.org

[joschi]

@lasdem Basically all HTTP requests which in turn trigger HTTP requests to rest_transport_uri from within the Graylog server fail, so I guess that https://graylog.example.org:443/api/ simply can't be accessed from within the system running Graylog.

[lasdem]

OK so how would I need to change the config to make calls from the web interface go over the proxy but internal requests go direct?
OR should I just make sure that all graylog server nodes can reach the proxy?
I am currently on the move, therefore I will be able to test stuff in a few hours.

[lasdem]

Here the relevant parts of our reverse proxy config:

Timeout 300
KeepAlive On
MaxKeepAliveRequests 10000
KeepAliveTimeout 15
LimitRequestline 51200
LimitRequestFields 10000
LimitRequestFieldsize 102400
LimitRequestBody 10240000

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

BalancerMember http://192.168.0.81:9000 smax=50 max=50 timeout=300 ttl=300 keepalive=On retry=60 loadfactor=1 route=81
BalancerMember http://192.168.0.82:9000 smax=50 max=50 timeout=300 ttl=300 keepalive=On retry=60 loadfactor=1 route=82
BalancerMember http://192.168.0.83:9000 smax=50 max=50 timeout=300 ttl=300 keepalive=On retry=60 loadfactor=1 route=83
ProxySet stickysession=ROUTEID nofailover=Off

BalancerMember http://192.168.0.81:12900 smax=50 max=50 timeout=300 ttl=300 keepalive=On retry=60 loadfactor=1 route=81
BalancerMember http://192.168.0.82:12900 smax=50 max=50 timeout=300 ttl=300 keepalive=On retry=60 loadfactor=1 route=82
BalancerMember http://192.168.0.83:12900 smax=50 max=50 timeout=300 ttl=300 keepalive=On retry=60 loadfactor=1 route=83
ProxySet stickysession=ROUTEID nofailover=Off

ProxyPass /api balancer://graylog-api/api
ProxyPassReverse /api balancer://graylog-api/api
ProxyPass / balancer://graylog/
ProxyPassReverse / balancer://graylog/
ProxyPreserveHost On

[jpancoast-kenzan]

I think I had a very similar issue. I was setting rest_transport_uri like you were but when I looked at the graylog logs on the box it was filling up with messages saying "couldn't connect to graylog.example.org..." (I'm in AWS, and the instance inside the VPC can't/couldn't talk to the External ELB I had setup). So, basically what @joschi said.

I chose @lasdem 's first fix option (although if you could I think the second option might be better) and here's how I did it:

in /etc/graylog/graylog-settings.json set "rest_transport_uri": false
run graylog-ctl reconfigure(-as-whatever)
Add this line to /opt/graylog/conf/graylog.conf:

web_endpoint_uri = https://graylog.example.org:443/api/

then run graylog-ctl restart

If someone can tell me how to set 'web_endpoint_uri' with graylog-ctl or in graylog-settings.json so a graylog-ctl reconfigure doesn't overwrite it I'd be interested to hear that...

Edit:
I found out about 'web_endpoint_uri' on this page: http://docs.graylog.org/en/2.0/pages/configuring_webif.html

[lasdem]

I have checked the connection and an https connection is possible from the node to the reverse proxy. I also looked into the log and found that the following seems to be the problem:

2016-05-03T21:22:55.239+02:00 WARN [ProxiedResource] Unable to call https://graylog.example.org/system/metrics/multiple on node <8ae4b7c4-a094-4eed-aaac-11690abc2874>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)

Looks like something does not like our companies PKI.

I will also look into @jpancoast-kenzan suggestion, but I do not seem to have a /etc/graylog/graylog-settings.json or /opt/graylog/conf/graylog.conf but a /etc/graylog/server/server.conf

[jpancoast-kenzan]

@lasdem I'm using the fresh 2.0 AMI's that graylog provides, so I'm guessing there are differences on your system that was upgraded.

[lasdem]

@jpancoast-kenzan I'm using the yum repository.
But I could set the settings in my system but now I get these errors:

2016-05-03T21:37:26.878+02:00 WARN [ProxiedResource] Unable to call http://192.168.0.81:12900/system/metrics/multiple on node <8ae4b7c4-a094-4eed-aaac-11690abc2874>, result: Not Found

The web interface now uses reverse proxy and the backend a direct connection to the ip of the node (which is great!). But the direct connection does not know it has to use /api

[lasdem]

I now also tried setting the transport uri to this setting:

rest_transport_uri = http://0.0.0.0:12900/api/

But than it tries to connect to

2016-05-03T22:46:10.938+02:00 WARN [ProxiedResource] Unable to call http://0.0.0.0:12900/system/inputstates on node <8ae4b7c4-a094-4eed-aaac-11690abc2874>, result: Not Found

[lasdem]

So we finally solved our Problem today.
Current Settings:

rest_enable_cors = true
rest_listen_uri = http://0.0.0.0:12900/
web_listen_uri = http://0.0.0.0:9000/
web_endpoint_uri = https://graylog.example.org:443/api/

And rest_transport_uri is commented out.
And Reverse Proxy:

https://graylog.example.org:443/api/ --> graylog-nodes:12900
https://graylog.example.org:443 --> graylog-nodes:9000

The Reverse Proxy has to strip away the /api/ from the requests.

The following things I would consider open Issues, but if you want you can close this issue.

• web_endpoint_uri is not documented, but a probably a much needed setting for companies who use a reverse proxy/loadbalance setup.
• rest_transport_uri = http://0.0.0.0:12900/api/ does not work as expected, because the /api/ is ignored

[joschi]

web_endpoint_uri is not documented, but a probably a much needed setting for companies who use a reverse proxy/loadbalance setup.

http://docs.graylog.org/en/2.0/pages/configuring_webif.html#configuration-options
Is anything crucial missing in that documentation?

[lasdem]

Thank you the documentation is not missing anything.
Small thing I would add is in the Apache example at the bottom of the page in the section "URI Configs in Graylog server conf:"

web_endpoint_uri = https://graylog.example.org:443/api/

It is explained above, but someone may miss it if only the example is looked at in depth.

[joschi]

@lasdem Thanks for your feedback! I'll close this issue now.