Graylog2-server: Nginx unable to find valid certification path

[avongluck-r1soft]

Problem description

After upgrading to Graylog 2.0, we're seeing the following error in the graylog server logs:

2016-05-04T13:42:34.897-05:00 WARN [ProxiedResource] Unable to call https://XXX.XXX.net:12900/system/metrics/multiple on node , caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)

Random sections of the graylog web app don't work (although data-processing is running)

We're handling TLS through an nginx proxy. Graylog is listening on localhost via unsecured http.

server {
  listen       443 ssl spdy;
  server_name  XXX.XXX.net;

  ssl_dhparam          /etc/ssl/certs/dhparams.pem;
  ssl_certificate      /etc/letsencrypt/live/XXX.XXX.net/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/XXX.XXX.net/privkey.pem;

  ssl_session_cache shared:SSL:1m;
  ssl_session_timeout  5m;

  # Some SSL cipher stuff omitted.

  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://127.0.0.1:9000;
  }
}

server {
  listen    XX.XX.XX.XX:12900 ssl spdy;
  server_name   XXX.XXX.net;

  ssl_dhparam          /etc/ssl/certs/dhparams.pem;
  ssl_certificate      /etc/letsencrypt/live/XXX.XXX.net/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/XXX.XXX.net/privkey.pem;

  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://127.0.0.1:12900;
  }
}

# cat /etc/graylog/server/server.conf | egrep "^web|^rest"
rest_listen_uri = http://127.0.0.1:12900
rest_transport_uri = https://XXX.XXX.net:12900
web_listen_uri = http://127.0.0.1:9000/
web_transport_uri = https://XXX.XXX.net/

[avongluck-r1soft]

I've done a little digging and this seems to be the core of the issue:

https://XXX.XXX.net/api/cluster/d2636a89-c572-4717-b7e8-f7cada1266c6/plugins

Result:
{"type":"ApiError","message":"Failed to connect to XXX.XXX.net/IP.ADD.RES.SS:12900"}

I'm not sure where it is getting that, but the rest address is definitely wrong.

[avongluck-r1soft]

(I forgot to mention I saw the error above after moving to the /api proxy method from the 12900 proxy method) It is almost like the node is referenced by 12900 somewhere it shouldn't be.

[avongluck-r1soft]

good lord. I had to add the LetsEncrypt CA to the java keystore...

/opt/jdk1.8.0_91/bin/keytool -trustcacerts -keystore cacerts -storepass changeit -noprompt -importcert -file /etc/letsencrypt/live/XXX.XXX.net/chain.pem

[aekondratiev]

btw for now java latest version works without any imports. mine 1.8.0_161-b12 work fine with letsencrypt.