Graylog2-server: 2.0.0-2.beta.1 cluster/id/metrics internal server error

[mikkolehtisalo]

I am testing the 2.0.0-2.beta.1 rpm, and REST API /cluster/id/metrics seems to respond with internal server error.

Each request yields this Exception in the server logs:

2016-03-24T10:22:48.203+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[?:1.8.0_71]
    at okhttp3.internal.io.RealConnection.connectTls(RealConnection.java:195) ~[graylog.jar:?]
    at okhttp3.internal.io.RealConnection.connectSocket(RealConnection.java:148) ~[graylog.jar:?]
    at okhttp3.internal.io.RealConnection.connect(RealConnection.java:111) ~[graylog.jar:?]
    at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:188) ~[graylog.jar:?]
    at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:127) ~[graylog.jar:?]
    at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:97) ~[graylog.jar:?]
    at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:289) ~[graylog.jar:?]
    at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:241) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponse(RealCall.java:240) ~[graylog.jar:?]
    at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:198) ~[graylog.jar:?]
    at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:54) ~[graylog.jar:?]
    at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:187) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:160) ~[graylog.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:57) ~[graylog.jar:?]
    at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
    at org.graylog2.rest.resources.cluster.ClusterMetricsResource.multipleMetrics(ClusterMetricsResource.java:98) ~[graylog.jar:?]
    at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[?:1.8.0_71]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_71]
    at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ~[?:1.8.0_71]
    ... 49 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146) ~[?:1.8.0_71]
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131) ~[?:1.8.0_71]
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_71]
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[?:1.8.0_71]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_71]
    at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ~[?:1.8.0_71]
    ... 49 more
2016-03-24T10:22:50.201+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:1.8.0_71]
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[?:1.8.0_71]
    at okhttp3.internal.io.RealConnection.connectTls(RealConnection.java:195) ~[graylog.jar:?]
    at okhttp3.internal.io.RealConnection.connectSocket(RealConnection.java:148) ~[graylog.jar:?]
    at okhttp3.internal.io.RealConnection.connect(RealConnection.java:111) ~[graylog.jar:?]
    at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:188) ~[graylog.jar:?]
    at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:127) ~[graylog.jar:?]
    at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:97) ~[graylog.jar:?]
    at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:289) ~[graylog.jar:?]
    at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:241) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponse(RealCall.java:240) ~[graylog.jar:?]
    at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:198) ~[graylog.jar:?]
    at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:54) ~[graylog.jar:?]
    at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:187) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:160) ~[graylog.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:57) ~[graylog.jar:?]
    at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
    at org.graylog2.rest.resources.cluster.ClusterMetricsResource.multipleMetrics(ClusterMetricsResource.java:98) ~[graylog.jar:?]
    at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[?:1.8.0_71]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_71]
    at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ~[?:1.8.0_71]
    ... 49 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146) ~[?:1.8.0_71]
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131) ~[?:1.8.0_71]
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_71]
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[?:1.8.0_71]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_71]
    at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_71]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_71]
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ~[?:1.8.0_71]
    ... 49 more

[dennisoelkers]

It looks like you have SSL enabled, but the SSL certificate is not validatable by the Graylog server. When fetching cluster metrics, the Graylog server goes and connects to each Graylog node in the cluster fetching the metrics, even to itself and even if it's only a single-node cluster. Therefore the SSL cert of the HTTPS endpoint of the server needs to be properly validatable, which means that you would need to add the CA cert or the node cert itself to all server's keychains.

[mikkolehtisalo]

It connects each Graylog node based on what? I have not knowingly enabled SSL.

rest_transport_uri is set to https address because there is load balancer in front (doing the SSL for browsers, reverse proxying http requests to Graylog). That setting is required, or the web-ui will fail to point to the correct address. I can not find any other relevant enabled configration stanzas from the configuration file.

I'm fine with adding the same certificate to the keychains, as long as I can figure out how to do that.

Although rest_enable_tls = false, should I still define rest_tls_cert_file and rest_tls_key_file ?

[mikkolehtisalo]

After some researching:

• If you use a load balancer, or any other reverse proxy in front of Graylog, you have to set rest_transport_uri. Otherwise the /config.js will point to the internal non-available addresses and the web-ui will fail.
• If you set rest_transport_uri it is used not only for building /config.js, but also to configure the way Graylog nodes call their respective REST APIs

IMHO the web-ui and internal configuration should be separated. In any case, I found two separate ways to go around this:

Approach 1

Get the certificate and private keys belonging to the external address. These are probably on the load balancer, or something. Configure REST API to use TLS:

rest_enable_tls = true
rest_tls_cert_file = /etc/ssl/certs/external-cert.pem
rest_tls_key_file = /etc/ssl/private/external-cert.pkcs8

Add the certificate to a Java Keystore (alternatively could add to the Java default ks, but you probably shouldn't use the default because Graylog would start trusting the default certs as well):

keytool -import -trustcacerts -alias root -file /etc/ssl/certs/external-cert.pem -keystore /etc/ssl/graylog.jks

Configure the Java process of graylog-server to use the keystore:

-Djavax.net.ssl.trustStore=/etc/ssl/graylog.jks -Djavax.net.ssl.trustStorePassword=password

Point the reverse proxy to https://node:12900/ instead of http://node:12900/.

Approach 2

Make a static config.js file:

window.appConfig = {
  gl2ServerUrl: 'https://graylog.mydomain.com:443/',
  gl2AppPathPrefix: '',
  rootTimeZone: 'Etc/UTC',
};

Configure the front-end load-balancer to serve that file. Testing on Apache I had to just exclude that address from ProxyPass, and save the file to /var/www/html/:

ProxyPass /config.js !

(I like the simplicity.)

[dennisoelkers]

Thanks for writing this down. Approach 2 is what we had in mind for advanced users like you. Bundling the web interface with the server is for deployment simplicity (and feasible in most cases) but not required, so they are still separateable. The issue at hand is the lack of documentation, so if you are fine with that we will add a slightly modified version of yours to our docs.