Activiti: connection between activiti modeler frontend and backend in activiti-cloud-full-example

Created on 5 Mar 2019 · 5Comments · Source: Activiti/Activiti

Hello,

Problem on nginx-ingress-1.3.1 app version 0.22.0 (everything works on nginx-ingress-1.1.2 app version 0.21.0)

Full-example run on docker-for-desktop with kubernetes. Everything run correct, can login on all apps. Problem with connection between modeler frontend and backend. Cannot create new project in modeler i get timeout. In backend i get error in logs:

2019-03-05 13:17:21.700 ERROR 1 --- [nio-8080-exec-3] o.k.a.rotation.AdapterTokenVerifier : Didn't find publicKey for kid: hMBObsroPIKbIy5NPggSJOHdCrfEQiokj-AB0B0ZKu4
2019-03-05 13:17:21.700 ERROR 1 --- [nio-8080-exec-3] o.k.a.BearerTokenRequestAuthenticator : Failed to verify token org.keycloak.common.VerificationException: Didn't find publicKey for specified kid
at org.keycloak.adapters.rotation.AdapterTokenVerifier.getPublicKey(AdapterTokenVerifier.java:124)
at org.keycloak.adapters.rotation.AdapterTokenVerifier.createVerifier(AdapterTokenVerifier.java:111)
at org.keycloak.adapters.rotation.AdapterTokenVerifier.verifyToken(AdapterTokenVerifier.java:47)
at org.keycloak.adapters.BearerTokenRequestAuthenticator.authenticateToken(BearerTokenRequestAuthenticator.java:103)
at org.keycloak.adapters.BearerTokenRequestAuthenticator.authenticate(BearerTokenRequestAuthenticator.java:88)
at org.keycloak.adapters.RequestAuthenticator.authenticate(RequestAuthenticator.java:68)
at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.authenticateInternal(AbstractKeycloakAuthenticatorValve.java:203)
at org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve.authenticate(KeycloakAuthenticatorValve.java:50)
at org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve.doAuthenticate(KeycloakAuthenticatorValve.java:57)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:572)
at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.invoke(AbstractKeycloakAuthenticatorValve.java:181)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$Wrapping

Source

aleksl

Most helpful comment

To complete my previous observations, I managed to solve the certificate issue by following the documentation of cert-manager.

I needed to update one of the annotation on the Ingress for activiti-cloud-gateway:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
...

In my case, I prefer use a ClusterIssuer as it fit my needs, i.e. more than one namespace to get certificates for.

mathieubodin on 25 Mar 2019

👍2 👀1

All 5 comments

Note this was reported as against the latest activiti charts (full example 1.1.8 for activiti 7.0.0.GA)

@salaboy , @fcorti , @mteodori you'll want to be aware of this

ryandawsonuk on 5 Mar 2019

Hi, I met something similar, despite the fact I'm running cloud-modeling against nginx-ingress-1.2.3 app version 0.22.0.

I managed to log in but once I land on the dashboard page I get a red warning Unknown error happened during the loading of projects.:

issue_activiti#2607_chrome_GET

Then I tried to create a project with the same results:

issue_activiti#2607_chrome_POST

I checked the logs of activiti-cloud-modeling-backend and noticed the following stack:

2019-03-23 09:06:51.890 ERROR 1 --- [nio-8080-exec-1] o.k.a.rotation.JWKPublicKeyLocator       : Error when sending request to retrieve realm keys

org.keycloak.adapters.HttpClientAdapterException: IO error
        at org.keycloak.adapters.HttpAdapterUtils.sendJsonHttpRequest(HttpAdapterUtils.java:58)
        at org.keycloak.adapters.rotation.JWKPublicKeyLocator.sendRequest(JWKPublicKeyLocator.java:99)
        at org.keycloak.adapters.rotation.JWKPublicKeyLocator.getPublicKey(JWKPublicKeyLocator.java:63)
        at org.keycloak.adapters.rotation.AdapterTokenVerifier.getPublicKey(AdapterTokenVerifier.java:121)
        at org.keycloak.adapters.rotation.AdapterTokenVerifier.createVerifier(AdapterTokenVerifier.java:111)
        at org.keycloak.adapters.rotation.AdapterTokenVerifier.verifyToken(AdapterTokenVerifier.java:47)
        at org.keycloak.adapters.BearerTokenRequestAuthenticator.authenticateToken(BearerTokenRequestAuthenticator.java:103)
        at org.keycloak.adapters.BearerTokenRequestAuthenticator.authenticate(BearerTokenRequestAuthenticator.java:88)
        at org.keycloak.adapters.RequestAuthenticator.authenticate(RequestAuthenticator.java:68)
        at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.authenticateInternal(AbstractKeycloakAuthenticatorValve.java:203)
        at org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve.authenticate(KeycloakAuthenticatorValve.java:50)
        at org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve.doAuthenticate(KeycloakAuthenticatorValve.java:57)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:572)
        at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.invoke(AbstractKeycloakAuthenticatorValve.java:181)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:642)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:461)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
        at org.apache.http.conn.ssl.SSLSocketFactory.createLayeredSocket(SSLSocketFactory.java:573)
        at org.keycloak.adapters.SniSSLSocketFactory.createLayeredSocket(SniSSLSocketFactory.java:114)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:557)
        at org.keycloak.adapters.SniSSLSocketFactory.connectSocket(SniSSLSocketFactory.java:109)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:134)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.keycloak.adapters.HttpAdapterUtils.sendJsonHttpRequest(HttpAdapterUtils.java:37)
        ... 26 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:221)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:626)
        ... 51 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
        ... 57 common frames omitted

2019-03-23 09:06:51.890 ERROR 1 --- [nio-8080-exec-1] o.k.a.rotation.AdapterTokenVerifier      : Didn't find publicKey for kid: hMBObsroPIKbIy5NPggSJOHdCrfEQiokj-AB0B0ZKu4
2019-03-23 09:06:51.890 ERROR 1 --- [nio-8080-exec-1] o.k.a.BearerTokenRequestAuthenticator    : Failed to verify token

org.keycloak.common.VerificationException: Didn't find publicKey for specified kid
        at org.keycloak.adapters.rotation.AdapterTokenVerifier.getPublicKey(AdapterTokenVerifier.java:124)
        at org.keycloak.adapters.rotation.AdapterTokenVerifier.createVerifier(AdapterTokenVerifier.java:111)
        at org.keycloak.adapters.rotation.AdapterTokenVerifier.verifyToken(AdapterTokenVerifier.java:47)
        at org.keycloak.adapters.BearerTokenRequestAuthenticator.authenticateToken(BearerTokenRequestAuthenticator.java:103)
        at org.keycloak.adapters.BearerTokenRequestAuthenticator.authenticate(BearerTokenRequestAuthenticator.java:88)
        at org.keycloak.adapters.RequestAuthenticator.authenticate(RequestAuthenticator.java:68)
        at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.authenticateInternal(AbstractKeycloakAuthenticatorValve.java:203)
        at org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve.authenticate(KeycloakAuthenticatorValve.java:50)
        at org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve.doAuthenticate(KeycloakAuthenticatorValve.java:57)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:572)
        at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.invoke(AbstractKeycloakAuthenticatorValve.java:181)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

It seems the issue might be the self generated certificate that isn't accepted by the JVM running the backend.

IMHO, It could be fixed with using valid and recognized certificates, such as those issued by let's encrypt.

For the record, my K8s is running in Azure, and I get the same behavior with nginx-ingress-1.1.2 app version 0.21.0.

mathieubodin on 23 Mar 2019

To complete my previous observations, I managed to solve the certificate issue by following the documentation of cert-manager.

I needed to update one of the annotation on the Ingress for activiti-cloud-gateway:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
...

In my case, I prefer use a ClusterIssuer as it fit my needs, i.e. more than one namespace to get certificates for.

mathieubodin on 25 Mar 2019

👍2 👀1

@mathieubodin great that you sort it out.. It will be great for documentation purposes if you can contribute with a short text on why you needed to do that, and which files from our HELM charts / Kubernetes resources (Ingress) in this case you needed to change for your particular setup. This might help a lot of users. We can create a FAQ section into our getting started guides where we can include such information.