Che: PKIX error and impossible to start workspaces on OCP 4.1

[slemeur]

Description

2019-06-21 08:52:06,511[557-wjt8m-47779]  [WARN ] [unknown.jul.logger 49]               - Problem getting Pod json from Kubernetes Client[masterUrl=https://172.30.0.1:443/api/v1, headers={}, connectTimeout=5000, readTimeout=30000, operationAttempts=3, operationSleep=1000, streamProvider=org.openshift.ping.common.stream.TokenStreamProvider@3d3d5e6a] for cluster [EclipseLinkCommandChannel], namespace [che7], labels [app=che]; encountered [java.lang.Exception: 3 attempt(s) with a 1000ms sleep to execute [OpenStream] failed. Last failure was [javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]]

And then:

2019-06-21 08:52:06,965[aceSharedPool-1]  [WARN ] [.i.k.KubernetesInternalRuntime 245]  - Failed to start Kubernetes runtime of workspace workspacef4huz4zllvbwgxom. Cause: Pod creation timeout exceeded. -id: workspacef4huz4zllvbwgxom.workspace -message: null

Reproduction Steps

• Deploy Che 7 RC 2 on OCP 4.1
• Create 5 workspaces
• stop all of them
• try to start a new workspace.

OS and version:
Che 7 RC 2 - OCP 4.1

[davidfestal]

@slemeur could you precise the installation mode, and if installed from the che operator, the detail of your CheCluster Custom resource ?

[skabashnyuk]

might be related to

1. kubernetes.KUBE_PING using token auth does not verify CA and always d.... https://github.com/jgroups-extras/jgroups-kubernetes/pull/69
2. CLOUD-3228 openshift.KUBE_PING doesn't work on OCP 4.1 since it only https://github.com/jboss-openshift/openshift-ping/pull/43

[skabashnyuk]

@mshaposhnik what is the state of this task?

[mshaposhnik]

I did brief investigation. We still can't upgrate to JGroups 4.x due to it is unsupported in Eclipse Link (https://bugs.eclipse.org/bugs/show_bug.cgi?id=531910)
So the only way to get it fixed is to ask jgroups-kubernetes guys to backport theirs CA fix into the 0.9.x banch (called stable) n theirs repo. It can't be done easily (by merge or cherry-pick) since old J-K versions have another code structure, modules etc

[l0rd]

We discussed yesterday that this is not a blocker for 7.0.0 release but will be a blocker for 7.1.0 release. Hence I am labelling it as P1 and setting the milestion to 7.1.0.

[mshaposhnik]

So, i performed an bunch of testing and seems that switch to JGroups version 4.x and KUBE_PING version 1.10+ solves the described error. So the main impediment as for now is accepting PR in the EclipseLinlk (https://github.com/eclipse-ee4j/eclipselink/pull/500) and waiting for release of it.

[ibuziuk]

@l0rd @slemeur looks like https://github.com/eclipse-ee4j/eclipselink/pull/500 it is utterly important to get this PR merged before GA, should we ask someone from foundation to speed-up the review process?

[slemeur]

@ibuziuk this is not for GA, but 7.1.0

[ibuziuk]

@slemeur correct, I was just reviewing issue for 7.1.0 and looks like eclipse-ee4j/eclipselink#500 might be a blocker for it, so just a heads up - we might need to ask / push for review sooner rather than later

[skabashnyuk]

Max already send a review request https://www.eclipse.org/lists/eclipselink-dev/msg07786.html

[ibuziuk]

it looks like no one on eclipse side reviewed https://github.com/eclipse-ee4j/eclipselink/pull/500
@slemeur @l0rd I guess this issue is a blocker for 7.1.0

[mshaposhnik]

PR is merged, let's wait for EL release and the switch to new version.

[skabashnyuk]

waiting for EclipseLink release

[nickboldt]

EL fix was pushed in https://github.com/eclipse-ee4j/eclipselink/pull/500 on Sept 11 for the 2.7 branch. But no new EL releases since 2.6.8 on 19 Jun and 2.7.4 on 18 Jan, so looks like we're still blocked here.

As this blocks https://issues.jboss.org/browse/CRW-304 we need this for Che 7.2 (or 7.3 if upstream EclipseLink won't deliver in time).

Can anyone push on the EclipseLink team to deliver their next 2.7.x release?

[skabashnyuk]

Lukas Jungmann response was

That they wish to have 2.7.5 this month

[nickboldt]

Does that mean in the next 8 days? Or in time for 7.2?

[skabashnyuk]

I believe that was a wish not an obligation. And I doubt that it can be a part of Eclipse Che 7.2.

[nickboldt]

Too bad they don't do releases as part of the Eclipse simrel trains, or we'd have had it last week. :'(