Che: Che server pod fails to connect to keycloak with self-signed TLS cert

Created on 8 Aug 2020 · 7Comments · Source: eclipse/che

# Describe the bug
Have been testing Che since April. Wanted to upgrade Che to the latest version.
After deleting existing Che install with chectl server:delete and trying to reinstall with chectl server:start --platform=k8s --installer=helm --domain=projectname-eks.myorg.com --multiuser --self-signed-cert -i quay.io/eclipse/che-server:7.16.2 the Che server install fails to with a timeout.

In the logs, it is failing to retrieve the OpenID config Error injecting constructor, java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.projectname-eks.myorg.com/auth/realms/che/.well-known/openid-configuration which seems to be caused by the Che server not trusting the certificate Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

The cert-manager CA is the one that was installed in April during the initial setup of Che.
I am able to reach the https://keycloak-che.projectname-eks.myorg.com/auth/realms/che/.well-known/openid-configuration endpoint from a browser with the CA cert installed. I can also curl the endpoint from another pod in the cluster (if I ignore the cert).

Che version

[ ] latest
[x] nightly
[x] other: server:7.16.2
I've tried both.

Steps to reproduce

chectl server:delete on working server installation
chectl server:start --platform=k8s --installer=helm --domain=projectname-eks.myorg.com --multiuser --self-signed-cert -i quay.io/eclipse/che-server:7.16.2 on same eks cluster

Expected behavior

Che server is able to retrieve the keycloak info with the self-signed cert

Runtime

[x] kubernetes (Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-fd1ea7", GitCommit:"fd1ea7c64d0e3ccbf04b124431c659f65330562a", GitTreeState:"clean", BuildDate:"2020-05-28T19:06:00Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"})
[ ] Openshift (include output of oc version)
[ ] minikube (include output of minikube version and kubectl version)
[ ] minishift (include output of minishift version and oc version)
[ ] docker-desktop + K8S (include output of docker version and kubectl version)
[ ] other: (please specify)

Screenshots

Installation method

[x] chectl - helm

PS C:\Users\jwalton> chectl server:delete
› Current Kubernetes context: 'arn:aws:eks:us-east-1:11111111111:cluster/projectname-eks-1'
You're going to remove Eclipse Che server in namespace 'che' on server 'https://11111111111111111.yl4.us-east-1.eks.amazonaws.com'. If you want to continue - press Y: y
  √ Verify Kubernetes API...OK
  √ Verify if Eclipse Che is deployed into namespace "che"
  √ Delete the Custom Resource of type checlusters.org.eclipse.che...OK
  √ Delete role binding che-operator...OK
  √ Delete role che-operator...OK
  √ Delete cluster role binding che-operator...OK
  √ Delete cluster role che-operator...OK
  √ Delete server and workspace rolebindings...OK
  √ Delete service accounts che-operator...OK
  √ Delete PVC che-operator...OK
  √ Check if OLM is pre-installed on the platform: false...OK
  √ Delete(OLM) custom catalog source eclipse-che-custom-catalog-source...OK
  √ Delete all deployments...OK
  √ Delete all services...OK
  √ Delete all ingresses...OK
  √ Delete configmaps for Eclipse Che server and operator...OK
  √ Delete rolebindings che, che-workspace-exec and che-workspace-view...OK
  √ Delete service accounts che, che-workspace...OK
  √ Delete PVC postgres-data and che-data-volume...OK
  √ Purge Eclipse Che Helm chart...OK
  √ Wait until Eclipse Che pod is deleted...done.
  √ Wait until Keycloak pod is deleted...done.
  √ Wait until Postgres pod is deleted...done.
  √ Wait until Plugin registry pod is deleted...done.
PS C:\Users\jwalton> chectl server:start --platform=k8s --installer=helm --domain=projectname-eks.myorg.com --multiuser --self-signed-cert -i quay.io/eclipse/che-server:7.16.2
› Current Kubernetes context: 'arn:aws:eks:us-east-1:11111111111:cluster/projectname-eks-1'
 »   Warning: "self-signed-cert" flag is deprecated and has no effect. Autodetection is used instead.
  √ Verify Kubernetes API...OK
  √ �  Looking for an already existing Eclipse Che instance
    √ Verify if Eclipse Che is deployed into namespace "che"...it is not
  √ ✈️  Kubernetes preflight checklist
    √ Verify if kubectl is installed
    √ Check Kubernetes version: Found v1.16.8-eks-fd1ea7.
    √ Verify domain is set...set to projectname-eks.myorg.com.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in 'C:\Users\jwalton\AppData\Local\Temp\chectl-logs\1596836763959'
  √ Start following logs
    ↓ Start following Operator logs [skipped]
    √ Start following Eclipse Che logs...done
    √ Start following Postgres logs...done
    √ Start following Keycloak logs...done
    √ Start following Plugin registry logs...done
    √ Start following Devfile registry logs...done
  √ Start following events
    √ Start following namespace events...done
  √ �‍  Running Helm to install Eclipse Che
    √ Check Helm Version: Found v2.16.6+gdd2e569
    √ Create Namespace (che)...does already exist.
    √ Check Eclipse Che TLS certificate...TLS certificate secret found
    √ Create Tiller Role Binding...it already exists.
    √ Create Tiller Service Account...it already exists.
    √ Create Tiller RBAC
    √ Create Tiller Service...it already exists.
    √ Preparing Eclipse Che Helm Chart...done.
    √ Updating Helm Chart dependencies...done.
    √ Deploying Eclipse Che Helm Chart...done.
  > ✅  Post installation checklist
    √ PostgreSQL pod bootstrap
      √ scheduling...done.
      √ downloading images...done.
      √ starting...done.
    √ Devfile registry pod bootstrap
      √ scheduling...done.
      √ downloading images...done.
      √ starting...done.
    √ Plugin registry pod bootstrap
      √ scheduling...done.
      √ downloading images...done.
      √ starting...done.
    > Eclipse Che pod bootstrap
      √ scheduling...done.
      √ downloading images...done.
      × starting
        → ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
    Show important messages
 »   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 »   Installation failed, check logs in 'C:\Users\jwalton\AppData\Local\Temp\chectl-logs\1596836763959'

[ ] OperatorHub
[ ] I don't know

Environment

[ ] my computer
- [ ] Windows
- [ ] Linux
- [ ] macOS
[x] Cloud
- [x] Amazon EKS
- [ ] Azure
- [ ] GCE
- [ ] other (please specify)
[ ] other: please specify

Eclipse Che Logs

2020-08-07 21:50:48,964[ost-startStop-1]  [ERROR] [o.a.c.c.C.[.[localhost].[/api] 175]  - Exception sending context initialized event to listener instance of class [org.eclipse.che.inject.CheBootstrap]
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.projectname-eks.myorg.com/auth/realms/che/.well-known/openid-configuration
  at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:71)
  at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.class(KeycloakSettings.java:54)
  while locating org.eclipse.che.multiuser.keycloak.server.KeycloakSettings
    for the 1st parameter of org.eclipse.che.multiuser.keycloak.server.KeycloakProfileRetriever.<init>(KeycloakProfileRetriever.java:40)
  at org.eclipse.che.multiuser.keycloak.server.KeycloakProfileRetriever.class(KeycloakProfileRetriever.java:33)
  while locating org.eclipse.che.multiuser.keycloak.server.KeycloakProfileRetriever
    for the 1st parameter of org.eclipse.che.multiuser.keycloak.server.dao.KeycloakProfileDao.<init>(KeycloakProfileDao.java:38)
  while locating org.eclipse.che.multiuser.keycloak.server.dao.KeycloakProfileDao
  while locating org.eclipse.che.api.user.server.spi.ProfileDao
    for the 2nd parameter of org.eclipse.che.multiuser.keycloak.server.KeycloakUserManager.<init>(KeycloakUserManager.java:58)
  at org.eclipse.che.multiuser.keycloak.server.KeycloakUserManager.class(KeycloakUserManager.java:58)
  while locating org.eclipse.che.multiuser.keycloak.server.KeycloakUserManager
  while locating org.eclipse.che.multiuser.api.account.personal.PersonalAccountUserManager
  while locating org.eclipse.che.api.user.server.UserManager
Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.projectname-eks.myorg.com/auth/realms/che/.well-known/openid-configuration
    at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:103)
    at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings$$FastClassByGuice$$e0d0786b.newInstance(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
    at com.google.inject.Guice.createInjector(Guice.java:87)
    at org.everrest.guice.servlet.EverrestGuiceContextListener.getInjector(EverrestGuiceContextListener.java:141)
    at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
    at org.everrest.guice.servlet.EverrestGuiceContextListener.contextInitialized(EverrestGuiceContextListener.java:86)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4689)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5155)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:970)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1840)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(Unknown Source)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown Source)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(Unknown Source)
    at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
    at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
    at java.base/java.net.URL.openStream(Unknown Source)
    at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:96)
    ... 52 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
    at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
    at java.base/sun.security.validator.Validator.validate(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
    ... 71 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
    at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
    ... 77 more

Additional context

PS C:\Users\jwalton> kubectl get pod -n che
NAME                               READY   STATUS    RESTARTS   AGE
che-748cf4b4b6-rdl4z               0/1     Running   16         76m
devfile-registry-d9fd7f648-7gcr2   1/1     Running   0          76m
keycloak-c87cdfc65-w8h5p           1/1     Running   0          76m
plugin-registry-58587b799b-kjkxc   1/1     Running   0          76m
postgres-77469cbb7-glqp8           1/1     Running   0          76m
PS C:\Users\jwalton> kubectl get pod -n che che-748cf4b4b6-rdl4z -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2020-08-07T21:46:25Z"
  generateName: che-748cf4b4b6-
  labels:
    app: che
    component: che
    pod-template-hash: 748cf4b4b6
  name: che-748cf4b4b6-rdl4z
  namespace: che
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: che-748cf4b4b6
    uid: 1c696572-af7a-48c1-96c5-1f5a8e196f55
  resourceVersion: "27930261"
  selfLink: /api/v1/namespaces/che/pods/che-748cf4b4b6-rdl4z
  uid: d792ae63-419d-4009-819c-fc2ef047d5c4
spec:
  containers:
  - env:
    - name: OPENSHIFT_KUBE_PING_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: CHE_INFRA_KUBERNETES_TLS__CERT
      valueFrom:
        secretKeyRef:
          key: tls.crt
          name: che-tls
          optional: false
    - name: CHE_INFRA_KUBERNETES_TLS__KEY
      valueFrom:
        secretKeyRef:
          key: tls.key
          name: che-tls
          optional: false
    envFrom:
    - configMapRef:
        name: che
    image: quay.io/eclipse/che-server:7.16.2
    imagePullPolicy: Always
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/system/state
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 120
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    name: che
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    - containerPort: 8000
      name: http-debug
      protocol: TCP
    - containerPort: 8888
      name: jgroups-ping
      protocol: TCP
    - containerPort: 8087
      name: http-metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/system/state
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 60
    resources:
      limits:
        memory: 600Mi
      requests:
        memory: 256Mi
    securityContext:
      runAsUser: 1724
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: che-token-bqbhc
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - env:
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ENDPOINT
      value: postgres
    image: quay.io/eclipse/che-endpoint-watcher:nightly
    imagePullPolicy: IfNotPresent
    name: wait-for-postgres
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: che-token-bqbhc
      readOnly: true
  - env:
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ENDPOINT
      value: keycloak
    image: quay.io/eclipse/che-endpoint-watcher:nightly
    imagePullPolicy: IfNotPresent
    name: wait-for-keycloak
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: che-token-bqbhc
      readOnly: true
  nodeName: ip-10-2-2-4.ec2.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1724
  serviceAccount: che
  serviceAccountName: che
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: che-token-bqbhc
    secret:
      defaultMode: 420
      secretName: che-token-bqbhc
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-08-07T21:47:25Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-08-07T21:46:25Z"
    message: 'containers with unready status: [che]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-08-07T21:46:25Z"
    message: 'containers with unready status: [che]'
    reason: ContainersNotReady
    status: "False"
  - lastProbeTime: null
    lastTransitionTime: "2020-08-07T21:46:25Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://4736691f1ccc551a02238eaa085104998d5479ea1ff21dad9506b071ab8e5a11
    image: quay.io/eclipse/che-server:7.16.2
    imageID: docker-pullable://quay.io/eclipse/che-server@sha256:646a5ec026f081fa8cebd64f0f7101465e8351fe5462504f2b895047d88ae77c
    lastState:
      terminated:
        containerID: docker://5fc2d9d366c2a9a13a1c742db1b4aa73aba079e8b4adbc3ecca5b3e61b68420f
        exitCode: 137
        finishedAt: "2020-08-07T23:03:34Z"
        reason: Error
        startedAt: "2020-08-07T23:00:36Z"
    name: che
    ready: false
    restartCount: 17
    started: true
    state:
      running:
        startedAt: "2020-08-07T23:03:35Z"
  hostIP: 10.2.2.4
  initContainerStatuses:
  - containerID: docker://09873ab6e826b0deb42ffdb284b6b2fa4f7e94423949ed5f8d5f2a2070436be1
    image: quay.io/eclipse/che-endpoint-watcher:nightly
    imageID: docker-pullable://quay.io/eclipse/che-endpoint-watcher@sha256:994c73f642c8b2c62b459aa96d8274419ba359bcb191c7116401a3c3c86ee2c6
    lastState: {}
    name: wait-for-postgres
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://09873ab6e826b0deb42ffdb284b6b2fa4f7e94423949ed5f8d5f2a2070436be1
        exitCode: 0
        finishedAt: "2020-08-07T21:46:53Z"
        reason: Completed
        startedAt: "2020-08-07T21:46:26Z"
  - containerID: docker://58fb4d4ef9ea11d477a1e03a59fb47426f0f3927472c5dd2839cf9e5debd3e40
    image: quay.io/eclipse/che-endpoint-watcher:nightly
    imageID: docker-pullable://quay.io/eclipse/che-endpoint-watcher@sha256:994c73f642c8b2c62b459aa96d8274419ba359bcb191c7116401a3c3c86ee2c6
    lastState: {}
    name: wait-for-keycloak
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://58fb4d4ef9ea11d477a1e03a59fb47426f0f3927472c5dd2839cf9e5debd3e40
        exitCode: 0
        finishedAt: "2020-08-07T21:47:24Z"
        reason: Completed
        startedAt: "2020-08-07T21:46:53Z"
  phase: Running
  podIP: 10.2.2.34
  podIPs:
  - ip: 10.2.2.34
  qosClass: Burstable
  startTime: "2020-08-07T21:46:25Z"

arechectl kinbug severitP1

Source

jwwaltoncredera

All 7 comments

@jwwaltoncredera
To reproduce the issue I need additional information:

What what the previous version of Eclipse Che you installed in April ?
What is chectl version output?

tolusha on 10 Aug 2020

@tolusha

The Che version was a nightly version from April (I ran the command as above without the image flag)
I originally installed chectl in mid April, I'm unsure which version that was but as part of my troubleshooting this issue I upgraded to chectl/0.0.20200731-next.a889d06 win32-x64 node-v10.22.0

jwwaltoncredera on 10 Aug 2020

@jwwaltoncredera
What does kubectl get secret print?
It was the time when we changed secret name from self-signed-cert to self-signed-certificate.
If so

chectl sever:delete
create secret self-signed-certificate the same as 'self-signed-cert`
chectl server:start ...

If you would like to install the latest stable version, pls do

chectl update stable
chectl server:start ... it isn't recommended to use chectl from the next channel and -i flag to specify a stable version of Che to install.

tolusha on 11 Aug 2020

@tolusha
In my testing I haven't seen a secret called self-signed-cert or self-signed-certificate and in my testing I have deleted and let chectl recreate these secrets a couple times.
Here are the current che secrets:

PS C:\Users\jwalton> kubectl get secret -n che
NAME                       TYPE                                  DATA   AGE
che-keycloak-token-fl9zs   kubernetes.io/service-account-token   3      10m
che-tls                    kubernetes.io/tls                     3      10m
che-token-bqbhc            kubernetes.io/service-account-token   3      10m
default-token-sqqsx        kubernetes.io/service-account-token   3      10m

I didn't do any pre-setup of cert-manager or the accompanying certs in April, I let chectl handle that. Is creating the self-signed-certificate ahead of time required?
I will switch my chectl to the stable version, test again, and report back.
Thanks for the help troubleshooting.

jwwaltoncredera on 11 Aug 2020

After deleting the server install and upgrading to stable it worked.
I did get one error on the first run on the namespace:

    × Create Namespace (che)
      → Error from server (AlreadyExists): namespaces "che" already exists

seems like this might be another issue as I would expect the default behavior reuse an existing namespace not error out the installer.
After deleting the namespace the install proceeded as expected :

    √ Check Eclipse Che TLS certificate...going to generate self-signed one
      √ Check Cert Manager deployment...already deployed
      √ Wait for cert-manager...ready
      √ Check Cert Manager CA certificate...already exists
      √ Set up Eclipse Che certificates issuer...already exists
      √ Request self-signed certificate...done
      √ Wait for self-signed certificate...ready
      √ Retrieving Che self-signed CA certificate... is exported to C:\Users\jwalton\cheCA.crt

jwwaltoncredera on 11 Aug 2020

👍1

@jwwaltoncredera
I got it.
We used to store CA certificate in che-tls secret instead of self-signed-certificate one.
It causes problems with updating to a newer version if an old che-tls secret exists in the workspace.
The workaround is to delete che-tls secret (another way is to deploy Eclipse Che in a clean workspace)

tolusha on 12 Aug 2020

👍1

I close this issue since everything works now.

tolusha on 12 Aug 2020

Was this page helpful?

0 / 5 - 0 ratings