Che: Che-server fails to bootstrap on minikube v1.9.2

Created on 14 May 2020  Β·  15Comments  Β·  Source: eclipse/che

Describe the bug

Deploying Che with self signed TLS certificates on Kubernetes fails with timeout error Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined.

Che version

  • [x] latest
  • [ ] nightly
  • [ ] other: please specify

Steps to reproduce

Deploy Che using chectl chectl server:start --platform=minikube --installer=operator --self-signed-cert

Expected behavior

Runtime

  • [ ] kubernetes (include output of kubectl version)
  • [ ] Openshift (include output of oc version)
  • [x] minikube (include output of minikube version and kubectl version)
  • [ ] minishift (include output of minishift version and oc version)
  • [ ] docker-desktop + K8S (include output of docker version and kubectl version)
  • [ ] other: (please specify)

Screenshots

Installation method

  • [x] chectl

    • provide a full command that was used to deploy Eclipse Che (including the output)

    • provide an output of chectl version command

  • [ ] OperatorHub
  • [ ] I don't know

Environment

  • [ ] my computer

    • [ ] Windows

    • [ ] Linux

    • [x] macOS

  • [ ] Cloud

    • [ ] Amazon

    • [ ] Azure

    • [ ] GCE

    • [ ] other (please specify)

  • [ ] other: please specify

chectl logs

Set current context to 'minikube'
  βœ” Verify Kubernetes API...OK
  βœ” πŸ‘€  Looking for an already existing Eclipse Che instance
    βœ” Verify if Eclipse Che is deployed into namespace "che"...it is not
  βœ” ✈️  Minikube preflight checklist
    βœ” Verify if kubectl is installed
    βœ” Verify if minikube is installed
    βœ” Verify if minikube is running
    ↓ Start minikube [skipped]
      β†’ Minikube is already running.
    βœ” Check Kubernetes version: Found v1.18.0.
    βœ” Verify if minikube ingress addon is enabled
    βœ” Enable minikube ingress addon
    βœ” Retrieving minikube IP and domain for ingress URLs...192.168.64.2.nip.io.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589471418284'
  βœ” Start following logs
    βœ” Start following Operator logs...done
    βœ” Start following Eclipse Che logs...done
    βœ” Start following Postgres logs...done
    βœ” Start following Keycloak logs...done
    βœ” Start following Plugin registry logs...done
    βœ” Start following Devfile registry logs...done
  βœ” Start following events
    βœ” Start following namespace events...done
  βœ” πŸƒβ€  Running the Eclipse Che operator
    βœ” Copying operator resources...done.
    βœ” Create Namespace (che)...It already exists.
    βœ” Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    βœ” Create ServiceAccount che-operator in namespace che...It already exists.
    βœ” Create Role che-operator in namespace che...It already exists.
    βœ” Create ClusterRole che-operator...It already exists.
    βœ” Create RoleBinding che-operator in namespace che...It already exists.
    βœ” Create ClusterRoleBinding che-operator...It already exists.
    βœ” Create CRD checlusters.org.eclipse.che...It already exists.
    βœ” Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    βœ” Create deployment che-operator in namespace che...It already exists.
    βœ” Create Eclipse Che cluster eclipse-che in namespace che...It already exists.
  ❯ βœ…  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      βœ– scheduling
        β†’ ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
        downloading images
        starting
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 β€Ί   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
 β€Ί   Installation failed, check logs in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589471418284'

minikube version

minikube version: v1.9.2

chectl version

chectl/7.12.2 darwin-x64 node-v10.13.0

areinstall kinbug severitP1
Source
picture Divya063

All 15 comments

@Divya063
Thank you for reporting.
Currently it is not possible to deploy Eclipse Che on minikube 1.9.2 due to https://github.com/kubernetes/minikube/issues/7218
There is workaround https://github.com/kubernetes/minikube/issues/7218#issuecomment-611364740
Another option: try minikube 1.8.2

tolusha picture tolusha on 15 May 2020
👍1

@tolusha Thank you for your response! I'm now using minikube 1.8.2 and still facing the same error.

chectl server:start --platform=minikube --installer=operator --self-signed-cert                                 ─╯
Set current context to 'minikube'
  βœ” Verify Kubernetes API...OK
  βœ” πŸ‘€  Looking for an already existing Eclipse Che instance
    βœ” Verify if Eclipse Che is deployed into namespace "che"...it is not
  βœ” ✈️  Minikube preflight checklist
    βœ” Verify if kubectl is installed
    βœ” Verify if minikube is installed
    βœ” Verify if minikube is running
    ↓ Start minikube [skipped]
      β†’ Minikube is already running.
    βœ” Check Kubernetes version: Found v1.18.0.
    βœ” Verify if minikube ingress addon is enabled
    βœ” Enable minikube ingress addon
    βœ” Retrieving minikube IP and domain for ingress URLs...192.168.64.2.nip.io.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589539010389'
  βœ” Start following logs
    βœ” Start following Operator logs...done
    βœ” Start following Eclipse Che logs...done
    βœ” Start following Postgres logs...done
    βœ” Start following Keycloak logs...done
    βœ” Start following Plugin registry logs...done
    βœ” Start following Devfile registry logs...done
  βœ” Start following events
    βœ” Start following namespace events...done
  βœ” πŸƒβ€  Running the Eclipse Che operator
    βœ” Copying operator resources...done.
    βœ” Create Namespace (che)...It already exists.
    βœ” Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    βœ” Create ServiceAccount che-operator in namespace che...It already exists.
    βœ” Create Role che-operator in namespace che...It already exists.
    βœ” Create ClusterRole che-operator...It already exists.
    βœ” Create RoleBinding che-operator in namespace che...It already exists.
    βœ” Create ClusterRoleBinding che-operator...It already exists.
    βœ” Create CRD checlusters.org.eclipse.che...It already exists.
    βœ” Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    βœ” Create deployment che-operator in namespace che...It already exists.
    βœ” Create Eclipse Che cluster eclipse-che in namespace che...It already exists.
  ❯ βœ…  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      βœ– scheduling
        β†’ ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
        downloading images
        starting
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 β€Ί   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
 β€Ί   Installation failed, check logs in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589539010389'
Divya063 picture Divya063 on 15 May 2020

@Divya063
I don't see that keycloak, postgres and other pods starting up.
Is that the very first install logs ?

tolusha picture tolusha on 15 May 2020

@tolusha After switching to minikube 1.8.2 this is the very first install logs.

Divya063 picture Divya063 on 15 May 2020

@Divya063
And that is very strange, because normally we have logs like that:

 βœ…  Post installation checklist
    βœ” PostgreSQL pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ” starting...done.
    βœ” Keycloak pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ” starting...done.
    βœ” Devfile registry pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ” starting...done.
    βœ” Plugin registry pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ” starting...done.
    ❯ Eclipse Che pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      β ‹ starting
      Retrieving Eclipse Che ser

Any manual work before deploying?

tolusha picture tolusha on 15 May 2020

@tolusha When I first ran chectl server:start --platform=minikube --installer=operator --self-signed-cert using minikube 1.9.2 it was stuck at the PostgreSQL pod:

βœ…  Post installation checklist
    βœ” PostgreSQL pod bootstrap
      βœ” scheduling...

Don't know if its relevant, After that I started following this guide - https://che.eclipse.org/running-eclipse-che-on-kubernetes-using-docker-desktop-for-mac-5d972ed511e1 but some commands were not working. One of the command was this helm upgrade β€” install che β€” namespace che β€” set global.cheWorkspacesNamespace=”che” β€” set global.ingressDomain=${CHE_DOMAIN}.nip.io ./ and the error I got was 

Release β€œche” does not exist. Installing it now.
Error: found in requirements.yaml, but missing in charts/ directory: che-devfile-registry, che-plugin-registry, che-postgres, che-keycloak, che-jaeger, prometheus, grafana`

To fix that I used helm dependency update

Divya063 picture Divya063 on 15 May 2020

@Divya063 Just to check, did you do a minikube delete before starting the 1.8 cluster? I don't believe the k8s version can be changed without recreating the whole VM.

The logs

    βœ” Copying operator resources...done.
    βœ” Create Namespace (che)...It already exists.
    βœ” Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    βœ” Create ServiceAccount che-operator in namespace che...It already exists.
    βœ” Create Role che-operator in namespace che...It already exists.
    βœ” Create ClusterRole che-operator...It already exists.
    βœ” Create RoleBinding che-operator in namespace che...It already exists.
    βœ” Create ClusterRoleBinding che-operator...It already exists.
    βœ” Create CRD checlusters.org.eclipse.che...It already exists.
    βœ” Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    βœ” Create deployment che-operator in namespace che...It already exists.
    βœ” Create Eclipse Che cluster eclipse-che in namespace che...It already exists.

make it look like there's already some Che deployment elements present in the cluster

amisevsk picture amisevsk on 15 May 2020

@amisevsk I didn't do minikube delete. Well, I started again by doing minikube delete:
At first it got stuck at Keycloak pod :

Logs

chectl server:start --platform=minikube --installer=operator --self-signed-cert
Set current context to 'minikube'
  βœ” Verify Kubernetes API...OK
  βœ” πŸ‘€  Looking for an already existing Eclipse Che instance
    βœ” Verify if Eclipse Che is deployed into namespace "che"...it is not
  βœ” ✈️  Minikube preflight checklist
    βœ” Verify if kubectl is installed
    βœ” Verify if minikube is installed
    βœ” Verify if minikube is running
    ↓ Start minikube [skipped]
      β†’ Minikube is already running.
    βœ” Check Kubernetes version: Found v1.17.3.
    βœ” Verify if minikube ingress addon is enabled
    βœ” Enable minikube ingress addon
    βœ” Retrieving minikube IP and domain for ingress URLs...192.168.64.3.nip.io
.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589612585730'
  βœ” Start following logs
    βœ” Start following Operator logs...done
    βœ” Start following Eclipse Che logs...done
    βœ” Start following Postgres logs...done
    βœ” Start following Keycloak logs...done
    βœ” Start following Plugin registry logs...done
    βœ” Start following Devfile registry logs...done
  βœ” Start following events
    βœ” Start following namespace events...done
  βœ” πŸƒβ€  Running the Eclipse Che operator
    βœ” Copying operator resources...done.
    βœ” Create Namespace (che)...It already exists.
    βœ” Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    βœ” Create ServiceAccount che-operator in namespace che...done.
    βœ” Create Role che-operator in namespace che...done.
    βœ” Create ClusterRole che-operator...done.
    βœ” Create RoleBinding che-operator in namespace che...done.
    βœ” Create ClusterRoleBinding che-operator...done.
    βœ” Create CRD checlusters.org.eclipse.che...done.
    βœ” Waiting 5 seconds for the new Kubernetes resources to get flushed...done
.
    βœ” Create deployment che-operator in namespace che...done.
    βœ” Create Eclipse Che cluster eclipse-che in namespace che...done.
  ❯ βœ…  Post installation checklist
    βœ” PostgreSQL pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ” starting...done.
    ❯ Keycloak pod bootstrap
      βœ” scheduling...done.
      βœ– downloading images
        β†’ ERR_TIMEOUT: Timeout set to pod wait timeout 300000
        starting
      Devfile registry pod bootstrap
      Plugin registry pod bootstrap
      Eclipse Che pod bootstrap
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 β€Ί   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000
 β€Ί   Installation failed, check logs in
 β€Ί   '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589612585
 β€Ί   730'

I started again, it jumped to Eclipse Che pod, but failed to start:

Logs

chectl server:start --platform=minikube --installer=operator --self-signed-cert
Set current context to 'minikube'
  βœ” Verify Kubernetes API...OK
  βœ” πŸ‘€  Looking for an already existing Eclipse Che instance
    βœ” Verify if Eclipse Che is deployed into namespace "che"...it is not
  βœ” ✈️  Minikube preflight checklist
    βœ” Verify if kubectl is installed
    βœ” Verify if minikube is installed
    βœ” Verify if minikube is running
    ↓ Start minikube [skipped]
      β†’ Minikube is already running.
    βœ” Check Kubernetes version: Found v1.17.3.
    βœ” Verify if minikube ingress addon is enabled
    βœ” Enable minikube ingress addon
    βœ” Retrieving minikube IP and domain for ingress URLs...192.168.64.3.nip.io
.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589613321876'
  βœ” Start following logs
    βœ” Start following Operator logs...done
    βœ” Start following Eclipse Che logs...done
    βœ” Start following Postgres logs...done
    βœ” Start following Keycloak logs...done
    βœ” Start following Plugin registry logs...done
    βœ” Start following Devfile registry logs...done
  βœ” Start following events
    βœ” Start following namespace events...done
  βœ” πŸƒβ€  Running the Eclipse Che operator
    βœ” Copying operator resources...done.
    βœ” Create Namespace (che)...It already exists.
    βœ” Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    βœ” Create ServiceAccount che-operator in namespace che...It already exists.
    βœ” Create Role che-operator in namespace che...It already exists.
    βœ” Create ClusterRole che-operator...It already exists.
    βœ” Create RoleBinding che-operator in namespace che...It already exists.
    βœ” Create ClusterRoleBinding che-operator...It already exists.
    βœ” Create CRD checlusters.org.eclipse.che...It already exists.
    βœ” Waiting 5 seconds for the new Kubernetes resources to get flushed...done
.
    βœ” Create deployment che-operator in namespace che...It already exists.
    βœ” Create Eclipse Che cluster eclipse-che in namespace che...It already exi
sts.
  ❯ βœ…  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ– starting
        β†’ ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 β€Ί   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 β€Ί   Installation failed, check logs in
 β€Ί   '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589613321
 β€Ί   876'


Next time I ran the command, the logs depicts that Eclipse Che is already deployed.

chectl server:start --platform=minikube --installer=operator --self-signed-cert
Set current context to 'minikube'
  βœ” Verify Kubernetes API...OK
  βœ” πŸ‘€  Looking for an already existing Eclipse Che instance
    βœ” Verify if Eclipse Che is deployed into namespace "che"
      βœ” Found running Eclipse Che deployment
      βœ” Found running postgres deployment
      βœ” Found running keycloak deployment
      βœ” Found running plugin registry deployment
      βœ” Found running devfile registry deployment
 β€Ί   Warning: Deployed Eclipse Che is found and the specified installation
 β€Ί   parameters will be ignored
  βœ” πŸ‘€  Starting already deployed Eclipse Che
    βœ” Scaling up Eclipse Che Deployments...done.
  ❯ βœ…  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      βœ” scheduling...done.
      βœ” downloading images...done.
      βœ– starting
        β†’ ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 β€Ί   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 β€Ί   Installation failed, check logs in
 β€Ί   '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589614448
 β€Ί   795'
Divya063 picture Divya063 on 16 May 2020

pls do.
kubectl delete namespace che
chectl server:start --platform=minikube --installer=operator --k8spodwaittimeout 600000

tolusha picture tolusha on 18 May 2020

--k8spodwaittimeout

that didn't help me.

skabashnyuk picture skabashnyuk on 18 May 2020

to fix deploy using custom storage resources (storage class and persistent volume for workspaces and che Postgres), and add storage class and workspacestorageclass to chectl, if using che-operator then edit the cr.yaml accordingly

gattytto picture gattytto on 23 May 2020

https://github.com/che-incubator/chectl/pull/461

gattytto picture gattytto on 23 May 2020

@skabashnyuk https://github.com/eclipse/che/issues/16545#issuecomment-611363921 here's details on how to. I think there's docs pr on this too but I can't find the link.

gattytto picture gattytto on 23 May 2020
👍1

https://www.eclipse.org/che/docs/che-7/installing-che-using-storage-classes/

tolusha picture tolusha on 25 May 2020

Fixed by https://github.com/che-incubator/chectl/pull/752

tolusha picture tolusha on 8 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

johnmcollier picture johnmcollier  Β·  3Comments
sudheerherle picture sudheerherle  Β·  3Comments
l0rd picture l0rd  Β·  3Comments
JamesDrummond picture JamesDrummond  Β·  3Comments
redeagle84 picture redeagle84  Β·  3Comments