Che: Time out error when downloading images

Created on 4 Apr 2020 · 13Comments · Source: eclipse/che

Describe the bug

Times out when scheduling/downloading images.
Logs indicate timeout waiting for postgres volumes to be bound

Che version

[x] latest

Steps to reproduce

ran
chectl server:start --platform minikube --multiuser

Expected behavior

Eclipse che is deployed and a url is generated

Runtime

[ ] kubernetes (include output of kubectl version)
[ ] Openshift (include output of oc version)
[x] minikube (include output of minikube version and kubectl version)
[ ] minishift (include output of minishift version and oc version)
[ ] docker-desktop + K8S (include output of docker version and kubectl version)
[ ] other: (please specify)

minikube version: v1.9.1
commit: d8747aec7ebf8332ddae276d5f8fb42d3152b5a1

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Screenshots

✔ Verify Kubernetes API...OK
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: Found v1.18.0.
✔ Verify if minikube ingress addon is enabled
✔ Enable minikube ingress addon
✔ Retrieving minikube IP and domain for ingress URLs...172.17.0.2.nip.io.
Eclipse Che logs will be available in '/tmp/chectl-logs/1586030573786'
✔ Start following logs
✔ Start following Operator logs...done
✔ Start following Eclipse Che logs...done
✔ Start following Postgres logs...done
✔ Start following Keycloak logs...done
✔ Start following Plugin registry logs...done
✔ Start following Devfile registry logs...done
✔ Start following events
✔ Start following namespace events...done
✔ 🏃‍ Running the Eclipse Che operator
✔ Copying operator resources...done.
✔ Create Namespace (che)...It already exists.
✔ Create ServiceAccount che-operator in namespace che...It already exists.
✔ Create Role che-operator in namespace che...It already exists.
✔ Create ClusterRole che-operator...It already exists.
✔ Create RoleBinding che-operator in namespace che...It already exists.
✔ Create ClusterRoleBinding che-operator...It already exists.
✔ Create CRD checlusters.org.eclipse.che...It already exists.
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
✔ Create deployment che-operator in namespace che...It already exists.
✔ Create Eclipse Che cluster eclipse-che in namespace che...It already exists.
❯ ✅ Post installation checklist
❯ Eclipse Che pod bootstrap
✖ scheduling
→ ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
downloading images
starting
Retrieving Eclipse Che server URL
Eclipse Che status check
› Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
› Installation failed, check logs in '/tmp/chectl-logs/1586030573786'

Installation method

[x] chectl <-->chectl server:start --platform minikube --multiuser

Environment

[x ] my computer
- [ ] Windows
- [x] Linux
- [ ] macOS
[ ] Cloud
- [ ] Amazon
- [ ] Azure
- [ ] GCE
- [ ] other (please specify)
[ ] other: please specify

Eclipse Che Logs

time="2020-04-04T20:02:45Z" level=info msg="Default 'info' log level is applied"
time="2020-04-04T20:02:45Z" level=info msg="Go Version: go1.12.12"
time="2020-04-04T20:02:45Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-04-04T20:02:45Z" level=info msg="operator-sdk Version: v0.5.0"
time="2020-04-04T20:02:45Z" level=info msg="Operator is running on Kubernetes"
time="2020-04-04T20:02:45Z" level=info msg="Registering Che Components Types"
time="2020-04-04T20:02:45Z" level=info msg="Starting the Cmd"
time="2020-04-04T20:02:45Z" level=info msg="Waiting for PVC postgres-data to be bound. Default timeout: 10 seconds"
time="2020-04-04T20:02:55Z" level=warning msg="Timeout waiting for a PVC postgres-data to be bound. Current phase is Pending"
time="2020-04-04T20:02:55Z" level=warning msg="Sometimes PVC can be bound only when the first consumer is created"
time="2020-04-04T20:02:56Z" level=info msg="Waiting for deployment postgres. Default timeout: 420 seconds"

arechectl kinbug severitP1

Source

cbyreddy

Most helpful comment

storage provision error, yes. Workaround is to use the storageClassName in crd:

minikube creates a VM for setting up the cluster so /data and /data/wksp have to be created and chmod 777 in the vm for this to work. Sames goes to whatever path you choose if you modify this values.

SIDE NOTE: this could also require to disable default tls option in yaml too:
tlsSupport: false

SIDE NOTE2: also the domain should be forced in yaml:
ingressDomain: 'minikube-lan-ip.nip.io'

````yaml

file: /usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml

postgresPVCStorageClassName: eclipseche
workspacePVCStorageClassName: eclipsechewksp
ingressDomain: 'minikube-lan-ip.nip.io' #CHANGE TO A REAL minikube-lan-ip
tlsSupport: false
````

create storage classes and volumes accordingly:
````yaml

file: storageclass_and_volumes.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: eclipsechewksp
labels:
type: local
spec:
storageClassName: eclipsechewksp
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:

path: "/data/wksp"

apiVersion: v1
kind: PersistentVolume
metadata:
name: eclipseche
labels:
type: local
spec:
storageClassName: eclipseche
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:

path: "/data/"

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: eclipsechewksp
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: k8s.io/minikube-hostpath

reclaimPolicy: Retain

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: eclipseche
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain
````

after this use the additional argument in chectl server:start:
bash chectl server:start --platform minikube --multiuser --che-operator-cr-yaml=/usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml

upon attempts to start chectl (using chectl server:delete and server:start again) the postgres folder (called userdata) has to be removed and the volumes in the minikube cluster have to ve removed and created again (using kubectl delete -f and apply -f with the provided yaml).

so to recap:
to remove the unsuccessfull che start garbage files and volumes.
bash chectl server:delete kubectl delete -f <storageclass_and_volumes.yaml> rm -rf /data/userdata
to try again:
bash kubectl apply -f <storageclass_and_volumes.yaml> chectl server:start --platform minikube --multiuser --che-operator-cr-yaml=/usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml

gattytto on 9 Apr 2020

👍2

All 13 comments

@cbyreddy
That's might be the cause. https://github.com/kubernetes/minikube/issues/7218
pls. downgrade minikube to v1.8 and try again.

tolusha on 6 Apr 2020

storage provision error, yes. Workaround is to use the storageClassName in crd:

minikube creates a VM for setting up the cluster so /data and /data/wksp have to be created and chmod 777 in the vm for this to work. Sames goes to whatever path you choose if you modify this values.

SIDE NOTE: this could also require to disable default tls option in yaml too:
tlsSupport: false

SIDE NOTE2: also the domain should be forced in yaml:
ingressDomain: 'minikube-lan-ip.nip.io'

````yaml

file: /usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml

postgresPVCStorageClassName: eclipseche
workspacePVCStorageClassName: eclipsechewksp
ingressDomain: 'minikube-lan-ip.nip.io' #CHANGE TO A REAL minikube-lan-ip
tlsSupport: false
````

create storage classes and volumes accordingly:
````yaml

file: storageclass_and_volumes.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: eclipsechewksp
labels:
type: local
spec:
storageClassName: eclipsechewksp
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:

path: "/data/wksp"

apiVersion: v1
kind: PersistentVolume
metadata:
name: eclipseche
labels:
type: local
spec:
storageClassName: eclipseche
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:

path: "/data/"

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: eclipsechewksp
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: k8s.io/minikube-hostpath

reclaimPolicy: Retain

gattytto on 9 Apr 2020

👍2

https://github.com/che-incubator/chectl/pull/461

gattytto on 9 Apr 2020

@cbyreddy
That's might be the cause. kubernetes/minikube#7218
pls. downgrade minikube to v1.8 and try again.

I was having issues with v1.8 too so I tried again using k3s. I got to a much later stage of the install process before it errored out again.

sudo chectl server:start --platform k8s  --multiuser --domain 192.168.1.137.nip.io
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Kubernetes preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify remote kubernetes status...done.
    ✔ Check Kubernetes version: Found v1.17.4+k3s1.
    ✔ Verify domain is set...set to 192.168.1.137.nip.io.
    ✔ Check if cluster accessible... ok
Eclipse Che logs will be available in '/tmp/chectl-logs/1586681575409'
  ✔ Start following logs
    ↓ Start following Operator logs [skipped]
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
   ✔ 🏃‍  Running Helm to install Eclipse Che
    ✔ Verify if helm is installed
    ✔ Check Helm Version: Found v3.1.2+gd878d4d
    ✔ Create Namespace (che)...done.
    ✔ Check Eclipse Che TLS certificate...going to generate self-signed one
      ✔ Check Cert Manager deployment...not deployed
      ✔ Deploy cert-manager...done
      ✔ Wait for cert-manager...ready
      ✔ Check Cert Manager CA certificate...generating new one
      ✔ Set up Eclipse Che certificates issuer...done
      ✔ Request self-signed certificate...done
      ✔ Wait for self-signed certificate...ready
      ✔ ❗[MANUAL ACTION REQUIRED] Please add local Eclipse Che CA certificate into your browser: /home/admin/cheCA.crt
    ✔ Check Cluster Role Binding...does not exists.
    ✔ Preparing Eclipse Che Helm Chart...done.
    ✔ Updating Helm Chart dependencies...done.
    ✔ Deploying Eclipse Che Helm Chart...done.
  ❯ ✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Devfile registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Plugin registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ❯ Eclipse Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✖ starting
        → ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
    Show important messages
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 ›   Installation failed, check logs in '/tmp/chectl-logs/1586681575409'

Any idea what could have gone wrong now? I don't think it is a storage provisioning issue but I'm not sure.
Here is the log output for the che pod
https://pastebin.com/9NatW21j

cbyreddy on 12 Apr 2020

I have had this problem with both microk8s and k3s!

zarinfam on 12 Apr 2020

@cbyreddy
chectl version ?

Pls provide logs for the second installation.
https://pastebin.com/9NatW21j isn't available

tolusha on 12 Apr 2020

@cbyreddy
chectl version ?

Pls provide logs for the second installation.
https://pastebin.com/9NatW21j isn't available

Sorry, here you go.
https://pastebin.com/aTY9zLRn

This seems to be the issue but I'm not sure

Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching keycloak-che.192.168.1.137.nip.io found

cbyreddy on 12 Apr 2020

tolusha on 13 Apr 2020

related one

16429

Is there anything I can try to fix the error or is it a bug?

cbyreddy on 13 Apr 2020

It is a bug. I will take a look on it later.

What I can suggest for now:

use a lower version of minikube (I personally use 1.5.2)
deploy che with helm installer:

chectl server:start --platform minikube --installer helm --multiuser

tolusha on 13 Apr 2020

@cbyreddy
Could you specify the chectl version you used?

tolusha on 24 Apr 2020

@zarinfam
Could you specify the chectl version you used?

tolusha on 28 Apr 2020

@cbyreddy
I am closing this one.
Feel free to open a new issue.

tolusha on 29 Apr 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Keeps going "offline"

LaneGeek · 3Comments

Cannot install Eclipse Che following docker instructions

redeagle84 · 3Comments

UI usage experience

skabashnyuk · 3Comments

Node server can't be launched inside of "Che-7" container

Ohrimenko1988 · 3Comments

Plugin brokering sometimes fails due to WebSocket closing

johnmcollier · 3Comments