Che: Devfile registry deployment error 'standard_init_linux.go:211: exec user process caused "no such file or directory"'

Created on 7 Jul 2020  ·  16Comments  ·  Source: eclipse/che

Describe the bug

chectl/0.0.20200702-next.080d31e linux-x64 node-v10.21.0 failed to deploy Eclipse Che nighlty on minikube because of devfile registry deployment error:
https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/basic-MultiUser-Che-check-e2e-tests-against-k8s/1942/console

command to install Eclipse Che

chectl server:start --k8spodreadytimeout=180000 --listr-renderer=verbose --platform=minikube --che-operator-cr-patch-yaml=custom-resource-patch.yaml

custom-resource-patch.yaml

spec:
  server:
    cheImage: 'quay.io/eclipse/che-server'
    cheImageTag: 'nightly'
  auth:
    updateAdminPassword: false

devfile registry deployment log

standard_init_linux.go:211: exec user process caused "no such file or directory"

crwctl output

› Current Kubernetes context: 'minikube'
› Installer type is set to: 'operator'
[21:32:15] Verify Kubernetes API [started]
[21:32:15] Verify Kubernetes API...OK [title changed]
[21:32:15] Verify Kubernetes API...OK [completed]
[21:32:15] 👀  Looking for an already existing Eclipse Che instance [started]
[21:32:15] Verify if Eclipse Che is deployed into namespace "che" [started]
[21:32:15] Verify if Eclipse Che is deployed into namespace "che"...it is not [title changed]
[21:32:15] Verify if Eclipse Che is deployed into namespace "che"...it is not [completed]
[21:32:15] 👀  Looking for an already existing Eclipse Che instance [completed]
[21:32:15] ✈️  Minikube preflight checklist [started]
[21:32:15] Verify if kubectl is installed [started]
[21:32:15] Verify if kubectl is installed [completed]
[21:32:15] Verify if minikube is installed [started]
[21:32:15] Verify if minikube is installed [completed]
[21:32:15] Verify if minikube is running [started]
[21:32:15] Verify if minikube is running [completed]
[21:32:15] Start minikube [started]
[21:32:15] Start minikube [skipped]
[21:32:15] → Minikube is already running.
[21:32:15] Check Kubernetes version [started]
[21:32:15] Check Kubernetes version: Found v1.14.3. [title changed]
[21:32:15] Check Kubernetes version: Found v1.14.3. [completed]
[21:32:15] Verify if minikube ingress addon is enabled [started]
[21:32:15] Verify if minikube ingress addon is enabled [completed]
[21:32:15] Enable minikube ingress addon [started]
[21:32:15] Enable minikube ingress addon [skipped]
[21:32:15] → Ingress addon is already enabled.
[21:32:15] Retrieving minikube IP and domain for ingress URLs [started]
[21:32:15] Retrieving minikube IP and domain for ingress URLs...10.0.103.203.nip.io. [title changed]
[21:32:15] Retrieving minikube IP and domain for ingress URLs...10.0.103.203.nip.io. [completed]
[21:32:15] Checking minikube version [started]
[21:32:15] Checking minikube version... 1.1.1 [title changed]
[21:32:15] Checking minikube version... 1.1.1 [completed]
[21:32:15] Check if cluster accessible [started]
[21:32:15] Check if cluster accessible [skipped]
[21:32:15] ✈️  Minikube preflight checklist [completed]
Eclipse Che logs will be available in '/tmp/chectl-logs/1594085535106'
[21:32:15] Start following logs [started]
[21:32:15] Start following Operator logs [started]
[21:32:15] Start following Operator logs...done [title changed]
[21:32:15] Start following Operator logs...done [completed]
[21:32:15] Start following Eclipse Che logs [started]
[21:32:15] Start following Eclipse Che logs...done [title changed]
[21:32:15] Start following Eclipse Che logs...done [completed]
[21:32:15] Start following Postgres logs [started]
[21:32:15] Start following Postgres logs...done [title changed]
[21:32:15] Start following Postgres logs...done [completed]
[21:32:15] Start following Keycloak logs [started]
[21:32:15] Start following Keycloak logs...done [title changed]
[21:32:15] Start following Keycloak logs...done [completed]
[21:32:15] Start following Plugin registry logs [started]
[21:32:15] Start following Plugin registry logs...done [title changed]
[21:32:15] Start following Plugin registry logs...done [completed]
[21:32:15] Start following Devfile registry logs [started]
[21:32:15] Start following Devfile registry logs...done [title changed]
[21:32:15] Start following Devfile registry logs...done [completed]
[21:32:15] Start following logs [completed]
[21:32:15] Start following events [started]
[21:32:15] Start following namespace events [started]
[21:32:15] Start following namespace events...done [title changed]
[21:32:15] Start following namespace events...done [completed]
[21:32:15] Start following events [completed]
[21:32:15] 🏃‍  Running the Eclipse Che operator [started]
[21:32:15] Copying operator resources [started]
[21:32:15] Copying operator resources...done. [title changed]
[21:32:15] Copying operator resources...done. [completed]
[21:32:15] Create Namespace (che) [started]
[21:32:16] Create Namespace (che)...done. [title changed]
[21:32:16] Create Namespace (che)...done. [completed]
[21:32:16] Create ServiceAccount che-operator in namespace che [started]
[21:32:16] Create ServiceAccount che-operator in namespace che...done. [title changed]
[21:32:16] Create ServiceAccount che-operator in namespace che...done. [completed]
[21:32:16] Create Role che-operator in namespace che [started]
[21:32:16] Create Role che-operator in namespace che...done. [title changed]
[21:32:16] Create Role che-operator in namespace che...done. [completed]
[21:32:16] Create ClusterRole che-operator [started]
[21:32:16] Create ClusterRole che-operator...done. [title changed]
[21:32:16] Create ClusterRole che-operator...done. [completed]
[21:32:16] Create RoleBinding che-operator in namespace che [started]
[21:32:16] Create RoleBinding che-operator in namespace che...done. [title changed]
[21:32:16] Create RoleBinding che-operator in namespace che...done. [completed]
[21:32:16] Create ClusterRoleBinding che-operator [started]
[21:32:16] Create ClusterRoleBinding che-operator...done. [title changed]
[21:32:16] Create ClusterRoleBinding che-operator...done. [completed]
[21:32:16] Create CRD checlusters.org.eclipse.che [started]
[21:32:16] Create CRD checlusters.org.eclipse.che...done. [title changed]
[21:32:16] Create CRD checlusters.org.eclipse.che...done. [completed]
[21:32:16] Waiting 5 seconds for the new Kubernetes resources to get flushed [started]
[21:32:21] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [title changed]
[21:32:21] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [completed]
[21:32:21] Create deployment che-operator in namespace che [started]
[21:32:21] Create deployment che-operator in namespace che...done. [title changed]
[21:32:21] Create deployment che-operator in namespace che...done. [completed]
[21:32:21] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace che [started]
[21:32:21] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace che...done. [title changed]
[21:32:21] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace che...done. [completed]
[21:32:21] 🏃‍  Running the Eclipse Che operator [completed]
[21:32:21] ✅  Post installation checklist [started]
[21:32:21] PostgreSQL pod bootstrap [started]
[21:32:21] scheduling [started]
[21:32:32] scheduling...done. [title changed]
[21:32:32] scheduling...done. [completed]
[21:32:32] downloading images [started]
[21:32:42] downloading images...done. [title changed]
[21:32:42] downloading images...done. [completed]
[21:32:42] starting [started]
[21:32:57] starting...done. [title changed]
[21:32:57] starting...done. [completed]
[21:32:57] PostgreSQL pod bootstrap [completed]
[21:32:57] Keycloak pod bootstrap [started]
[21:32:57] scheduling [started]
[21:32:57] scheduling...done. [title changed]
[21:32:57] scheduling...done. [completed]
[21:32:57] downloading images [started]
[21:33:14] downloading images...done. [title changed]
[21:33:14] downloading images...done. [completed]
[21:33:14] starting [started]
[21:33:51] starting...done. [title changed]
[21:33:51] starting...done. [completed]
[21:33:51] Keycloak pod bootstrap [completed]
[21:33:51] Devfile registry pod bootstrap [started]
[21:33:51] scheduling [started]
[21:34:37] scheduling...done. [title changed]
[21:34:37] scheduling...done. [completed]
[21:34:37] downloading images [started]
[21:34:40] downloading images...done. [title changed]
[21:34:40] downloading images...done. [completed]
[21:34:40] starting [started]
[21:37:43] starting [failed]
[21:37:43] → ERR_TIMEOUT: Timeout set to pod ready timeout 180000
[21:37:43] Devfile registry pod bootstrap [failed]
[21:37:43] → ERR_TIMEOUT: Timeout set to pod ready timeout 180000
[21:37:43] ✅  Post installation checklist [failed]
[21:37:43] → ERR_TIMEOUT: Timeout set to pod ready timeout 180000
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 180000
 ›   Installation failed, check logs in '/tmp/chectl-logs/1594085535106'

Operator logs

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/basic-MultiUser-Che-check-e2e-tests-against-k8s/1942/artifact/logs-and-configs/che-logs/che-operator-6f44c784c7-fw6t6.pod.log

Che version

  • [ ] latest
  • [x] nightly
  • [ ] other: please specify

Steps to reproduce

  1. Download chectl/0.0.20200702-next.080d31e linux-x64 node-v10.21.0
  2. Install minikube 1.1.1
  3. Deploy Eclipse Che to minikube by the command:
chectl server:start --k8spodreadytimeout=180000 --listr-renderer=verbose --platform=minikube --che-operator-cr-patch-yaml=custom-resource-patch.yaml

Expected behavior

Eclipse Che has been deployed successfully.

Runtime

  • [ ] kubernetes (include output of kubectl version)
  • [ ] Openshift (include output of oc version)
  • [x] minikube 1.1.1 (include output of minikube version and kubectl version)
  • [x] minishift 3.11 (include output of minishift version and oc version)
  • [ ] docker-desktop + K8S (include output of docker version and kubectl version)
  • [ ] other: (please specify)

Screenshots

Installation method

  • [x] chectl
chectl server:start --k8spodreadytimeout=180000 --listr-renderer=verbose --platform=minikube --che-operator-cr-patch-yaml=custom-resource-patch.yaml
  • [ ] OperatorHub
  • [ ] I don't know

Environment

  • [ ] my computer

    • [ ] Windows

    • [ ] Linux

    • [ ] macOS

  • [ ] Cloud

    • [ ] Amazon

    • [ ] Azure

    • [ ] GCE

    • [ ] other (please specify)

  • [x] other: CRW CCI

Eclipse Che Logs

Additional context

kinbug severitblocker teadeploy
Source
picture dmytro-ndp

Most helpful comment

I've opened two PRs https://github.com/eclipse/che-plugin-registry/pull/522 and https://github.com/eclipse/che-devfile-registry/pull/264

picture benoitf on 7 Jul 2020
👍2

All 16 comments

docker run -ti quay.io/eclipse/che-devfile-registry:nightly
standard_init_linux.go:211: exec user process caused "no such file or directory"

[12:28:15] tolusha@fractal ~/projects $ docker images | grep quay.io/eclipse/che-devfile-registry
quay.io/eclipse/che-devfile-registry                                                               nightly              f8b94cf05319        9 hours ago         65.3MB
tolusha picture tolusha on 7 Jul 2020

Meanwhile I build a new devfile registry image locally and it run successfully.
Could we trigger a CI to rebuild the image?

tolusha picture tolusha on 7 Jul 2020

https://quay.io/repository/eclipse/che-devfile-registry?tab=tags
The image size 19.1 MB instead of > 100MB. So, it is broken

tolusha picture tolusha on 7 Jul 2020

The same true for plugin registry.

tolusha picture tolusha on 7 Jul 2020

Could we trigger a CI to rebuild the image?

I don't have permission to trigger CI job manually https://ci.centos.org/view/Devtools/job/devtools-che-che-nightly/

dmytro-ndp picture dmytro-ndp on 7 Jul 2020

we could override nightly image on quay with previous sha1

benoitf picture benoitf on 7 Jul 2020

done

benoitf picture benoitf on 7 Jul 2020

@benoitf: I have already asked ci.centos team to rebuild https://ci.centos.org/view/Devtools/job/devtools-che-che-nightly/

dmytro-ndp picture dmytro-ndp on 7 Jul 2020

yes, it's ok as well, but during the meantime, image is ok

benoitf picture benoitf on 7 Jul 2020

Installation of Eclipse Che has been restored after rebuilding of nightly images https://ci.centos.org/view/Devtools/job/devtools-che-che-nightly/210/

dmytro-ndp picture dmytro-ndp on 7 Jul 2020

FYI I have updated as well the che plugin registry nightly image (else it was still failing a couple of hours ago)

benoitf picture benoitf on 7 Jul 2020
👍1

still getting this error for the plugin registry after building it locally using the build.sh script within the last two hours. bash seems to be missing from the built image, the entrypoint is a bash script. not sure what's going on with that.

cjsteele picture cjsteele on 7 Jul 2020

@cjsteele looking at https://quay.io/repository/eclipse/che-plugin-registry?tab=tags nightly tag is ok but others seem not

100MB vs 19MB

benoitf picture benoitf on 7 Jul 2020
👍2

I've opened two PRs https://github.com/eclipse/che-plugin-registry/pull/522 and https://github.com/eclipse/che-devfile-registry/pull/264

benoitf picture benoitf on 7 Jul 2020
👍2

@benoitf I applied your patches to my registries and they are recovered again. 👍

monaka picture monaka on 8 Jul 2020

related PRs are merged, closing

benoitf picture benoitf on 8 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

JamesDrummond picture JamesDrummond  ·  3Comments
LaneGeek picture LaneGeek  ·  3Comments
sudheerherle picture sudheerherle  ·  3Comments
johnmcollier picture johnmcollier  ·  3Comments
Ohrimenko1988 picture Ohrimenko1988  ·  3Comments