Che: 0/1 nodes are available: 1 Insufficient memory

[tony-bony]

Describe the bug

Error when starting workspace:

Unified Che Plugin Broker
Starting VS Code and Theia plugins processing
Starting Che plugins and editor processing
Downloading VS Code extension for plugin 'ms-python/python/latest'
List of plugins and editors to install
- ms-python/python/latest - Linting, Debugging (multi-threaded, remote), Intellisense, code formatting, refactoring, unit tests, snippets, and more.
- eclipse/che-machine-exec-plugin/0.0.1 - Che Plug-in with che-machine-exec service to provide creation terminal or tasks for Eclipse CHE workspace machines.
- eclipse/che-theia/7.0.0-next - Eclipse Theia, get the latest release each day.
All plugins have been successfully processed
0/1 nodes are available: 1 Insufficient memory.
Error: Failed to run the workspace: "Unrecoverable event occurred: 'FailedScheduling', '0/1 nodes are available: 1 Insufficient memory.', 'workspacell2g2kpcqi40b7ww.python-5d87d587d8-7lct5'"

Che version

• 7.0.0-rc-5.0-SNAPSHOT

Steps to reproduce

1. Open Dashboard
2. Click 'Create Workspace'
3. Select 'Python' (Stack with Python 3.6)
4. Add project 'python-hello-world'
5. Add plugin 'Che machine-exec Service latest'
6. Add plugin 'Eclipse IDE (ws-skeleton publisher) latest'
7. Click 'Create & Open'

Expected behavior

Workspace should start

Runtime

docker --version
Docker version 18.03.0-ce, build 0520e24

docker-compose --version
docker-compose version 1.20.1, build 5d8c71b

docker-machine --version
docker-machine version 0.14.0, build 89b8332

minikube version
minikube version: v1.2.0

kubectl version --client
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T14:25:20Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}

helm version

Client: &version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"clean"}

chectl --version
chectl/0.0.2-a74ad81 darwin-x64 node-v10.4.1

minikube status

host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

minikube start --memory=10240

😄  minikube v1.2.0 on darwin (amd64)
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing virtualbox VM for "minikube" ...
⌛  Waiting for SSH access ...
🐳  Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
🔄  Relaunching Kubernetes v1.15.0 using kubeadm ... 
⌛  Verifying: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"

kubectl get nodes

NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   13h   v1.15.0

kubectl get services
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 13h

kubectl get pod --namespace ch

che-54f4459fb4-jtfms                1/1     Running   0          37m
devfile-registry-549b9bbd5b-jg7h6   1/1     Running   1          12h
plugin-registry-75478b67dd-bnvf2    1/1     Running   1          12h

chectl --help server
control Che server

chectl workspace:list

 ›   Warning: Not implemented yet
 ›   Warning: Not implemented yet
 ›   Warning: Not implemented yet
TypeError: Expected property `task` to be of type `function`, got `undefined`
    at new Task (/snapshot/chectl/node_modules/listr/lib/task.js:25:10)
    at Listr.add (/snapshot/chectl/node_modules/listr/index.js:71:21)
    at new Listr (/snapshot/chectl/node_modules/listr/index.js:50:8)
    at List.<anonymous> (/snapshot/chectl/lib/commands/workspace/list.js:0:0)
    at Generator.next (<anonymous>)
    at __awaiter (/snapshot/chectl/node_modules/tslib/tslib.js:110:75)
    at new Promise (<anonymous>)
    at Object.__awaiter (/snapshot/chectl/node_modules/tslib/tslib.js:106:16)
    at List.run (/snapshot/chectl/lib/commands/workspace/list.js:0:0)
    at List._run (/snapshot/chectl/node_modules/@oclif/command/lib/command.js:44:31)

Screenshots

Installation method

• chectl

Environment

OS: macOS Mojave 10.14.6
Browser: Google Chrome Version 75.0.3770.142 (Official Build) (64-bit)
Running Eclipse Che locally

[sleshchenko]

@tony-bony I see that you specify --memory=10240 and it should be more than enough for Che and 1 workspace. But error clearly says that K8s is not able to run more pods.
Do you have something else deployed on your Che? How many memory do you have on a host?

[amisevsk]

Additionally, note that minikube will only pick up --memory=10240 on its first launch; if you previously launched without that option you would need to minikube delete and restart.

Could you provide the output of kubectl describe node? Che is just forwarding an event from kube in that failure.

[tony-bony]

I did first --memory=4096 then minikube stop then start with --memory=10240 so I should try to delete first.

kubectl describe node gives:

Name:               minikube
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=minikube
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 29 Jul 2019 23:16:35 +0200
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 30 Jul 2019 15:33:22 +0200   Mon, 29 Jul 2019 23:16:30 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 30 Jul 2019 15:33:22 +0200   Mon, 29 Jul 2019 23:16:30 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 30 Jul 2019 15:33:22 +0200   Mon, 29 Jul 2019 23:16:30 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Tue, 30 Jul 2019 15:33:22 +0200   Mon, 29 Jul 2019 23:16:30 +0200   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  10.0.2.15
  Hostname:    minikube
Capacity:
 cpu:                2
 ephemeral-storage:  17784772Ki
 hugepages-2Mi:      0
 memory:             2038624Ki
 pods:               110
Allocatable:
 cpu:                2
 ephemeral-storage:  16390445849
 hugepages-2Mi:      0
 memory:             1936224Ki
 pods:               110
System Info:
 Machine ID:                 b4270150662b440d979a07322785f94d
 System UUID:                A6BD2323-B65C-47C5-99FA-E7CBC8A12353
 Boot ID:                    2e54af68-e03b-49c2-b228-cadc4c32836e
 Kernel Version:             4.15.0
 OS Image:                   Buildroot 2018.05.3
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.9.6
 Kubelet Version:            v1.15.0
 Kube-Proxy Version:         v1.15.0
Non-terminated Pods:         (16 in total)
  Namespace                  Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                         ------------  ----------  ---------------  -------------  ---
  che                        che-54f4459fb4-jtfms                         0 (0%)        0 (0%)      512Mi (27%)      1000Mi (52%)   179m
  che                        devfile-registry-549b9bbd5b-jg7h6            0 (0%)        0 (0%)      16Mi (0%)        32Mi (1%)      15h
  che                        plugin-registry-75478b67dd-bnvf2             0 (0%)        0 (0%)      16Mi (0%)        32Mi (1%)      15h
  kube-system                coredns-5c98db65d4-7l6mm                     100m (5%)     0 (0%)      70Mi (3%)        170Mi (8%)     16h
  kube-system                coredns-5c98db65d4-lspgg                     100m (5%)     0 (0%)      70Mi (3%)        170Mi (8%)     16h
  kube-system                default-http-backend-59f7ff8999-8xnsv        20m (1%)      20m (1%)    30Mi (1%)        30Mi (1%)      15h
  kube-system                etcd-minikube                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         16h
  kube-system                kube-addon-manager-minikube                  5m (0%)       0 (0%)      50Mi (2%)        0 (0%)         16h
  kube-system                kube-apiserver-minikube                      250m (12%)    0 (0%)      0 (0%)           0 (0%)         16h
  kube-system                kube-controller-manager-minikube             200m (10%)    0 (0%)      0 (0%)           0 (0%)         14h
  kube-system                kube-proxy-f6wfp                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         16h
  kube-system                kube-scheduler-minikube                      100m (5%)     0 (0%)      0 (0%)           0 (0%)         16h
  kube-system                kubernetes-dashboard-7b8ddcb5d6-ld4x9        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h7m
  kube-system                nginx-ingress-controller-7b465d9cf8-czgfx    0 (0%)        0 (0%)      0 (0%)           0 (0%)         15h
  kube-system                storage-provisioner                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         16h
  kube-system                tiller-deploy-5f669f7664-f7xwc               0 (0%)        0 (0%)      0 (0%)           0 (0%)         16h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                775m (38%)   20m (1%)
  memory             764Mi (40%)  1434Mi (75%)
  ephemeral-storage  0 (0%)       0 (0%)

[amisevsk]

Yeah, the issue is you need to minikube delete and then minikube start --memory=10240Mi. In the output you can see

Capacity:
 cpu:                2
 ephemeral-storage:  17784772Ki
 hugepages-2Mi:      0
 memory:             2038624Ki   <--- 2048Mi

I've been bitten by this before.

[tony-bony]

Deleting minikube and starting with minikube start --memory=8192 solved the issue.

Thank you all for your quick support!