Keda: [v2] starting container process caused "exec: \"keda\": executable file not found in $PATH": unknown

[mboutet]

Trying to install keda 2.0.0-beta chart on AKS cluster 1.18.6 fails (keda-operator pod does not start)

Expected Behavior

It should work.

Actual Behavior

The keda-operator pod fails to start. Here is the output of kubectl describe pod -n keda keda-operator-559d647557-bt8gt:

Name:                 keda-operator-559d647557-bt8gt
Namespace:            keda
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 aks-default-38241096-vmss000001/10.240.0.5
Start Time:           Fri, 18 Sep 2020 15:29:16 -0400
Labels:               app=keda-operator
                      pod-template-hash=559d647557
Annotations:          <none>
Status:               Running
IP:                   10.244.4.142
IPs:
  IP:           10.244.4.142
Controlled By:  ReplicaSet/keda-operator-559d647557
Containers:
  keda-operator:
    Container ID:  docker://6d3a0653f3d96723c42e1dbafe1e0282db820cc6875e2da73b679372246ea06e
    Image:         docker.io/kedacore/keda:2.0.0-beta
    Image ID:      docker-pullable://kedacore/keda@sha256:94a99db79bf50fd4ba39c5ab4b263d1091b4caf5a098c6b80c593a117a0cedc1
    Port:          <none>
    Host Port:     <none>
    Command:
      keda
    Args:
      --zap-level=info
      --zap-time-encoding=epoch
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"keda\": executable file not found in $PATH": unknown
      Exit Code:    127
      Started:      Fri, 18 Sep 2020 15:30:49 -0400
      Finished:     Fri, 18 Sep 2020 15:30:49 -0400
    Ready:          False
    Restart Count:  4
    Limits:
      cpu:     125m
      memory:  64Mi
    Requests:
      cpu:     100m
      memory:  64Mi
    Environment:
      WATCH_NAMESPACE:
      POD_NAME:         keda-operator-559d647557-bt8gt (v1:metadata.name)
      OPERATOR_NAME:    keda-operator
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from keda-operator-token-2kmnt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  keda-operator-token-2kmnt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  keda-operator-token-2kmnt
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  mode=system
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                                      Message
  ----     ------     ----                 ----                                      -------
  Normal   Scheduled  2m28s                default-scheduler                         Successfully assigned keda/keda-operator-559d647557-bt8gt to aks-default-38241096-vmss000001
  Warning  BackOff    63s (x5 over 116s)   kubelet, aks-default-38241096-vmss000001  Back-off restarting failed container
  Normal   Pulling    55s (x5 over 2m27s)  kubelet, aks-default-38241096-vmss000001  Pulling image "docker.io/kedacore/keda:2.0.0-beta"
  Normal   Pulled     55s (x5 over 2m25s)  kubelet, aks-default-38241096-vmss000001  Successfully pulled image "docker.io/kedacore/keda:2.0.0-beta"
  Normal   Created    55s (x5 over 2m24s)  kubelet, aks-default-38241096-vmss000001  Created container keda-operator
  Warning  Failed     55s (x5 over 2m23s)  kubelet, aks-default-38241096-vmss000001  Error: failed to start container "keda-operator": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"keda\": executable file not found in $PATH": unknown

Steps to Reproduce the Problem

1. Create a fresh AKS cluster v1.18.6
2. Install keda chart 2.0.0-beta
3. The keda-operator pod fails to start.

Specifications

• KEDA Version: 2.0.0-beta
• Platform & Version: : AKS 1.18.6
• Kubernetes Version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T19:10:58Z", GoVersion:"go1.15.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"1994a5495a40a663921c5ecfee7dd9a8c61704fa", GitTreeState:"clean", BuildDate:"2020-07-23T22:06:44Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Scaler(s): None

[mboutet]

Comparing https://github.com/kedacore/keda/blob/v2/config/manager/manager.yaml#L30 and https://github.com/kedacore/charts/blob/master/keda/templates/12-keda-deployment.yaml#L49 shows that the chart is missing the leading front slash. Could that be the problem?

[turbaszek]

I got similar problem. I was building the image locally and using it on local k8s cluster. The most fascinating thing was that I published this image and @zroubalik tried it and there was no problem 😳 What helped me was to prune docker and restart the cluster 🤷🏼‍♂️

[mboutet]

Turns out it's an issue with the helm chart that still has the manifests for the v1 version (even if it was tagged with 2.0.0-beta). I opened a PR in the chart repo and it's linked in this issue.