Minikube: addon enable fails after update: kubectl: cannot execute binary file

[ImShakthi]

The exact command to reproduce the issue: minikube addons enable ingress

The full output of the command that failed:
💣 enable failed: running callbacks: [addon apply: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.17.0/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml: Process exited with status 126
stdout:

stderr:
/var/lib/minikube/binaries/v1.17.0/kubectl: /var/lib/minikube/binaries/v1.17.0/kubectl: cannot execute binary file
]

The output of the minikube logs command:

The operating system version: macOSv10.15.2

[tstromberg]

Hey @ImShakthi

Thank you for opening this bug. I have a theory that this bug is upgrade related, but am still a bit perplexed. Do you mind collecting some information for me first. I would like the output of:

• minikube version
• minikube ssh "ls -laR /var/lib/minikube/binaries"
• minikube ssh "uname -a"

I'm pretty sure the bug here is that we rewrote the way addons are enabled in minikube v1.17, which causes a regression if a user upgrades minikube, but leaves a running cluster from a previous release.

The workaround I suggest is to run minikube delete to recreate the cluster. If that isn' possible, I believe that re-running minikube start will install the missing binary, but leave your pods up and running.

Please let me know what the result is eiher way.

The long-term fix I propose is that "addons enable" should check if kubectl is installed, and if not, install it.

[tstromberg]

If someone wants to fix this issue for other future upgraders, one easy solution would be to add something here that checks if kubectl is installed in the cluster:

https://github.com/kubernetes/minikube/blob/239be7ed941c73a9b5b0b4d1833f23c19b29c0e2/pkg/addons/addons.go#L211

A simple file existence check like test -x /path/to/kubectl should do. If the kubectl binary is not executable, you can call this function to install the missing binaries:

https://github.com/kubernetes/minikube/blob/239be7ed941c73a9b5b0b4d1833f23c19b29c0e2/pkg/minikube/bootstrapper/bsutil/binaries.go#L35

You will want to change the TransferBinaries function to directly take a Kubernetes version string instead of needlessly requiring config.KubernetesConfig.

[ImShakthi]

@tstromberg Thank you for responding.

I'm able to enable addons after minikube delete and minikube start.

Here the output for the command you request for,

1. minikube version

minikube version: v1.7.1
commit: 7de0325eedac0fbe3aabacfcc43a63eb5d029fda

1. minikube ssh "ls -laR /var/lib/minikube/binaries"

/var/lib/minikube/binaries:
total 12
drwxr-xr-x 3 root root 4096 Feb  9 04:15 .
drwxr-xr-x 6 root root 4096 Feb  9 04:15 ..
drwxr-xr-x 2 root root 4096 Feb  9 04:15 v1.17.2

/var/lib/minikube/binaries/v1.17.2:
total 189860
drwxr-xr-x 2 root root      4096 Feb  9 04:15 .
drwxr-xr-x 3 root root      4096 Feb  9 04:15 ..
-rwxr-xr-x 1 root root  39342080 Feb  9 04:15 kubeadm
-rwxr-xr-x 1 root root  43491328 Feb  9 04:15 kubectl
-rwxr-xr-x 1 root root 111568408 Feb  9 04:15 kubelet

1. minikube ssh "uname -a"

Linux minikube 4.19.88 #1 SMP Tue Feb 4 22:25:03 PST 2020 x86_64 GNU/Linux

[afbjorklund]

Usually "cannot execute binary file" means there is something wrong with the program, like os or arch.

In this case, the host kubectl binary is for darwin - where as the minikube binaries are for linux.

Are you able to confirm, by checking the file contents of the /var/lib/minikube/binaries/v1.17.2/kubectl

It might be a regression, if we are trying to store both kubectl binaries in the same cache dir... ?

[ImShakthi]

@afbjorklund I did ssh into my minikube and was able to see var/lib/minikube/binaries/v1.17.2/kubectl file. I'm not pretty sure about the reason for the issue, yet curious to know the root cause and fix it.

[afbjorklund]

@ImShakthi : I was wondering if it is a darwin binary or a linux binary, not if it exists (it did)

Can you run something like file on it ? Should say; "ELF 64-bit LSB executable, x86-64"

[afbjorklund]

Get a slightly different error when I try to reproduce it, though.

$ curl -LO "https://storage.googleapis.com/kubernetes-release/release/v1.7.2/bin/darwin/amd64/kubectl"
$ file kubectl 
kubectl: Mach-O 64-bit x86_64 executable
$ uname
Linux
$ chmod +x kubectl 
$ ./kubectl 
bash: ./kubectl: cannot execute binary file: Exec format error

So there could be another reason for the error you are seeing.

[afbjorklund]

My bad, "Exec format error" is the error code (ENOEXEC) - not the error message.

It is that problem, was able to duplicate by copying the Mac binary into the cache...

⚠️  Enabling 'storage-provisioner' returned an error: running callbacks: [addon apply: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.17.2/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 126
stdout:

stderr:
/var/lib/minikube/binaries/v1.17.2/kubectl: /var/lib/minikube/binaries/v1.17.2/kubectl: cannot execute binary file
]

I think Windows users got lucky, since there it is called kubectl.exe instead.

@priyawadhwa : why are we copying and running kubectl in the cluster ?

[tstromberg]

Yeah, the bug here is that we write the macOS binary of kubectl and the Linux version to the same path. Let's make sure to get this fixed ASAP. Help wanted!

FYI - This only affects macOS users who have run minikube kubectl

[afbjorklund]

Suggestion is to move the host binaries into a ”client” subdirectory

[priyawadhwa]

@afbjorklund we have kubectl in the cluster to enable and disable addons since the addon manager was removed.

I can take a look at this one!