Calico: error=fork/exec /usr/bin/kubectl: no such file or directory

[ElvisTan]

error=fork/exec /usr/bin/kubectl: no such file or directory，when use Migrating a cluster from flannel to Calico。calico version is v3.11.1

[tmjd]

When are you seeing that error? Are you getting logs from something that is reporting that or is that when you're executing commands from the docs?
Please provide more context. Thanks

[ElvisTan]

when i Migrating a cluster from flannel to Calico
https://docs.projectcalico.org/v3.11/getting-started/kubernetes/installation/migration-from-flannel

kubectl apply -f https://docs.projectcalico.org/v3.11/manifests/flannel-migration/calico.yaml

configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

kubectl apply -f https://docs.projectcalico.org/v3.11/manifests/flannel-migration/migration-job.yaml

configmap/flannel-migration-config created
clusterrole.rbac.authorization.k8s.io/flannel-migration-controller created
clusterrolebinding.rbac.authorization.k8s.io/flannel-migration-controller created
job.batch/flannel-migration created
serviceaccount/flannel-migration-controller created

kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE
calico-kube-controllers-648f4868b8-nnw2m 1/1 Running 0 79s
coredns-5644d7b6d9-2jh7z 1/1 Running 1 3d21h
coredns-5644d7b6d9-c2fcq 1/1 Running 1 3d21h
etcd-k8master 1/1 Running 1 3d21h
flannel-migration-cv9pm 0/1 Error 3 49s
kube-apiserver-k8master 1/1 Running 1 3d21h
kube-controller-manager-k8master 1/1 Running 1 3d21h
kube-flannel-ds-amd64-98vqx 1/1 Running 2 3d21h
kube-flannel-ds-amd64-rss6z 1/1 Running 1 3d21h
kube-flannel-ds-amd64-wt4k6 1/1 Running 1 3d21h
kube-proxy-2t8mq 1/1 Running 1 3d21h
kube-proxy-85vks 1/1 Running 1 3d21h
kube-proxy-8cc7x 1/1 Running 1 3d21h
kube-scheduler-k8master 1/1 Running 1 3d21h
[root@k8master src]# kubectl logs -n kube-system flannel-migration-cv9pm
2019-12-31 02:57:58.457 [INFO][1] main.go 88: Loaded configuration from environment config=&config.Config{LogLevel:"info", ReconcilerPeriod:"5m", CompactionPeriod:"10m", EnabledControllers:"flannelmigration", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", HealthEnabled:true, SyncNodeLabels:true, DatastoreType:"kubernetes"}
2019-12-31 02:57:58.464 [INFO][1] k8s.go 228: Using Calico IPAM
W1231 02:57:58.464526 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2019-12-31 02:57:58.466 [INFO][1] main.go 109: Ensuring Calico datastore is initialized
2019-12-31 02:57:58.481 [INFO][1] main.go 165: Loaded Flannel configuration from environment flannelConfig=&flannelmigration.Config{FlannelNetwork:"", FlannelDaemonsetName:"kube-flannel-ds-amd64", FlannelMTU:0, FlannelIPMasq:true, FlannelSubnetLen:24, FlannelAnnotationPrefix:"flannel.alpha.coreos.com", FlannelVNI:1, FlannelPort:8472, CalicoDaemonsetName:"calico-node", CniConfigDir:"/etc/cni/net.d", PodNodeName:"k8node2", FlannelSubnetEnv:"", DebugWaitBeforeExit:0, DebugWaitBeforeStart:0}
2019-12-31 02:57:58.481 [INFO][1] main.go 183: Starting status report routine
2019-12-31 02:57:58.481 [INFO][1] main.go 368: Starting controller ControllerType="FlannelMigration"
2019-12-31 02:57:58.481 [INFO][1] migration_controller.go 154: Starting Migration controller
2019-12-31 02:57:58.506 [INFO][1] migration_controller.go 258: Trying to read Flannel env config by executing into daemonet pod.
2019-12-31 02:57:58.517 [ERROR][1] k8s_resources.go 496: Kubectl exec kube-flannel-ds-amd64-rss6z(kube-flannel) error.
---[exec kube-flannel-ds-amd64-rss6z --namespace=kube-system -c=kube-flannel cat run/flannel/subnet.env]---

---

2019-12-31 02:57:58.517 [ERROR][1] migration_controller.go 159: Error checking status, Stopping Migration controller. error=fork/exec /usr/bin/kubectl: no such file or directory
2019-12-31 02:57:58.517 [FATAL][1] migration_controller.go 138: Migration controller stopped.

[tmjd]

@song-jiang does the migration depend on kubectl on each of the nodes? or do you know what might be the issue here?

[song-jiang]

@tmjd The kubectl binary should be built into migration-controller image. https://github.com/projectcalico/kube-controllers/blob/master/docker-images/flannel-migration/Dockerfile.amd64#L32
I will check v3.11.1 image and see if it is there.

[song-jiang]

@ElvisTan Looks like flannel-migration-controller:v3.11.x image is broken. Could you try calico version v3.10.2 instead?

[rafaelvanoni]

@ElvisTan Looks like flannel-migration-controller:v3.11.x image is broken. Could you try calico version v3.10.2 instead?

@song-jiang I just installed v3.11.1 on a cluster and /bin/kubectl was present. Can you post the steps you ran that showed otherwise? Thanks

[mnaser]

I am running into this too, I think the issue is that it's trying to run from /usr/bin/kubectl instead of /bin/kubectl

2020-01-01 23:06:03.994 [ERROR][1] migration_controller.go 159: Error checking status, Stopping Migration controller. error=fork/exec /usr/bin/kubectl: no such file or directory

[mnaser]

In order to verify:

$ docker run --rm --entrypoint /usr/bin/kubectl calico/flannel-migration-controller:v3.10.2 version
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

$ docker run --rm --entrypoint /usr/bin/kubectl calico/flannel-migration-controller:v3.11.0 version
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/usr/bin/kubectl\": stat /usr/bin/kubectl: no such file or directory": unknown.

$ docker run --rm --entrypoint /usr/bin/kubectl calico/flannel-migration-controller:v3.11.1 version
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/usr/bin/kubectl\": stat /usr/bin/kubectl: no such file or directory": unknown.

[ElvisTan]

Using images flannel-migration-controller:v3.10.2 does not have this problem