Kops: protokube.service: Executable /usr/bin/docker missing, skipping: No such file or directory

[zetaab]

1. What kops version are you running? The command kops version, will display
this information.

master

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.19.1

3. What cloud provider are you using?

openstack

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create cluster \
  --cloud openstack \
  --name rolling2.k8s.local \
  --state ${KOPS_STATE_STORE} \
  --zones zone-1,zone-2,zone-3 \
  --network-cidr 10.1.0.0/16 \
  --image debian-10-010920-devops \
  --master-count=3 \
  --node-count=2 \
  --node-size m1.medium \
  --master-size m1.medium \
  --etcd-storage-type solidfire \
  --topology private \
  --bastion \
  --networking calico \
  --api-loadbalancer-type public \
  --os-kubelet-ignore-az=true \
  --os-ext-net int-net

5. What happened after the commands executed?

everything went fine in cli. However, when going to master and check logs - something wrong with the docker.

Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[4425]: protokube.service: Executable /usr/bin/docker missing, skipping: No such file or directory
Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[4426]: protokube.service: Executable /usr/bin/docker missing, skipping: No such file or directory
Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[1]: Started Kubernetes Protokube Service.
Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[4427]: protokube.service: Failed to execute command: No such file or directory
Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[4427]: protokube.service: Failed at step EXEC spawning /usr/bin/docker: No such file or directory
Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[1]: protokube.service: Main process exited, code=exited, status=203/EXEC
Sep 30 11:18:08 master-zone-3-1-rolling2-k8s-local systemd[1]: protokube.service: Failed with result 'exit-code'.

root@master-zone-3-1-rolling2-k8s-local:/var/log# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: https://docs.docker.com
root@master-zone-3-1-rolling2-k8s-local:/var/log# systemctl start docker
Failed to start docker.service: Unit docker.socket not found.

/kind bug

[hakman]

@zetaab please ping me if you need any help with this.

[zetaab]

@hakman yes I need, the weird thing here is that last version 1.19 alpha4 works fine, but master does not work. Something is changed somewhere

perhaps https://github.com/kubernetes/kops/pull/9969 is the problem

[hakman]

One idea would be to build it all, upload to S3 and use KOPS_BASE_URL to deploy.
If #9969 is the problem you could just set Docker version to 19.03.11 in config and should work.

[DerrickMartinez]

Resolved. But need to bump rules_docker to 0.14.4 https://github.com/kubernetes/kops/pull/10007

[zetaab]

seems that nowadays we need build everything to get it working correctly

[codablock]

I'm still having issues with docker not being installed, even though I'm on the latest master with #10007. It worked fine with alpha-4.

Is there anything additional I have to do? Maybe update the VM images?

[hakman]

@codablock you should be able to test using these steps:
https://kops.sigs.k8s.io/development/adding_a_feature/#testing

Also, this may be related to the newer Docker version that was added so, just setting the Docker version to 19.03.11 may get it to work without rebuilding anything.

[h3poteto]

I encountered same error in master branch. I can resolve to specify the Docker version:

spec:
  containerRuntime: docker
  docker:
    version: 19.03.11

Thank you!