Kind: node-image build failure due to unavailability of kubeadm build artifact

Created on 27 Mar 2019  ·  19Comments  ·  Source: kubernetes-sigs/kind

Building node-image is getting failed as kubeadm build artifact is not getting copied from the container to host machine.
Note: Observed failure on MacOS but same is getting succeeded from Ubuntu.

Is there any way we can find the root cause of the issue?

Following are the logs from system (MacOS)

➜  kubernetes (master) ✔ make
+++ [0327 17:26:49] Building go targets for darwin/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0327 17:27:00] Building go targets for darwin/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0327 17:27:06] Building go targets for darwin/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0327 17:27:22] Building go targets for darwin/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
+++ [0327 17:27:34] Building go targets for darwin/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0327 17:27:37] Building go targets for darwin/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    cmd/kube-scheduler
    vendor/k8s.io/apiextensions-apiserver
    cluster/gce/gci/mounter
    cmd/kubectl
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/genswaggertypedocs
    cmd/linkcheck
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
➜  kubernetes (master) ✔ kind build base-image
Sending build context to Docker daemon  2.187MB
Step 1/17 : ARG BASE_IMAGE="ubuntu:18.04"
Step 2/17 : FROM ${BASE_IMAGE}
 ---> 94e814e2efa8
Step 3/17 : ARG DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> b3f4877772f1
Step 4/17 : COPY clean-install /usr/local/bin/clean-install
 ---> Using cache
 ---> 9a93ad223a2e
Step 5/17 : RUN chmod +x /usr/local/bin/clean-install
 ---> Using cache
 ---> 20ce904bc10d
Step 6/17 : RUN clean-install       apt-transport-https ca-certificates curl software-properties-common gnupg2 lsb-release       systemd systemd-sysv libsystemd0       conntrack iptables iproute2 ethtool socat util-linux mount ebtables udev kmod aufs-tools       bash rsync     && find /lib/systemd/system/sysinit.target.wants/ -name "systemd-tmpfiles-setup.service" -delete     && rm -f /lib/systemd/system/multi-user.target.wants/*     && rm -f /etc/systemd/system/*.wants/*     && rm -f /lib/systemd/system/local-fs.target.wants/*     && rm -f /lib/systemd/system/sockets.target.wants/*udev*     && rm -f /lib/systemd/system/sockets.target.wants/*initctl*     && rm -f /lib/systemd/system/basic.target.wants/*     && echo "ReadKMsg=no" >> /etc/systemd/journald.conf
 ---> Using cache
 ---> f9367a639793
Step 7/17 : ARG DOCKER_VERSION="18.06.*"
 ---> Using cache
 ---> 29c83033da6f
Step 8/17 : ARG APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE="false"
 ---> Using cache
 ---> 72b71513bb11
Step 9/17 : RUN curl -fsSL "https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg" | apt-key add -     && apt-key fingerprint 0EBFCD88     && add-apt-repository         "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"     && clean-install "docker-ce=${DOCKER_VERSION}"
 ---> Using cache
 ---> 6b0d3f3bd92f
Step 10/17 : ARG CNI_VERSION="0.7.5"
 ---> Using cache
 ---> 81cb7c51b166
Step 11/17 : ARG CNI_BASE_URL="https://storage.googleapis.com/kubernetes-release/network-plugins/"
 ---> Using cache
 ---> 81dc81220892
Step 12/17 : RUN export ARCH=$(dpkg --print-architecture)     && export CNI_TARBALL="cni-plugins-${ARCH}-v${CNI_VERSION}.tgz"     && export CNI_URL="${CNI_BASE_URL}${CNI_TARBALL}"     && curl -sSL --retry 5 --output /tmp/cni.tgz "${CNI_URL}"     && sha256sum /tmp/cni.tgz     && mkdir -p /opt/cni/bin     && tar -C /opt/cni/bin -xzf /tmp/cni.tgz     && rm -rf /tmp/cni.tgz
 ---> Using cache
 ---> ce0bd26b0970
Step 13/17 : ENV container docker
 ---> Using cache
 ---> f554f30e46a6
Step 14/17 : STOPSIGNAL SIGRTMIN+3
 ---> Using cache
 ---> 3e636878635f
Step 15/17 : COPY [ "entrypoint/entrypoint", "/usr/local/bin/" ]
 ---> 95f7354110d7
Step 16/17 : ENTRYPOINT [ "/usr/local/bin/entrypoint", "/sbin/init" ]
 ---> Running in 764692a641b4
Removing intermediate container 764692a641b4
 ---> e990bcdaeb3f
Step 17/17 : VOLUME [ "/var/lib/docker" ]
 ---> Running in 307650f42fcc
Removing intermediate container 307650f42fcc
 ---> ddd11bdc3fef
Successfully built ddd11bdc3fef
Successfully tagged kindest/base:latest
➜  kubernetes (master) ✔ kind build node-image --base-image kindest/base:latest --loglevel debug
INFO[17:34:25] Starting to build Kubernetes
DEBU[17:34:25] Running: build/run.sh [build/run.sh make all WHAT=cmd/kubeadm cmd/kubectl cmd/kubelet KUBE_BUILD_PLATFORMS=darwin/amd64 KUBE_VERBOSE=0]
+++ [0327 17:34:26] Verifying Prerequisites....
+++ [0327 17:34:26] Using Docker for MacOS
+++ [0327 17:34:27] Building Docker image kube-build:build-a11a133c2a-5-v1.12.1-2
+++ [0327 17:34:31] Keeping container kind-control-plane
+++ [0327 17:34:31] Keeping container kube-build-data-67ab253b5d-5-v1.12.1-2
+++ [0327 17:34:31] Keeping container kind-control-plane
+++ [0327 17:34:31] Keeping container kube-build-data-67ab253b5d-5-v1.12.1-2
+++ [0327 17:34:31] Keeping container kind-control-plane
+++ [0327 17:34:31] Keeping container kube-build-data-67ab253b5d-5-v1.12.1-2
+++ [0327 17:34:31] Keeping image kube-build:build-a11a133c2a-5-v1.12.1-2
+++ [0327 17:34:31] Keeping image kube-build:build-67ab253b5d-5-v1.12.1-2
+++ [0327 17:34:31] Creating data container kube-build-data-a11a133c2a-5-v1.12.1-2
+++ [0327 17:34:42] Syncing sources to container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:34:43] Stopping any currently running rsyncd container
+++ [0327 17:34:43] Starting rsyncd container
+++ [0327 17:34:44] Running rsync
+++ [0327 17:35:25] Stopping any currently running rsyncd container
+++ [0327 17:35:25] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0327 17:35:25] Running build command...
+++ [0327 17:35:35] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0327 17:35:46] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0327 17:35:51] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0327 17:36:04] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
+++ [0327 17:36:14] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0327 17:36:15] Building go targets for darwin/amd64:
    cmd/kubeadm
    cmd/kubectl
    cmd/kubelet
+++ [0327 17:38:28] Syncing out of container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:38:28] Stopping any currently running rsyncd container
+++ [0327 17:38:28] Starting rsyncd container
+++ [0327 17:38:29] Running rsync
+++ [0327 17:38:34] Stopping any currently running rsyncd container
DEBU[17:38:35] Running: /usr/bin/make [make quick-release-images KUBE_BUILD_HYPERKUBE=n KUBE_BUILD_CONFORMANCE=n]
+++ [0327 17:38:35] Verifying Prerequisites....
+++ [0327 17:38:36] Using Docker for MacOS
+++ [0327 17:38:37] Building Docker image kube-build:build-a11a133c2a-5-v1.12.1-2
+++ [0327 17:38:41] Syncing sources to container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:38:44] Running build command...
+++ [0327 17:39:18] Building go targets for linux/amd64:
    cmd/cloud-controller-manager
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kube-scheduler
    cmd/kube-proxy
+++ [0327 17:41:42] Syncing out of container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:41:53] Building images: linux-amd64
+++ [0327 17:41:55] Starting docker build for image: cloud-controller-manager-amd64
+++ [0327 17:41:55] Starting docker build for image: kube-apiserver-amd64
+++ [0327 17:41:55] Starting docker build for image: kube-controller-manager-amd64
+++ [0327 17:41:55] Starting docker build for image: kube-scheduler-amd64
+++ [0327 17:41:55] Starting docker build for image: kube-proxy-amd64
+++ [0327 17:42:07] Deleting docker image k8s.gcr.io/kube-scheduler:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:42:09] Deleting docker image k8s.gcr.io/kube-proxy:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:42:11] Deleting docker image k8s.gcr.io/cloud-controller-manager:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:42:11] Deleting docker image k8s.gcr.io/kube-controller-manager:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:42:14] Deleting docker image k8s.gcr.io/kube-apiserver:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:42:14] Docker builds done
DEBU[17:42:14] Running: hack/print-workspace-status.sh [hack/print-workspace-status.sh]
INFO[17:42:15] Finished building Kubernetes
INFO[17:42:15] Building node image in: /private/var/folders/g2/s4cxr85551v5rlxr4rxw4f284c4257/T/kind-node-image259487533
Error: error building node image: failed to copy build artifact: lstat /Users/mayank.gaikwad/work/go-workspace/src/k8s.io/kubernetes/_output/dockerized/bin/linux/amd64/kubelet: no such file or directory
➜  kubernetes (master) ✔ 
➜  kubernetes (master) ✔ kind build node-image --base-image kindest/base:latest --loglevel debug --help
build the node image which contains kubernetes build artifacts and other kind requirements

Usage:
  kind build node-image [flags]

Flags:
      --base-image string   name:tag of the base image to use for the build (default "kindest/base:v20190320-962dc1b")
  -h, --help                help for node-image
      --image string        name:tag of the resulting image to be built (default "kindest/node:latest")
      --kube-root string    Path to the Kubernetes source directory (if empty, the path is autodetected)
      --type string         build type, one of [bazel, docker, apt] (default "docker")

Global Flags:
      --loglevel string   logrus log level [panic, fatal, error, warning, info, debug] (default "warning")
➜  kubernetes (master) ✔ kind build node-image --base-image kindest/base:latest --kube-root .
+++ [0327 17:47:28] Verifying Prerequisites....
+++ [0327 17:47:28] Using Docker for MacOS
+++ [0327 17:47:29] Building Docker image kube-build:build-a11a133c2a-5-v1.12.1-2
+++ [0327 17:47:33] Keeping container kube-build-data-a11a133c2a-5-v1.12.1-2
+++ [0327 17:47:33] Keeping container kind-control-plane
+++ [0327 17:47:33] Keeping container kube-build-data-67ab253b5d-5-v1.12.1-2
+++ [0327 17:47:33] Keeping container kube-build-data-a11a133c2a-5-v1.12.1-2
+++ [0327 17:47:33] Keeping container kind-control-plane
+++ [0327 17:47:33] Keeping container kube-build-data-67ab253b5d-5-v1.12.1-2
+++ [0327 17:47:33] Keeping container kube-build-data-a11a133c2a-5-v1.12.1-2
+++ [0327 17:47:33] Keeping container kind-control-plane
+++ [0327 17:47:33] Keeping container kube-build-data-67ab253b5d-5-v1.12.1-2
+++ [0327 17:47:33] Keeping image kube-build:build-a11a133c2a-5-v1.12.1-2
+++ [0327 17:47:33] Keeping image kube-build:build-67ab253b5d-5-v1.12.1-2
+++ [0327 17:47:34] Syncing sources to container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:47:34] Stopping any currently running rsyncd container
+++ [0327 17:47:34] Starting rsyncd container
+++ [0327 17:47:35] Running rsync
+++ [0327 17:47:35] Stopping any currently running rsyncd container
+++ [0327 17:47:36] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0327 17:47:36] Running build command...
+++ [0327 17:48:11] Building go targets for darwin/amd64:
    cmd/kubeadm
    cmd/kubectl
    cmd/kubelet
+++ [0327 17:48:29] Syncing out of container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:48:30] Stopping any currently running rsyncd container
+++ [0327 17:48:30] Starting rsyncd container
+++ [0327 17:48:31] Running rsync
+++ [0327 17:48:33] Stopping any currently running rsyncd container
+++ [0327 17:48:34] Verifying Prerequisites....
+++ [0327 17:48:34] Using Docker for MacOS
+++ [0327 17:48:36] Building Docker image kube-build:build-a11a133c2a-5-v1.12.1-2
+++ [0327 17:48:40] Syncing sources to container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:48:42] Running build command...
+++ [0327 17:49:15] Building go targets for linux/amd64:
    cmd/cloud-controller-manager
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kube-scheduler
    cmd/kube-proxy
+++ [0327 17:49:39] Syncing out of container
Object "-Version" is unknown, try "ip help".
+++ [0327 17:49:45] Building images: linux-amd64
+++ [0327 17:49:48] Starting docker build for image: cloud-controller-manager-amd64
+++ [0327 17:49:48] Starting docker build for image: kube-apiserver-amd64
+++ [0327 17:49:48] Starting docker build for image: kube-controller-manager-amd64
+++ [0327 17:49:48] Starting docker build for image: kube-scheduler-amd64
+++ [0327 17:49:48] Starting docker build for image: kube-proxy-amd64
+++ [0327 17:49:57] Deleting docker image k8s.gcr.io/kube-scheduler:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:49:57] Deleting docker image k8s.gcr.io/kube-proxy:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:50:00] Deleting docker image k8s.gcr.io/cloud-controller-manager:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:50:01] Deleting docker image k8s.gcr.io/kube-controller-manager:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:50:03] Deleting docker image k8s.gcr.io/kube-apiserver:v1.15.0-alpha.0.1564_6c22cff31747c1
+++ [0327 17:50:03] Docker builds done
Error: error building node image: failed to copy build artifact: lstat _output/dockerized/bin/linux/amd64/kubeadm: no such file or directory
➜  kubernetes (master) ✔

note:
-- make is successfully building all necessary binaries.
-- kind build node-image is failing with an Error: error building node image: failed to copy build artifact: .....

kinsupport
Source
picture mgdevstack

Most helpful comment

this _should_ be fixed by #413, will leave this open until you can confirm

picture BenTheElder on 27 Mar 2019
🎉2

All 19 comments

Docker Version: 2.0.0.3 (31259) Stable
MacOS: 10.13.6 (17G3025)
K8S src: master
kind version: 0.3.0-alpha

mgdevstack picture mgdevstack on 27 Mar 2019

https://github.com/kubernetes/kubernetes/issues/75679

https://github.com/kubernetes/kubernetes/pull/75670

BenTheElder picture BenTheElder on 27 Mar 2019

Thanks for pointing @BenTheElder. Seems issue due to k/k.
I'll keep track and would close this if building artifacts are fixed.

mgdevstack picture mgdevstack on 27 Mar 2019

that issue should be fixed with the latest k/k changes, does updating work for you?

BenTheElder picture BenTheElder on 27 Mar 2019

No, even after having the latest updates from kubernetes/kubernetes#75679, It's not working. (same issue of artifact)

Basically Kind is a super important Conformant platform available which allows having multi-node cluster. ASAP this problem is resolved, I'll be happy to extend work with this cluster.

mgdevstack picture mgdevstack on 27 Mar 2019

+++ [0327 17:49:39] Syncing out of container
Object "-Version" is unknown, try "ip help".

that looks suspect

BenTheElder picture BenTheElder on 27 Mar 2019

so that particular output seems to be coming from https://github.com/kubernetes/kubernetes/blob/a7966022d78bce63fa8212ea2519c99f49c2570e/build/common.sh#L298-L300

on my mac which ip returns no result, what is ip on your machine??

BenTheElder picture BenTheElder on 27 Mar 2019

From log, its obvious that few binaries are getting build into darwin and few into linux but kubeadm is expected to be present inside dockerized/bin/linux/amd64/kubeadm leading to error

Error: error building node image: failed to copy build artifact: lstat _output/dockerized/bin/linux/amd64/kubeadm: no such file or directory

Generated binaries at my end are -
/go-workspace/src/k8s.io/kubernetes/_output/dockerized

├── bin
│   ├── darwin
│   │   └── amd64
│   │       ├── kubeadm
│   │       ├── kubectl
│   │       └── kubelet
│   └── linux
│       └── amd64
│           ├── cloud-controller-manager
│           ├── conversion-gen
│           ├── deepcopy-gen
│           ├── defaulter-gen
│           ├── go-bindata
│           ├── go2make
│           ├── kube-apiserver
│           ├── kube-controller-manager
│           ├── kube-proxy
│           ├── kube-scheduler
│           └── openapi-gen
└── go
mgdevstack picture mgdevstack on 27 Mar 2019

on my mac which ip returns no result, what is ip on your machine??
For me, its /usr/local/bin/ip

mgdevstack picture mgdevstack on 27 Mar 2019

so that particular output seems to be coming from https://github.com/kubernetes/kubernetes/blob/a7966022d78bce63fa8212ea2519c99f49c2570e/build/common.sh#L298-L300

Seems build container has no ip file available. This image kube-build:build-a11a133c2a-5-v1.12.1-2 can be updated to include ip file by appropriate package.

mgdevstack picture mgdevstack on 27 Mar 2019

ip would be on the host, it's trying to figure out the rsync bit.

BenTheElder picture BenTheElder on 27 Mar 2019

I think I found the bug 🤞

BenTheElder picture BenTheElder on 27 Mar 2019
🎉1

I'm also trying to understand from hack/lib/golang.sh, Which line is deciding to generate binaries for which platform and at the end how these binaries would be picked up from their corresponding path.

mgdevstack picture mgdevstack on 27 Mar 2019

this is possibly fixed now https://github.com/kubernetes-sigs/kind/pull/413, about to verify

BenTheElder picture BenTheElder on 27 Mar 2019

this _should_ be fixed by #413, will leave this open until you can confirm

BenTheElder picture BenTheElder on 27 Mar 2019
🎉2

Many thanks for such a quick troubleshoot and issue fix. It's working. We can close this.

mgdevstack picture mgdevstack on 28 Mar 2019

Thanks all. :+1:

/close

tao12345666333 picture tao12345666333 on 28 Mar 2019

@tao12345666333: Closing this issue.

In response to this:

Thanks all. :+1:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot picture k8s-ci-robot on 28 Mar 2019

awesome! I'm cutting a patch-release with the fix :sweat_smile:

BenTheElder picture BenTheElder on 28 Mar 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

cig0 picture cig0  ·  4Comments
mithunvikram picture mithunvikram  ·  3Comments
fgimenez picture fgimenez  ·  4Comments
raphaelauv picture raphaelauv  ·  3Comments
rajalokan picture rajalokan  ·  3Comments