Minikube: Install Kubernetes v1.19.3 with minikube failed in Github actions

[wjsi]

The error message is invalid checksum: no checksum found in: https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256 but the hash file do exist. It works well when installing Kubernetes v1.19.2.

Steps to reproduce the issue:

OS: ubuntu 16.04

Log url: https://github.com/mars-project/mars/runs/1256818740?check_suite_focus=true

Install script:

#!/bin/bash
set -e
export CHANGE_MINIKUBE_NONE_USER=true

sudo apt-get -q update || true
sudo apt-get install -yq conntrack jq

get_latest_release() {
  curl --silent "https://api.github.com/repos/$1/releases" |
    jq -c '[.[] | select(.prerelease == false)][0].tag_name' |
    sed -E 's/.*"([^"]+)".*/\1/'
}

K8S_VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)
if [[ "$K8S_VERSION" == *"alpha"* ]] || [[ "$K8S_VERSION" == *"beta"* ]] || [[ "$K8S_VERSION" == *"rc"* ]]; then
  K8S_VERSION=$(get_latest_release "kubernetes/kubernetes")
fi

curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/$K8S_VERSION/bin/linux/amd64/kubectl && \
  chmod +x kubectl && sudo mv kubectl /usr/local/bin/

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && \
  chmod +x minikube && sudo mv minikube /usr/local/bin/

# fails here
sudo minikube start --vm-driver=none --kubernetes-version=$K8S_VERSION
sudo chown -R $(id -u):$(id -g) $HOME/.minikube
sudo chown -R $(id -u):$(id -g) $HOME/.kube

minikube update-context

JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}'
until kubectl get nodes -o jsonpath="$JSONPATH" 2>&1 | grep -q "Ready=True"; do
  sleep 1
done

Full output of failed command:

* minikube v1.14.0 on Ubuntu 16.04
* Using the none driver based on user configuration
* Starting control plane node minikube in cluster minikube
* Running on localhost (CPUs=2, Memory=6969MB, Disk=85174MB) ...
* OS release is Ubuntu 16.04.7 LTS
* Preparing Kubernetes v1.19.3 on Docker 19.03.13+azure ...

    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s    > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s    > kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0sX Exiting due to K8S_INSTALL_FAILED: updating control plane: downloading binaries: downloading kubectl: download failed: https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256: getter: &{Ctx:context.Background Src:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256 Dst:/home/runner/.minikube/cache/linux/v1.19.3/kubectl.download Pwd: Mode:2 Detectors:[0x2d333e0 0x2d333e0 0x2d333e0 0x2d333e0 0x2d333e0 0x2d333e0] Decompressors:map[bz2:0x2d333e0 gz:0x2d333e0 tar.bz2:0x2d333e0 tar.gz:0x2d333e0 tar.xz:0x2d333e0 tbz2:0x2d333e0 tgz:0x2d333e0 txz:0x2d333e0 xz:0x2d333e0 zip:0x2d333e0] Getters:map[file:0xc000b77300 http:0xc000b87a20 https:0xc000b87a40] Dir:false ProgressListener:0x2cfd9e0 Options:[0xc8ec80]}: invalid checksum: no checksum found in: https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256
* 
* If the above advice does not help, please let us know: 
  - https://github.com/kubernetes/minikube/issues/new/choose

Error: Process completed with exit code 100.

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

[RA489]

/kind support

[nixpanic]

This is a problem that we have in our CI environment too now. Will watch this space for updates :crossed_fingers:

[AlekSi]

We see the same problem with 1.17.13 and 1.18.10

[nixpanic]

This is not really an issue with minikube itself, but the generated checksum files are different.

FAIL https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256
OK https://storage.googleapis.com/kubernetes-release/release/v1.19.2/bin/linux/amd64/kubectl.sha256

The failing file does not contain a \n. This is probably something that changed in the release process... The https://github.com/hashicorp/go-getter package might be convinced to accept checksum files without trailing \n.

[nixpanic]

The change in hashicorp/go-getter#285 was tested with the following:

1. add a unit-test for the download function
   save download_test.go as ./pkg/minikube/download/download_test.go
2. vendor all pkgs
   $ go mod vendor
3. build a unit-test executable
   $ go test -mod=vendor -c -o e2e-unit.download ./pkg/minikube/download
4. run the unit-test, it will FAIL
   $ ./e2e-unit.download -test.v === RUN TestDownloadOK I1020 13:45:42.185907 2840778 download.go:78] Downloading: https://storage.googleapis.com/kubernetes-release/release/v1.19.2/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.2/bin/linux/amd64/kubectl.sha256 -> /tmp/minikube.test.download > kubectl.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s > kubectl: 41.01 MiB / 41.01 MiB [---------------] 100.00% 22.26 MiB p/s 2s --- PASS: TestDownloadOK (2.60s) === RUN TestDownloadFails I1020 13:45:44.785290 2840778 download.go:78] Downloading: https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256 -> /tmp/minikube.test.download > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s download_test.go:22: failed to download getter: &{Ctx:context.Background Src:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256 Dst:/tmp/minikube.test.download.download Pwd: Mode:2 Detectors:[0x19eb018 0x19eb018 0x19eb018 0x19eb018 0x19eb018 0x19eb018] Decompressors:map[bz2:0x19eb018 gz:0x19eb018 tar.bz2:0x19eb018 tar.gz:0x19eb018 tar.xz:0x19eb018 tbz2:0x19eb018 tgz:0x19eb018 txz:0x19eb018 xz:0x19eb018 zip:0x19eb018] Getters:map[file:0xc0005327c0 http:0xc00046c3e0 https:0xc00046c400] Dir:false ProgressListener:0x19bcdd0 Options:[0xc042b0]}: invalid checksum: no checksum found in: https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256 --- FAIL: TestDownloadFails (0.42s) FAIL
5. make the change to go-getter
   $ vim vendor/github.com/hashicorp/go-getter/checksum.go
6. build a patched unit-test executable
   $ go test -mod=vendor -c -o e2e-unit.download.patched ./pkg/minikube/download
7. run the unit-test again, it should now pass
   $ ./e2e-unit.download.patched -test.v === RUN TestDownloadOK I1020 13:25:42.961301 2838001 download.go:78] Downloading: https://storage.googleapis.com/kubernetes-release/release/v1.19.2/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.2/bin/linux/amd64/kubectl.sha256 -> /tmp/minikube.test.download > kubectl.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s > kubectl: 41.01 MiB / 41.01 MiB [---------------] 100.00% 22.25 MiB p/s 2s --- PASS: TestDownloadOK (2.61s) === RUN TestDownloadFails I1020 13:25:45.567012 2838001 download.go:78] Downloading: https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl.sha256 -> /tmp/minikube.test.download > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s > kubectl: 41.01 MiB / 41.01 MiB [---------------] 100.00% 22.33 MiB p/s 2s --- PASS: TestDownloadFails (2.55s) PASS

[nixpanic]

@afbjorklund it seems that your fork of go-getter is used. Care to cherry-pick hashicorp/go-getter#285 and update go.mod here?

[afbjorklund]

I rebased to v1.4.2 and pushed a new branch as https://github.com/afbjorklund/go-getter/tree/tags-new-checksum (c0da14b)

[afbjorklund]

Seems like the bug was introduced by @saschagrunert here: https://github.com/kubernetes/release/commit/f8a92740d4f4bf112f033ed0318de5ea6c2997fa (it's a problem with krel push)

[afbjorklund]

Seems to be working now (after #9500): ./out/minikube start --kubernetes-version=v1.19.3 --download-only