Minikube: Kubelet and Kubeadm v1.10.0 instead of v1.12.0

[jroucheton]

BUG REPORT

Minikube version (use minikube version):
v0.30.0

OS (e.g. from /etc/os-release):
Windows

VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
Virtualbox

ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
v0.30.0

What happened:
minikube start downloads by default kubelet and kubeadm v1.10.0

What you expected to happen:
minikube start should download be default the latest version for kubelet and kubeadm which is v1.12.0

[sofianinho]

I am in the same situation. I found this option to workaround the default parameters:

sudo -E minikube start --kubernetes-version v1.12.2

I'm waiting the end of the download to see how it goes...

So this is the result (not good):

$ sudo -E minikube start --kubernetes-version v1.12.2 --vm-driver=none

Starting local Kubernetes v1.12.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.12.2
Downloading kubelet v1.12.2
Finished Downloading kubeadm v1.12.2
Finished Downloading kubelet v1.12.2
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E1029 12:57:16.796277   30534 start.go:297] Error starting cluster:  kubeadm init error 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns
 running command: : running command: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns

 output: [init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
    [WARNING Swap]: running with swap on is not supported. Please disable swap
    [WARNING Hostname]: hostname "minikube" could not be reached
    [WARNING Hostname]: hostname "minikube" lookup minikube on 8.8.8.8:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver certificate and key.
Error creating PKI assets: failed to write certificate "apiserver-kubelet-client": certificate apiserver-kubelet-client is not signed by corresponding CA
: running command: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns

.: exit status 1

[sofianinho]

Ok that was nasty, but I eventually overcame my issue, hopefully this experience will serve someone else.
The problem was that the generated certificates from one install to the other were reused. Each time I tried to reinstall minikube properly (because something went wrong), this was my (brute-ish) cleanup procedure:

# Cleanup
sudo rm -rf ~/.minikube/ ~/.kube/ /data/minikube /etc/kubernetes/ /var/lib/kube* $(which kubelet) $(which kubeadm) $(which kubectl)
#Reinstall from the README instructions on this repo

Later on I figured there was a kubeadm reset command I could try, without luck (same output as above). Up until I noticed inside the /var/lib/kubeadm.yamlfile that there was a reference to another folder I did not notice. Here is the content of my kubeadm.yaml

apiEndpoint:
  advertiseAddress: 192.168.1.17
  bindPort: 8443
apiVersion: kubeadm.k8s.io/v1alpha3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: minikube
  taints: []
---
apiServerExtraArgs:
  enable-admission-plugins: "Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"

apiVersion: kubeadm.k8s.io/v1alpha3
certificatesDir: /var/lib/minikube/certs/
clusterName: kubernetes
controlPlaneEndpoint: localhost:8443
etcd:
  local:
    dataDir: /data/minikube
kind: ClusterConfiguration
kubernetesVersion: v1.12.2
networking:
  dnsDomain: cluster.local
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12

The line I'm talking about is certificatesDir: /var/lib/minikube/certs/. So naturally, in continuation of my brute-ish approach I deleted that /var/lib/minikube folder as well in my cleanup command (rm -rf /var/lib/minikube). Everything was fine after that

sudo minikube start --vm-driver=none --kubernetes-version v1.12.2
Starting local Kubernetes v1.12.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
===================
WARNING: IT IS RECOMMENDED NOT TO RUN THE NONE DRIVER ON PERSONAL WORKSTATIONS
    The 'none' driver will run an insecure kubernetes apiserver as root that may leave the host vulnerable to CSRF attacks

When using the none driver, the kubectl config and credentials generated will be root owned and will appear in the root home directory.
You will need to move the files to the appropriate location and then set the correct permissions.  An example of this is below:

    sudo mv /root/.kube $HOME/.kube # this will write over any previous configuration
    sudo chown -R $USER $HOME/.kube
    sudo chgrp -R $USER $HOME/.kube

    sudo mv /root/.minikube $HOME/.minikube # this will write over any previous configuration
    sudo chown -R $USER $HOME/.minikube
    sudo chgrp -R $USER $HOME/.minikube

This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
Loading cached images from config file.

Some comments about this journey:

• I still do not know why minikube was defaulting to 1.10 (my version was the 0.30). You have to specify the wanted version with the flag (that's fine by me!)
• The very awkward install and running procedure especially with the vm-driver=none, where you have to be sudoer and files would spread to /var/lib. Even by setting the none user env variable, it still asks for my root credentials after a re-run. I understand that has to do with the installation in /usr/local/bin among other things. Could not this be avoided through the use of docker as normal user + installation in local folders (.local/share and .config, for instance) instead of /var/lib ?

[tstromberg]

I can totally understand the confusion here, but the behavior is expected in this case. This is somewhat related to #3249.

We'll be updating the default shortly to 1.12 which will also help with some of the confusion: #3180