Minikube: Omission of --kubernetes-version upgrades cluster to the default version

[jar349]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment:
Mac Sierra (10.12.6)

Minikube version (use minikube version):

• OS (e.g. from /etc/os-release): Buildroot 2017.02
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.23.6.iso

What happened:
minikube upgraded itself from 1.8 to 1.9 when it was stopped and started

What you expected to happen:
don't upgrade

How to reproduce it (as minimally and precisely as possible):

minikube start --kubernetes-version v1.8.0
kubectl version
minikube stop
minikube start
kubectl version

[r2d4]

You'll either need to supply the flag everytime or use minikube config set kubernetes-version v1.8.0

[jar349]

Thank you for the workarounds. If I start a kube, is it unreasonable to expect that it will stay the same until deleted and re-created, or otherwise explicitly re-configured? If this isn't a bug, I at least consider current behavior to be non-normal.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[utdrmac]

Agree with @jar349 in that if I launch minikube specifying 1.13.1, then stop, then start it back up, it should NOT auto-upgrade k8s to latest. No other software I have does this behavior; not at least without prompting/asking me first. This is very non-standard behavior and is causing LOTS of problems with out students.

[tace]

I can't believe how stupid feature this is. No sw should ever upgrade itself without explicitly asking it! And user should ask it explicitly, no prompt even is not enough.

[utdrmac]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@utdrmac: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jar349]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@jar349: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[afbjorklund]

Might as well have opened a new ticket for discussing this, but I suppose we can start over here...

Currently minikube only has one single "start" command, that is used for both create and for start.
This means that there is no way to just start an existing VM, without for instance also upgrading it.

It is going to be hard to change this (minikube start), but we could at least make it opt-out ?
In the future we would also like to add a suspend/resume feature, if possible (with libmachine).

As stated above, the workaround for both vm-driver and kubernetes-version is using config.

[afbjorklund]

@tace the other ways of installing kubeadm also does this, unless explictly holding the version back:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

So a similar feature ("hold") for minikube could also be an alternative, perhaps ? But probably opt-in.

[jar349]

Is there an issue discussing the suspend/resume feature? A quick search for suspend turned up nothing that looked like a discussion or future plans. If you link such an issue here, I feel like we could close this thread for good since future readers can follow that issue.

[afbjorklund]

We’ve discussed it in the office hours, and in docker-machine context, but I don’t think there is any issue. The bot tended to auto-close them anyway, as you have discovered.

It is an unrelated feature, either way. You can suspend/resume today, but you have to do it from VirtualBox. Tends to happen automatically when it goes to sleep.

[afbjorklund]

For _this_ issue, I think we could use the Kubernetes version recorded in the profile for re-start. Unless the user has explicitly given a new one?

[jar349]

I think we could use the Kubernetes version recorded in the profile for re-start

Sounds reasonable.

Also, a line of output to the effect of: Reusing version 1.x from previous start. Use --kubernetes-version to change would probably clear up some future confusion.

[tstromberg]

This behavior is now documented: https://minikube.sigs.k8s.io/docs/reference/configuration/kubernetes/

In a nutshell, if you do not specify --kubernetes-version, minikube will use the default version for that minikube release. minikube will not magically update itself passed that version at a later date.

I agree that this behavior could be rather surprising. I think a good case could be made for making it so that the omission of --kubernetes-version defaults to whatever the configuration previously specified, and printing a warning similar to:

• Kubernetes v1.16.0 is now available. If you would like to upgrade, specify:
  --kubernetes-version=v1.16.0

Thoughts?

[jar349]

I think a good case could be made for making it so that the omission of --kubernetes-version defaults to whatever the configuration previously specified, and printing a warning similar to:

Kubernetes v1.16.0 is now available. If you would like to upgrade, specify:
--kubernetes-version=v1.16.0

Yes, I am in favor of this. In my opinion, _using an argument to do something_ is less surprising than _doing something unless an argument is passed_.

[tstromberg]

Sounds good. Let's commit to doing this.

[nanikjava]

/assign @nanikjava

[nanikjava]

Initial investigation

• Without using the --kubernetes-version flag the code will automatically install/upgrade the current VM to use the specific version declared in the code

  minikube start

• Using --kubernetes-versionflag will install the specific version

• On existing VM running just 'minikube start' will automatically update kubernetes to the latest version (as specified in the code)

Analysis

• The UpdateCluster(..) is the central function that drive copying, configuring and installing all the different kubernetes files. This function will always be called whether there is upgrade to kubernetes version or not.

• The behaviour should be when user specify--kubernetes-version minikube should just install the specified version for both new or existing VM. When the flag is not specified then minikube should just run the VM without doing anything to the installed version of kubernetes

• Introduce a new flag --upgrade-kubernetes to allow user to automatically update kubernetes if they don't know what is the latest version

[tstromberg]

@nanikjava - Your approach sounds great.

I'd like to personally be able to set minikube config set upgrade-kubernetes true for automatic-updates (preserve old behavior), but I acknowledge that most people want predictability.

[tstromberg]

Thought: rather than having two flags which may have a confusing interaction (--kubernetes-version and --upgrade-kubernetes), could we just get by with --kubernetes-version?

• Leave --kubernetes-version to default as "", which means 'currently deployed version'
• If an update is available, output a tip to inform the user that they may use --kubernetes-version=<version>
• If --kubernetes-version is specifically set, upgrade the cluster rather than ignore the argument.

In a future iteration, we could also build upon your work to support:

• --kubernetes-version=latest
• --kubernetes-version=stable

[nanikjava]

• Leave --kubernetes-version to default as "", which means 'currently deployed version'
• If an update is available, output a tip to inform the user that they may use --kubernetes-version=<version>
• If --kubernetes-version is specifically set, upgrade the cluster rather than ignore the argument.

Logic sounds good. Will implement it.

[nanikjava]

@nanikjava - Your approach sounds great.

I'd like to personally be able to set minikube config set upgrade-kubernetes true for automatic-updates (preserve old behavior), but I acknowledge that most people want predictability.

It's a good idea to have auto upgrade however I think most minikube user would like to have some sort of control over things that are deployed inside the VM.

[tstromberg]

Closing. This was fixed in v1.6! Thank you @nanikjava