Kops: Instance groups stuck in "NeedsUpdate"

[trondhindenes]

1. What kops version are you running? The command kops version, will display
this information.
1.12.1

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
v1.12.7

3. What cloud provider are you using?
aws

4. What commands did you run? What is the simplest way to reproduce this issue?
After running kops rolling-update, my instance groups are still reporting "NeedsUpdate". I don't understand this, since nodes are literally 5 minutes old. How can I see "why" kops things they need update? I've tried looking at the documentation for rolling-update parameters but the documentation isn't really clear about what flags are available or what they do.
So I guess my bug is: I don't understand why kops reports NeedsUpdate for my nodes, and I don't understand how I can figure out why.

5. What happened after the commands executed?
Instance Groups are still reporting NeedsUpdate.

6. What did you expect to happen?
Instance Groups should report Ready since they just got rolling-updated.

[JRemitz]

This happened to me after changing to a launch template of mixed instance types and using spot instances. I don't know if related, but otherwise my cluster had no issues up to that point.

Running kops update cluster shows that the SpotPrice wants to be modified, despite updating and rolling cluster. The update doesn't appear to change it.

Will modify resources:
  LaunchTemplate/nodes.example.com
        SpotPrice                 -> 0.10

Despite updating with kops update cluster --yes the Spot Price continues to tell me that it needs to be updated. If it's helpful to reference, I am using a kops template that has this setting.

[JRemitz]

May be related to #7066 with the issue I'm seeing above.

[yurrriq]

We appear to be seeing this with kops 1.13.0, k8s 1.13.10, and a kops template with maxPrice

We are using {{ .maxPrice | quote }} so the resulting config looks like "0.0.1".

[smirgel]

We are also getting stuck with eternal "NeedsUpdate".

In our case I think it is because the following code looks for default version in the launch template:
https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/awsup/aws_cloud.go#L607

But the default version is the first version, not the latest:

I guess default version should be updated when a new template is rolled out?

We are generating terraform code which I think might be the issue here.

[smirgel]

Looks like it is a problem related to terraform in our case:
https://github.com/terraform-providers/terraform-provider-aws/issues/4655

[Deepak1100]

I am also facing the same issue
The launch template is set to the default version is the first version of the template insisted of the latest version.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Vlli]

Facing the same issue again.
It was fixed in version 1.14.0, after upgrading to 1.15.2 the bug is back again.
Every Instance-Group with mixed instance is "needed to update" every time we perform an rolling-update even the instance-group was updated minutes ago.

[kirrmann]

I also still experience this error with kops 1.16.0

[dataviruset]

It would be great with a feature in Kops that would show you why an InstanceGroup would need to be updated. A diff of some kind?

[hakman]

This issue is fixed in 1.18 and should be testable with Kops 1.18.0-alpha.2.
I also created backports for 1.16 https://github.com/kubernetes/kops/pull/8808 and 1.17 https://github.com/kubernetes/kops/pull/8807.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[hakman]

/close

[k8s-ci-robot]

@hakman: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MeirP-3]

Hello,

I encounter this issue with kops 1.18.2 (git-84495481e4).
The weird thing is, one instance group which usually has 0-4 nodes got the update.
But for the other instance group which usually has 8-13 nodes, kops reports status NEEDUPDATE for all the nodes except one.

Kubernetes version: 1.14.2

using kops with "+Spotinst" feature flag.

[MeirP-3]

Submitted new issue #10166