Kops: Nodes of mixedinstancegroups are in "NeedsUpdate" state even after rolling-update.

[vivekgarg20]

1. What kops version are you running? The command kops version, will display
this information.
Version 1.12.0-beta.2 (git-d1453d22a)
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a"
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
Applied mixed instance policy to one of instance group as introduced
Sample spec

  mixedInstancesPolicy:
    instances:
    - m5.2xlarge
    - c5.2xlarge
    - r5.2xlarge
    onDemandAboveBase: 0
    spotInstancePools: 3

did rolling-update --yes for instance groupl
5. What happened after the commands executed?
Kops nodes remained in "NeedsUpdate" state.
6. What did you expect to happen?
Nodes to go into "Ready" state.

**7. Please provide your cluster manifest. Execute
specs provided above.

[vivekgarg20]

@gambol99 you may want to look at it. Going to test with beta.1 as well. Will update status accordingly.

[granular-ryanbonham]

Can you post the exact kops commands you ran to update your cluster and to do the rolling-update. I thought we had this fixed in beta 2.

[vivekgarg20]

kops update cluster --yes
kops rolling-update cluster --instance-group node-m5-1b --yes

[vasilgramov]

Could this be happening when explicity we are specifying which instance groups to be rolled out?

[vivekgarg20]

@vasilgramov Nopes, tried rolling-update for full cluster, same state. Though "validate cluster" command displays ready state. It says NeedsUpdate if I fire "rolling-update cluster" again

[bzuelke]

Also running into this, tested out the mixed instance types and still shows that it needs an update. Is there a potential for a cherry pick fix for this, it's a pretty decent annoyance tbh.

[clifford-sanders]

We also encountered this problem. Debugging revealed that the version of the launch templated associated with the ASG is not set.

The check happens in https://github.com/kubernetes/kops/blob/release/pkg/cloudinstances/cloud_instance_group.go#L69

if newGroupName == currentGroupName {
    c.Ready = append(c.Ready, cm)
} else {
    c.NeedUpdate = append(c.NeedUpdate, cm)
}

with

    newGroupName = nodes.somename-20190605111108:
currentGroupName = nodes.somename-20190605111108:1

The missing 1 is the version of the launch template associated with the ASG.

The launch template name is determined in https://github.com/kubernetes/kops/blob/release/upup/pkg/fi/cloudup/awsup/aws_cloud.go#L597

name = aws.StringValue(g.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification.LaunchTemplateName)
version := aws.StringValue(g.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification.Version)
if name != "" {
    launchTemplate := name + ":" + version
    return launchTemplate, nil
}

Problem is that version is empty.

When setting the Launch Template Version in the AWS console to 1 the nodes are not marked as NEEDSUPDATE anymore.

[gambol99]

Apologizes for the long delay .. I've not had a chance to work on kops for a while .. Admittedly we've never seen the problem, but that's probably due to us cutting out own releases off master rather than using the officials. Based on the above, i'm guessing eca2b0a27aaa706dc760ce7d2708a54aa85a6f0b introduced a potential issue .. I'll take a look and get it fixed ..

[gambol99]

So yes ... built off master and replicated the problem, reverted https://github.com/kubernetes/kops/commit/eca2b0a27aaa706dc760ce7d2708a54aa85a6f0b and used binary which fixes it .. The names will always be different given the additional of : and an empty version

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Deepak1100]

A similar issue is there if I use export KOPS_FEATURE_FLAGS="+EnableLaunchTemplates" flag.
even after update nodes stays in NeedsUpdate status

I am using kops 1.14.1 version.

NAME            STATUS      NEEDUPDATE  READY   MIN MAX NODES
master-ap-south-1a-1    NeedsUpdate 1       0   1   1   1
master-ap-south-1a-2    NeedsUpdate 1       0   1   1   1
master-ap-south-1b-1    NeedsUpdate 1       0   1   1   1

can be reopen this issue.

[akhfa]

A similar issue is there if I use export KOPS_FEATURE_FLAGS="+EnableLaunchTemplates" flag.
even after update nodes stays in NeedsUpdate status

I am using kops 1.14.1 version.

NAME          STATUS      NEEDUPDATE  READY   MIN MAX NODES
master-ap-south-1a-1  NeedsUpdate 1       0   1   1   1
master-ap-south-1a-2  NeedsUpdate 1       0   1   1   1
master-ap-south-1b-1  NeedsUpdate 1       0   1   1   1

can be reopen this issue.

Still happened in Version 1.15.2 (git-ad595825a). Doesn't try latest version yet

[MoShitrit]

This was fixed with versions 1.16.1 and up.