Eksctl: Failed to update managed nodegroups from 1.16 to 1.17

Created on 21 Aug 2020  Â·  11Comments  Â·  Source: weaveworks/eksctl

What happened?
I want to upgrade my managed nodegroup from 1.16 to 1.17. The nodegroups AMI version is: 1.16.13-20200814.

eksctl get nodegroups --cluster=bs-k-dev-v1 -o yaml node-group-az-b-v1
- Cluster: bs-k-dev-v1
  CreationTime: "2020-03-05T07:22:31.924Z"
  DesiredCapacity: 1
  ImageID: ""
  InstanceType: t3.xlarge
  MaxSize: 4
  MinSize: 1
  Name: node-group-az-b-v1
  NodeInstanceRoleARN: ""
  StackName: eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1

Running upgrade node group runs into this error:

eksctl upgrade nodegroup -f clusters/bs-k-dev-v1.yml --cluster=bs-k-dev-v1 --name=node-group-az-b-v1 --kubernetes-version=1.17
[ℹ]  Update nodegroup stack
[✖]  unexpected status "UPDATE_ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1"
[ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: CREATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: CREATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: CREATE_IN_PROGRESS – "User Initiated"
Error: error updating nodegroup stack: waiting for CloudFormation stack "eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1": ResourceNotReady: failed waiting for successful resource state

Without setting the kubernetes version, eksctl upgrade doesn't do anything.

eksctl upgrade nodegroup -f clusters/bs-k-dev-v1.yml --cluster=bs-k-dev-v1 --name=node-group-az-b-v1 
[ℹ]  nodegroup "node-group-az-b-v1" is already up-to-date

What you expected to happen?
The expected result is an updated nodegroup to kubernetes version 1.17

How to reproduce it?
Include the steps to reproduce the bug

Anything else we need to know?
What OS are you using, are you using a downloaded binary or did you compile eksctl, what type of AWS credentials are you using (i.e. default/named profile, MFA) - please don't include actual credentials though!

Versions
Please paste in the output of these commands:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.11", GitCommit:"ea5f00d93211b7c80247bf607cfa422ad6fb5347", GitTreeState:"clean", BuildDate:"2020-08-13T15:20:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}


$ eksctl version
0.26.0

Logs

kinbug
Source
picture hikhvar

Most helpful comment

I still see this issue with eksctl 0.28.0:

@hikhvar thanks for the detailed report. We've found the issue and a fix will be out in the next release.

picture cPu1 on 5 Oct 2020
🎉1 👍1

All 11 comments

Updating the nodegroup via the AWS console works with one click and without problems.

hikhvar picture hikhvar on 24 Aug 2020

Updating the nodegroup via the AWS console works with one click and without problems.

Updating should work via the console, this is a CloudFormation-specific issue. We're working on a fix.

cPu1 picture cPu1 on 24 Aug 2020
👍1

@hikhvar the fix is merged and will be out in the next release. Thanks for reporting this issue.

Additionally, just for confirming, could you share the output of these commands?

aws cloudformation describe-stacks --stack-name eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1 --region <region>

aws cloudformation get-template --stack-name eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1 --region <region>
cPu1 picture cPu1 on 24 Aug 2020 
aws cloudformation describe-stacks --stack-name eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1 --region <region>

{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:eu-central-1:1234567890:stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1/....",
            "StackName": "eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1",
            "Description": "EKS Managed Nodes (SSH access: false) [created by eksctl]",
            "CreationTime": "2020-03-05T07:22:31.924Z",
            "LastUpdatedTime": "2020-08-21T08:34:57.299Z",
            "RollbackConfiguration": {},
            "StackStatus": "UPDATE_ROLLBACK_COMPLETE",
            "DisableRollback": false,
            "NotificationARNs": [],
            "Capabilities": [
                "CAPABILITY_IAM"
            ],
            "Tags": [
                {
                    "Key": "alpha.eksctl.io/cluster-name",
                    "Value": "bs-k-dev-v1"
                },
                {
                    "Key": "alpha.eksctl.io/nodegroup-name",
                    "Value": "node-group-az-b-v1"
                },
                {
                    "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                    "Value": "bs-k-dev-v1"
                },
                {
                    "Key": "alpha.eksctl.io/nodegroup-type",
                    "Value": "managed"
                },
                {
                    "Key": "monitoring",
                    "Value": "disabled"
                },
                {
                    "Key": "alpha.eksctl.io/eksctl-version",
                    "Value": "0.14.0-rc.0"
                },
                {
                    "Key": "Name",
                    "Value": "bs-k-dev-v1-az-b-v1"
                }
            ],
            "EnableTerminationProtection": false,
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

and:

aws cloudformation get-template --stack-name eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1 --region <region>

{
    "TemplateBody": {
        "AWSTemplateFormatVersion": "2010-09-09",
        "Description": "EKS Managed Nodes (SSH access: false) [created by eksctl]",
        "Resources": {
            "ManagedNodeGroup": {
                "Type": "AWS::EKS::Nodegroup",
                "Properties": {
                    "ReleaseVersion": "1.15.11-20200618",
                    "ClusterName": "bs-k-dev-v1",
                    "NodegroupName": "node-group-az-b-v1",
                    "ScalingConfig": {
                        "MinSize": 1,
                        "MaxSize": 4,
                        "DesiredSize": 1
                    },
                    "DiskSize": 20,
                    "Subnets": [
                        "subnet-asdgasd"
                    ],
                    "InstanceTypes": [
                        "t3.xlarge"
                    ],
                    "AmiType": "AL2_x86_64",
                    "NodeRole": {
                        "Fn::GetAtt": "NodeInstanceRole.Arn"
                    },
                    "Labels": {
                        "alpha.eksctl.io/cluster-name": "bs-k-dev-v1",
                        "alpha.eksctl.io/nodegroup-name": "node-group-az-b-v1",
                        "role": "worker"
                    },
                    "Tags": {
                        "Name": "bs-k-dev-v1-az-b-v1",
                        "alpha.eksctl.io/nodegroup-name": "node-group-az-b-v1",
                        "alpha.eksctl.io/nodegroup-type": "managed",
                        "monitoring": "disabled"
                    }
                }
            },
            "NodeInstanceRole": {
                "Type": "AWS::IAM::Role",
                "Properties": {
                    "AssumeRolePolicyDocument": {
                        "Statement": [
                            {
                                "Action": [
                                    "sts:AssumeRole"
                                ],
                                "Effect": "Allow",
                                "Principal": {
                                    "Service": [
                                        "ec2.amazonaws.com"
                                    ]
                                }
                            }
                        ],
                        "Version": "2012-10-17"
                    },
                    "ManagedPolicyArns": [
                        "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
                        "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
                        "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
                    ],
                    "Path": "/"
                }
            }
        }
    },
    "StagesAvailable": [
        "Original",
        "Processed"
    ]
}
hikhvar picture hikhvar on 24 Aug 2020

Thanks! Your nodegroup was created with 0.14.0-rc.0, which is quite old and the nodegroup template format has changed since then. If you have more nodegroups like that, the next version of eksctl will let you ugprade them.

cPu1 picture cPu1 on 24 Aug 2020

Thank you for the quick response!

hikhvar picture hikhvar on 24 Aug 2020

I still see this issue with eksctl 0.28.0:

 eksctl upgrade nodegroup -f clusters/bs-k-dev-v1.yml --cluster=bs-k-dev-v1 --name=node-group-az-b-v1 
[ℹ]  upgrading nodegroup version
[ℹ]  Update nodegroup stack
[✖]  unexpected status "UPDATE_ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1"
[ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_ROLLBACK_IN_PROGRESS – "The following resource(s) failed to update: [ManagedNodeGroup]. "
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_FAILED – "Version and ReleaseVersion updates cannot be combined with other updates"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: UPDATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: UPDATE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: CREATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_COMPLETE
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/NodeInstanceRole: CREATE_COMPLETE
[ℹ]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1: CREATE_IN_PROGRESS – "User Initiated"
Error: error updating nodegroup stack: waiting for CloudFormation stack "eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1": ResourceNotReady: failed waiting for successful resource state

aws cloudformation describe-stacks --stack-name eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1
{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:eu-central-1:123455609:stack/eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1/13df1c80-5eb2-11ea-a015-0a24e2904074",
            "StackName": "eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1",
            "Description": "EKS Managed Nodes (SSH access: false) [created by eksctl]",
            "CreationTime": "2020-03-05T07:22:31.924Z",
            "LastUpdatedTime": "2020-09-22T11:50:51.745Z",
            "RollbackConfiguration": {},
            "StackStatus": "UPDATE_ROLLBACK_COMPLETE",
            "DisableRollback": false,
            "NotificationARNs": [],
            "Capabilities": [
                "CAPABILITY_IAM"
            ],
            "Tags": [
                {
                    "Key": "alpha.eksctl.io/cluster-name",
                    "Value": "bs-k-dev-v1"
                },
                {
                    "Key": "alpha.eksctl.io/nodegroup-name",
                    "Value": "node-group-az-b-v1"
                },
                {
                    "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                    "Value": "bs-k-dev-v1"
                },
                {
                    "Key": "alpha.eksctl.io/nodegroup-type",
                    "Value": "managed"
                },
                {
                    "Key": "monitoring",
                    "Value": "disabled"
                },
                {
                    "Key": "alpha.eksctl.io/eksctl-version",
                    "Value": "0.26.0"
                },
                {
                    "Key": "Name",
                    "Value": "bs-k-dev-v1-az-b-v1"
                }
            ],
            "EnableTerminationProtection": false,
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

aws cloudformation get-template --stack-name eksctl-bs-k-dev-v1-nodegroup-node-group-az-b-v1
{
    "TemplateBody": {
        "AWSTemplateFormatVersion": "2010-09-09",
        "Description": "EKS Managed Nodes (SSH access: false) [created by eksctl]",
        "Resources": {
            "ManagedNodeGroup": {
                "Type": "AWS::EKS::Nodegroup",
                "Properties": {
                    "ReleaseVersion": "1.15.11-20200618",
                    "ClusterName": "bs-k-dev-v1",
                    "NodegroupName": "node-group-az-b-v1",
                    "ScalingConfig": {
                        "MinSize": 1,
                        "MaxSize": "10",
                        "DesiredSize": "4"
                    },
                    "DiskSize": 20,
                    "Subnets": [
                        "subnet-b17bf8ca"
                    ],
                    "InstanceTypes": [
                        "t3.xlarge"
                    ],
                    "AmiType": "AL2_x86_64",
                    "NodeRole": {
                        "Fn::GetAtt": "NodeInstanceRole.Arn"
                    },
                    "Labels": {
                        "alpha.eksctl.io/cluster-name": "bs-k-dev-v1",
                        "alpha.eksctl.io/nodegroup-name": "node-group-az-b-v1",
                        "role": "worker"
                    },
                    "Tags": {
                        "Name": "bs-k-dev-v1-az-b-v1",
                        "alpha.eksctl.io/nodegroup-name": "node-group-az-b-v1",
                        "alpha.eksctl.io/nodegroup-type": "managed",
                        "monitoring": "disabled"
                    }
                }
            },
            "NodeInstanceRole": {
                "Type": "AWS::IAM::Role",
                "Properties": {
                    "AssumeRolePolicyDocument": {
                        "Statement": [
                            {
                                "Action": [
                                    "sts:AssumeRole"
                                ],
                                "Effect": "Allow",
                                "Principal": {
                                    "Service": [
                                        "ec2.amazonaws.com"
                                    ]
                                }
                            }
                        ],
                        "Version": "2012-10-17"
                    },
                    "ManagedPolicyArns": [
                        "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
                        "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
                        "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
                    ],
                    "Path": "/"
                }
            }
        }
    },
    "StagesAvailable": [
        "Original",
        "Processed"
    ]
}

Maybe the issue is: I triggered an upgade of the managed nodegroup via the eks UI to 1.17?

hikhvar picture hikhvar on 22 Sep 2020

Hi! I also experience this issue in 1.17 via CDK 1.64.1 with LaunchTemplate (no releaseVersion provided to the nodegroup) version update.

damienpuig-h4 picture damienpuig-h4 on 30 Sep 2020
👀1

@albertomurillo

To reproduce this:

  • Define a CloudFormation stack that container a LaunchTemplate + NodeGroup
  • In the LaunchTemplate , specify the TagsSpecifications with some tags that always update (e.g. timestamp)
  • References the Latest LaunchTemplate Version in the NodeGroup LaunchTemplate spec (id / version tuple with version being either the CDK LaunchConfiguration property or "$Latest")
  • Update the CloudFormation stack

Expected Behavior:

CloudFormation should recreate a new version of the LaunchTemplate, Then use the newly created LaunchTemplate id in the nodegroup, which triggers the node group update.

Instead, I hit the same Version and ReleaseVersion updates cannot be combined with other updates issue.

Interestingly - the update works fine if the update happen in a N+1 deployment, referencing manually (not the launch template prop) the launch template created in deployment N.

Hope that makes sense, happy to provide mode details.

Note: This might not be related to eksctl - but I think is still worth mentioning!

damienpuig-h4 picture damienpuig-h4 on 1 Oct 2020

I still see this issue with eksctl 0.28.0:

@hikhvar thanks for the detailed report. We've found the issue and a fix will be out in the next release.

cPu1 picture cPu1 on 5 Oct 2020
🎉1 👍1

Thank you for fixing this. Today I run the upgrade from 1.17 to 1.18 without any issues.

hikhvar picture hikhvar on 22 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jcleal picture jcleal  Â·  4Comments
martina-if picture martina-if  Â·  4Comments
errordeveloper picture errordeveloper  Â·  3Comments
arun-gupta picture arun-gupta  Â·  3Comments
scottyhq picture scottyhq  Â·  3Comments