Sceptre: unable to execute a change-set on stack that has dependencies

[nunogrl]

hello

So I'm trying to execute a change set on command line to a stack.
In this case is to add a record on route53.
The stack prod/route53/mydomain-com-zone creates the domain zone
the stack prod/route53/mydomain-com-alias-03-records depends on the previous one and creates alias records

#
#  Create the change-set
#
$ sceptre  create prod/route53/mydomain-com-alias-03-records add-new-alias
Do you want to create change set 'add-new-alias' for 'prod/route53/mydomain-com-alias-03-records' [y/N]: y
[2019-09-26 15:13:00] - prod/route53/mydomain-com-zone - Successfully initiated creation of Change Set 'add-new-alias'
[2019-09-26 15:13:00] - prod/route53/mydomain-com-alias-03-records - Successfully initiated creation of Change Set 'add-new-alias'

#
#  List the change-set
#
$ sceptre list change-sets prod/route53/mydomain-com-alias-03-records # add-new-alias
{
    "prod/route53/mydomain-com-zone": [
        {
            "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:stack/project01-prod-route53-mydomain-com-zone/XXXfXXeX-dfa5-11e9-8180-0615f1e712e0",
            "StackName": "project01-prod-route53-mydomain-com-zone",
            "ChangeSetId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:changeSet/add-new-alias/XXXfXXeX-5a2a-4f85-831e-58ad9a7697f6",
            "ChangeSetName": "add-new-alias",
            "ExecutionStatus": "UNAVAILABLE",
            "Status": "FAILED",
            "StatusReason": "The submitted information didn't contain changes. Submit different information to create a change set.",
            "CreationTime": "2019-09-26 14:13:00.051000+00:00"
        }
    ]
}
{
    "prod/route53/mydomain-com-alias-03-records": [
        {
            "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:stack/project01-prod-route53-mydomain-com-alias-03-records/XXXfXXeX-dfad-11e9-b0d1-0290198c1bb6",
            "StackName": "project01-prod-route53-mydomain-com-alias-03-records",
            "ChangeSetId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:changeSet/add-new-alias/5bad72aa-a0f2-42ec-b9b0-ce219c5bf4c0",
            "ChangeSetName": "add-new-alias",
            "ExecutionStatus": "AVAILABLE",
            "Status": "CREATE_COMPLETE",
            "CreationTime": "2019-09-26 14:13:00.463000+00:00"
        }
    ]
}

#
#  Describe the change-set
#
$ sceptre describe change-set prod/route53/mydomain-com-alias-03-records add-new-alias
{
    "ChangeSetName": "add-new-alias",
    "StackName": "project01-prod-route53-mydomain-com-zone",
    "CreationTime": "2019-09-26 14:13:00.051000+00:00",
    "ExecutionStatus": "UNAVAILABLE",
    "Status": "FAILED",
    "StatusReason": "The submitted information didn't contain changes. Submit different information to create a change set.",
    "Changes": []
}
{
    "ChangeSetName": "add-new-alias",
    "StackName": "project01-prod-route53-mydomain-com-alias-03-records",
    "CreationTime": "2019-09-26 14:13:00.463000+00:00",
    "ExecutionStatus": "AVAILABLE",
    "Status": "CREATE_COMPLETE",
    "Changes": [
        {
            "ResourceChange": {
                "Action": "Add",
                "LogicalResourceId": "adduk10dstagingaliasrecord",
                "ResourceType": "AWS::Route53::RecordSet",
                "Scope": []
            }
        }
    ]
}

#
#  Execute the change-set
#
$ sceptre execute prod/route53/mydomain-com-alias-03-records add-new-alias
Do you want to execute_change_set change set 'add-new-alias' for ' prod/route53/mydomain-com-alias-03-records' [y/N]: y
"An error occurred (InvalidChangeSetStatus) when calling the ExecuteChangeSet operation: ChangeSet [arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:stack/project01-prod-route53-mydomain-com-zone/XXXfXXeX-dfa5-11e9-8180-0615f1e712e0] cannot be executed in its current status of [FAILED]"
$

I can execute the change-set on the AWS console, but here we can see that it applied the change-set to the stack and to the dependency as well.

So it started to apply it to a stack that showed Status: FAILED, and after failing on this one it just gave up on the others.

This could go really bad, if I was using generic names across stacks

Anyway I couldn't find a way to pass the option --ignore-dependencies on this situation.

Thanks for your time on this

[ngfgrant]

hey @nunogrl

Sorry to hear you are having difficulty.

Could you try to run

sceptre --ignore-dependencies create prod/route53/mydomain-com-alias-03-records add-new-alias?

The ignore-dependencies should just work as normal. Hope that helps.

[nunogrl]

I've tried that before, but it crashes :(

$ sceptre --ignore-dependencies create prod/route53/mydomain-com-alias-03-records add-new-alias
Traceback (most recent call last):
  File "/home/nuno/Envs/sceptre-220/bin/sceptre", line 10, in <module>
    sys.exit(cli())
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/sceptre/cli/helpers.py", line 36, in decorated
    return func(*args, **kwargs)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/sceptre/cli/create.py", line 44, in create_command
    plan.create_change_set(change_set_name)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/sceptre/plan/plan.py", line 245, in create_change_set
    return self._execute(*args)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/sceptre/plan/plan.py", line 40, in _execute
    return executor.execute(*args)
  File "/home/nuno/Envs/sceptre-220/lib/python3.7/site-packages/sceptre/plan/executor.py", line 49, in execute
    with ThreadPoolExecutor(max_workers=self.num_threads) as executor:
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 129, in __init__
    raise ValueError("max_workers must be greater than 0")
ValueError: max_workers must be greater than 0

[ngfgrant]

ok thanks which version of sceptre are you running?

[nunogrl]

sceptre --version
Sceptre, version 2.2.1

[nunogrl]

I'm attaching an example so you can replicate this problem.
Instead of ALIAS record, I'm attaching an A record, as this type has no dependencies

Thanks

sceptre_route53_a_record.zip

[ngfgrant]

Thanks that is useful

[nunogrl]

After deleting the change-set on the dependency I'm still not able to execute the change-set.

$ sceptre execute prod/route53/mydomain-com-a-records test-a-record 
Do you want to execute_change_set change set 'test-a-record' for 'prod/route53/mydomain-com-a-records' [y/N]: y
"An error occurred (ChangeSetNotFound) when calling the ExecuteChangeSet operation: ChangeSet [test-a-record] does not exist"

As a workaround for this, I wrote this to clean up useless change-sets

I called it execute-change-set.sh
the parameters are

1. the stack name (the sceptre convention)
2. the change-set name

it's just a quick fix, until there's a definitive solution

#!/bin/sh

CHANGESET=${2:-.}
STACK=${1:-prod}
# get list of stacks
sceptre list change-sets ${STACK} | jq -r '.[]|.[]| "\(.StackName) \(.ChangeSetName) \(.ExecutionStatus) \(.ChangeSetId)"' | egrep "${CHANGESET}" | while read line; do
    set -- $(echo $line)
    if [ "${3}" = "UNAVAILABLE" ]; then
        echo "DELETE: CHANGE-SET \"${2}\" on \"${1}\""
        aws cloudformation delete-change-set --change-set-name ${4}
    else
        echo "EXECUTE: CHANGE-SET \"${2}\" on \"${1}\""
        aws cloudformation execute-change-set --change-set-name ${4}
    fi
done

Here follows how I'm using this:

$ sceptre create prod/route53/mydomain-com-a-records test25-a-record
Do you want to create change set 'test25-a-record' for 'prod/route53/mydomain-com-a-records' [y/N]: y
[2019-09-27 15:11:21] - prod/route53/mydomain-com-zone - Successfully initiated creation of Change Set 'test25-a-record'
[2019-09-27 15:11:21] - prod/route53/mydomain-com-a-records - Successfully initiated creation of Change Set 'test25-a-record'
$ sceptre describe change-set  prod/route53/mydomain-com-a-records test25-a-record
{
    "ChangeSetName": "test25-a-record",
    "StackName": "tests-prod-route53-mydomain-com-zone",
    "CreationTime": "2019-09-27 14:11:21.481000+00:00",
    "ExecutionStatus": "UNAVAILABLE",
    "Status": "FAILED",
    "StatusReason": "The submitted information didn't contain changes. Submit different information to create a change set.",
    "Changes": []
}
{
    "ChangeSetName": "test25-a-record",
    "StackName": "tests-prod-route53-mydomain-com-a-records",
    "CreationTime": "2019-09-27 14:11:21.907000+00:00",
    "ExecutionStatus": "AVAILABLE",
    "Status": "CREATE_COMPLETE",
    "Changes": [
        {
            "ResourceChange": {
                "Action": "Add",
                "LogicalResourceId": "addtest25arecord",
                "ResourceType": "AWS::Route53::RecordSet",
                "Scope": []
            }
        },
        {
            "ResourceChange": {
                "Action": "Remove",
                "LogicalResourceId": "addtest41324arecord",
                "PhysicalResourceId": "test41324.mydomain.com",
                "ResourceType": "AWS::Route53::RecordSet",
                "Scope": []
            }
        }
    ]
}

$ sh execute-change-set.sh prod/route53/mydomain-com-a-records test25-a-record
DELETE: CHANGE-SET "test25-a-record" on "tests-prod-route53-mydomain-com-zone"
EXECUTE: CHANGE-SET "test25-a-record" on "tests-prod-route53-mydomain-com-a-records"

[sebastiankasprzak]

hitting same problem with change sets execution.

Is the issue with running stacks with --ignore-dependencies worth separate thread? hit that problem when trying to run sceptre update command.

[mvanholsteijn]

ignore-dependencies does not do it, because my dependencies are caused by stack_output input parameters. The change-set cannot be created without the correct input parameters.