Aws-cli: add a command to cloudformation wait any final state, and display events while waiting

[hnafar]

This is specially useful during automated deployments.

• wait until stack is in any final state:
  
  
  
  • CREATE_FAILED
  
  
  • DELETE_COMPLETE
  
  
  • DELETE_FAILED
  
  
  • ROLLBACK_COMPLETE
  
  
  • ROLLBACK_FAILED
  
  
  • UPDATE_COMPLETE
  
  
  • UPDATE_ROLLBACK_COMPLETE
  
  
  • UPDATE_ROLLBACK_FAILED
  
  
• log stack events emitted while waiting.
  Currently I achieve this using the following script, but having this as part of the cli would be much more helpful:

cloudformation_tail() {
  local stack="$1"
  local region="$2"
  local lastEvent
  local lastEventId
  local stackStatus=$(aws cloudformation describe-stacks --region $region --stack-name $stack | jq -c -r .Stacks[0].StackStatus)

  until \
    [ "$stackStatus" = "CREATE_COMPLETE" ] \
    || [ "$stackStatus" = "CREATE_FAILED" ] \
    || [ "$stackStatus" = "DELETE_COMPLETE" ] \
    || [ "$stackStatus" = "DELETE_FAILED" ] \
    || [ "$stackStatus" = "ROLLBACK_COMPLETE" ] \
    || [ "$stackStatus" = "ROLLBACK_FAILED" ] \
    || [ "$stackStatus" = "UPDATE_COMPLETE" ] \
    || [ "$stackStatus" = "UPDATE_ROLLBACK_COMPLETE" ] \
    || [ "$stackStatus" = "UPDATE_ROLLBACK_FAILED" ]; do

    #[[ $stackStatus == *""* ]] || [[ $stackStatus == *"CREATE_FAILED"* ]] || [[ $stackStatus == *"COMPLETE"* ]]; do
    lastEvent=$(aws cloudformation describe-stack-events --region $region --stack $stack --query 'StackEvents[].{ EventId: EventId, LogicalResourceId:LogicalResourceId, ResourceType:ResourceType, ResourceStatus:ResourceStatus, Timestamp: Timestamp }' --max-items 1 | jq .[0])
    eventId=$(echo "$lastEvent" | jq -r .EventId)
    if [ "$eventId" != "$lastEventId" ]
    then
        lastEventId=$eventId
        echo $(echo $lastEvent | jq -r '.Timestamp + "\t-\t" + .ResourceType + "\t-\t" + .LogicalResourceId + "\t-\t" + .ResourceStatus')
    fi
    sleep 3
    stackStatus=$(aws cloudformation describe-stacks --region $region --stack-name $stack | jq -c -r .Stacks[0].StackStatus)
  done

  echo "Stack Status: $stackStatus"
}

cloudformation_tail $SERVICE_STACK_NAME $REGION

which produces the following result on an update:

Awaiting completion...
2017-10-11T19:28:51.066Z - AWS::CloudFormation::Stack - my-stack-name - UPDATE_IN_PROGRESS
2017-10-11T19:28:57.591Z - AWS::ECS::TaskDefinition - TaskDefinition - UPDATE_IN_PROGRESS
2017-10-11T19:29:00.282Z - AWS::ECS::Service - Service - UPDATE_IN_PROGRESS
2017-10-11T19:36:05.919Z - AWS::CloudFormation::Stack - my-stack-name - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
Stack Status: UPDATE_COMPLETE

[jamesls]

Thanks for the suggestion, marking as a feature request.

[pcgeek86]

FYI the AWS Tools for PowerShell currently offer this feature, starting with version 3.3.119.0. The SDK includes a Wait-CFNStack command, which pauses execution until the desired CloudFormation stack state has been reached.

There's an example of its use in the blog post and documentation, linked to below.

AWS Developer Blog Post (2017-07-11)
AWS Tools for PowerShell Documentation Page

Cheers,
Trevor Sullivan

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[ghost]

Would be great to have this implemented.
My current solution look like this

aws cloudformation deploy --stack-name "${stack_name}" \
  --template-file "${stack_template_file}" \
  --no-fail-on-empty-changeset \
  | while read -r line; do echo "$line"
      if [[ "$line" = "Waiting for stack create/update to complete" ]]; then
        "aws-cloudformation-deploy-watch.sh" --stack-name "${stack_name}"
      fi
    done

aws-cloudformation-deploy-watch.sh

#!/usr/bin/env sh -euo pipefail

############################## Usage ################################
#
# aws-cloudformation-deploy-watch.sh --stack 'STACK_NAME'
#
#####################################################################

while [ $# -gt 0 ]; do
  case $1 in
    --stack-name) stack_name_opt="$2"; shift 2;;    
    --profile)    profile_opt="$2"; shift 2 ;;
    --region)     region_opt="$2"; shift 2;;
    *) echo "$0: unexpected argument: $1" >&2; exit 1;;
  esac
done

_TAB=$'\t'
most_recent_event_id='???'
while : ; do
  # get recent stack events
  most_recent_event_id_sed_escaped="$(echo "$most_recent_event_id" | sed -e 's/[]\/$*.^[]/\\&/g')"
  recent_events="$(aws cloudformation describe-stack-events \
      $([ -n "${stack_name_opt:-}" ] && echo "--stack-name $stack_name_opt") \
      --query 'StackEvents[*].[EventId,Timestamp,LogicalResourceId,ResourceStatus,ResourceStatusReason]' \
      $([ -n "${profile_opt:-}" ] && echo "--profile $profile_opt") \
      $([ -n "${region_opt:-}" ] && echo "--region $region_opt") \
      --output text \
    | sed  "/${_TAB}User Initiated\$/q" | sed "/^${most_recent_event_id_sed_escaped}${_TAB}/,\$d" \
    | tail -r)"
  if [ -n "$recent_events" ]; then
    # save last event id
    most_recent_event_id=$(echo "$recent_events" | tail -1 | awk -F $'\t' '{printf $1}')
    while read -r event; do
      # print new event
      echo "$event" | awk -F $'\t' '{printf "%-28s %-48s %-24s %s\n",$2,$3,$4,$5}' | sed "s/None$//"
      # looking for stack resource events
      event_resouce=$(echo "$event" | awk -F $'\t' '{printf $3}')
      if [ "$event_resouce" == "$stack_name_opt" ]; then
        stack_status="$(echo "$event" | awk '{printf $4}')"
        # exit on final stack status
        if [[ "$stack_status" = *'_FAILED' ]] || [[ "$stack_status" = *'_ROLLBACK_COMPLETE' ]]; then exit 1; fi
        if [[ "$stack_status" = *'_COMPLETE' ]]; then exit 0; fi
      fi
    done <<< "$recent_events"
  fi
  sleep 1
done

[thearthur]

Can we re-open this here since the uservoice migration appears no longer to be a thing? (or at least the links here are broken)

[lantrix]

It seems this has been implemented here:
https://docs.aws.amazon.com/cli/latest/reference/cloudformation/wait/index.html

However you need to call different operations with the CLI to differentiate between waiting for a Stack Create vs. a Stack Update to be complete:

aws cloudformation wait stack-create-complete --stack-name mystack
aws cloudformation wait stack-update-complete --stack-name mystack

The Powershell implementation of this (Wait-CFNStack) uses one command to hit DescribeStacks API and wait for the status of your choice:

Wait-CFNStack -StackName mystack -Timeout 300 -Status CREATE_COMPLETE,UPDATE_COMPLETE

It's a bit of a strange situation though. CLI and Powershell SDK usage are flipped in the way they work.

With the CLI I can use aws cloudformation deploy (_creating and then executing a change set_) which will update a stack _or_ create a new stack with one command. Great for my CD that doesn't need to know if it is a first time deploy or an update. However I then have to determine if the stack was created OR updated to pick one of the two AWS CLI commands:

• CREATE_COMPLETE aws cloudformation wait stack-create-complete
• UPDATE_COMPLETE aws cloudformation wait stack-update-complete

However with PowerShell SDK I can also use the _DescribeStacks API_ to determine if the stack was created OR updated, however I get a _single cmdlet_ I can use Wait-CFNStack. Great for CD - HOWEVER I'm unable to have a single command to create or update my stack. I have to pick one _or_ the other.

• New-CFNStack - the _CreateStack API operation_
  
  
  
  • New-CFNStack -StackName "myStack" -TemplateBody "{TemplateContent}"
  
  
• Update-CFNStack -the _UpdateStack API operation_
  
  
  
  • Update-CFNStack -StackName "myStack" -TemplateBody "{TemplateContent}"
  
  

I've written PowerShell to detect if the stack exists, and do change sets - but this is onerous.

[revmischa]

This feature would be super useful for scripting test deployments of my cloudformation stack in CI

[weickmanna]

+1

[weickmanna]

+1

[mattBrzezinski]

Would love to see, any updates on this feature request?

[mdforbes500]

Any progress on this? It is really annoying to have to specify in jenkins whether you need an update or creation.

[jmclean-cnexus]

+1