Containers-roadmap: [ECS] [CodeDeploy] [CloudFormation]: CloudFormation support for BLUE/GREEN deployments on ECS

[ilyasotkov]

The feature was announced in November: https://www.youtube.com/watch?v=01ewawuL-IY

For blue/green deployments, AWS CloudFormation supports deployments on AWS Lambda compute platforms only.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codedeploy-deploymentgroup.html

Not exactly sure whether AWS::ECS::Service.DeploymentConfiguration would be affected as well.

[ghost]

would be good to get more info on this and what are the plans.

[ngamradt-turner]

We would like to start experimenting with this feature, but without CloudFormation support, it is not practical for us to use.

[ghost]

We would like to start experimenting with this feature, but without CloudFormation support, it is not practical for us to use.

we have been promised that CFN will be getting 1st class treatment going forward ... we shall see :)

[deleugpn]

@steven-cuthill-otm who promised that?

[ghost]

@steven-cuthill-otm who promised that?
jeff barr, have a read ...
https://www.reddit.com/r/aws/comments/avx449/a_quick_cloudformation_update/

[deleugpn]

I read that thread and didn't see any moment the promise to get CloudFormation support for resources as 1st class treatment. The wording says they'll open their roadmap to allow us to tell what we would like to prioritize, which for me does not translate as getting 1st class treatment.

[sarojgharat]

Any update on this ?

[mmoulton]

I'm also very interested in this capability. Specifically I'd like to use it with CDK (https://github.com/awslabs/aws-cdk/issues/2056)

[angusfz]

Waiting for CFN support

[nitzanyemal123]

+1 For CF support

[mrbueno92]

+1 For CF support

[sarojgharat]

Any update on this?

On Wed, May 8, 2019, 15:07 Mauricio Bueno [e-mail] wrote:

+1 For CF support

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/containers-roadmap/issues/130#issuecomment-490478153,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG7GOJFC4I55K4LUQYWREI3PULF7HANCNFSM4GSAGLIQ
.

[andrewhiles]

Do we know if there's been any movement with this?

[JoeAlamo]

This would be a killer feature to have.

[paddie]

Yep, blocked by this as well. I think that this article summarises pretty clearly why this is not a trivial thing for CF to handle:

It turns out that there’s actually a very good reason why this is not a supported configuration. CodeDeploy is going to be referring to an Auto Scaling Group, and it’s likely that you have defined that ASG in your CloudFormation template, and used !Ref to link the deployment group to it.

The problem comes when you do your first blue-green deployment. CodeDeploy does a blue-green deployment by cloning the ASG to make a new one with the same parameters, but whose instances run the newer version of your application. Once the new version is deployed and healthy, it winds down the old ASG. Then, it updates the deployment group configuration to point to the new ASG, and deletes the old ASG.

This is fine until you go back to your CloudFormation template and need to make a change that touches the ASG. CloudFormation will try to update the ASG that it created, only to find it doesn’t exist any more. Boom, template update failure and rollback.

[ScOut3R]

In regards to ECS blue/green deployments CodeDeploy uses two already existing Target Groups to switch traffic back and forth. While the referenced article holds correct information it does not relate to ECS blue/green deployments but EC2 based ones.
I have a service where I deployed almost everything via CloudFormation, including the two Target Groups used for blue/green deployments. I provisioned the CodeDeploy side with Terraform and everything works quite neatly. Since CodeDeploy does not destroy/clone/recreate the Target Groups both parties are happy.
Apart from capacity concerns as a consumer I don't see any blockers for ECS support.

[okram999]

@ScOut3R - how do the service know that the deployment type is bluegreen, if you configure them using cloudformation. The option to specify deployment type as bluegreen is what where the challenge starts. no?

[ScOut3R]

@okram999 The blue/green CodeDeploy configuration is deployed by Terraform. I pass in as input variables the ECS Service and the two Target Groups (which are built by CloudForamtion) to Terraform which builds the CodeDeploy and CodePipeline configuration.

[noahxp]

+1 For CF support

[pferretti]

+1 For CF support

[ngamradt-turner]

Is there an update on this feature?

[mwarkentin]

Probably helpful to add your votes and follow on the issues on the cloudformation roadmap:

• https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/56
• https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/37

[dslove]

I need this too.

[ramanasatyavolu]

Is there a feature provided in CloudFormation to launch blue green deployment through ECS

[surajtikoo]

Waiting for a long time .Is any time line when it be be available in cloudFormation to launch blue green deployment through ecs fargate?

[mwarkentin]

Looks like this just showed up: https://docs.aws.amazon.com/en_pv/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-service-deploymentcontroller.html

Haven’t had a chance to try it.

[surajtikoo]

@mwarkentin Thanks for your Update, but i think with the new feature of ALB which support multiple target group with single Listeners is quite amazing. We can use weighted routing which is much easier and faster.
You are quite late with this feature. :)

https://aws.amazon.com/blogs/aws/new-advanced-request-routing-for-aws-application-load-balancers/

[deleugpn]

I don't see how the ALB weighted request fit into CI/CD strategy. I want a limited time for weighted request instead of supporting 2 versions at all times.

[surajtikoo]

@deleugpn. You can still handle this with your custom pipeline in build step to route the traffic, but i agree with your point if we need to test before forwarding traffic to your live environment then multiple versions support is quite useful.

[ngamradt-turner]

Since this is still in the "Coming Soon" column, I am guessing that this feature will not come out until after re:Invent, or will something be announced yet during re:Invent?

[deleugpn]

I think by this point it's crystal clear that AWS do not disclose release date until the day of the release.

[cb-manideep]

+1 Need this.

[Lasim]

+1

[bubeamos]

Hi @coultn I see that there a property value CODE_DEPLOY in the "DeploymentController:" property docs here does this mean that this is now available ? And this issue need to be updated ?

[mwarkentin]

Also hoping that this will work with the new CodeDeploy canary deployment strategies!

https://github.com/aws/containers-roadmap/issues/229

[richardmaltais]

@paddie For that specific matter, using Blue-Green with CodeDeploy on EC2 instances, we used a custom setup to keep the original ASG, then CodeDeploy-generated ASGs are switched over as usual, drained when needed and the updated LaunchConfiguration is also "copied" over through a Lambda, so that CloudFormation doesn't freak out when comes the time to update it. However, it comes with a few limitations:

• not possible to change the capacity of the ASG (the CodeDeploy-generated one)
• we have to have set the capacity to 0 when applying changes to the LaunchConfiguration and the ASG. Then, the Lambdas do the rest

However, we didn't try to experiment with resource import for the CodeDeploy generated ASG.

[tuttlem]

+1 For CF support

[KiamarzFallahi]

We are excited to announce you can now use AWS CloudFormation to perform Amazon ECS blue/green and canary deployments through AWS CodeDeploy. Blue/green deployments are a safe deployment strategy provided by AWS CodeDeploy for minimizing interruptions caused by changing application versions.

To learn more visit our announcement and the user guide.

[sethstone]

I've been tracking this ticket for a few months and I was excited to see it close, however, the solution provided did not solve the problem I expected. It seems that the new capability allows one to use Cloudformation as the deployment controller - in particular you would be monitoring the deployment in Cloudformation rather than the CodeDeploy console and release changes through stack updates. This may be exactly what others were looking for, but I was expecting was to have the ability to create a deployment group (with CF) using the ECS BlueGreen deployment type and be able to manage the deployment process (once created by CF) in the CodeDeploy console.

Here's an example that I hoped would work after this change

 DeploymentGroup:
    Type: AWS::CodeDeploy::DeploymentGroup
    Properties: 
      ApplicationName: !Ref CodeDeployApplication
      AutoRollbackConfiguration: 
        Enabled: true
        Events:
          - DEPLOYMENT_FAILURE
      BlueGreenDeploymentConfiguration: 
        DeploymentReadyOption: 
          ActionOnTimeout: CONTINUE_DEPLOYMENT
          WaitTimeInMinutes: 0
        TerminateBlueInstancesOnDeploymentSuccess:
          Action: TERMINATE
          TerminationWaitTimeInMinutes: 5
      DeploymentConfigName: CodeDeployDefault.ECSAllAtOnce
      DeploymentGroupName: !Ref CodeDeployDeploymentGroupName
      DeploymentStyle: 
        DeploymentOption: WITH_TRAFFIC_CONTROL
        DeploymentType: BLUE_GREEN
      LoadBalancerInfo: 
        TargetGroupPairInfoList: 
          TargetGroups:
            - Name: !Sub ${ProjectPrefix}-tg1
            - Name: !Sub ${ProjectPrefix}-tg2
          ProdTrafficRoute: 
            ListenerArns: !Ref ALBProductionListener
      ServiceRoleArn: !GetAtt CodeDeployServiceRole.Arn
      ECSServices:
        - ServiceName: !Sub ${ProjectPrefix}-service
        - ClusterName: !Sub ${ProjectPrefix}-ecs-fargate-cluster

Based on the Youtube video I thought this is what we were trying to accomplish because it shows several Deploy stages with ECS Blue/Green Deploy (not a Cloudformation deploy).

Could someone let me know if this is an issue that is being worked on on the roadmap and/or am I doing something wrong to expect this to work? (I based this CF template on a working CLI command that I have)

Thanks!

PS: The error I get is "Encountered unsupported property BlueGreenDeploymentConfiguration"

[KiamarzFallahi]

@sethstone You are correct. This feature does not support creating and configuring a CodeDeploy deployment group to naively perform blue/green deployments for ECS. That is a coverage gap being tracked: https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/37

[clareliguori]

That issue was also closed:
https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/37#issuecomment-632784182

@sethstone A new issue was created on the CloudFormation roadmap here requesting coverage in AWS::CodeDeploy::DeploymentGroup for ECS blue-green, it would be great for you to comment there with your example template snippet. Thanks!
https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/483

[mwarkentin]

There are a couple of limitations which mean that this feature won't work with our ECS platform (convox):

• Declaring output values or importing values from other stacks is not currently supported for templates defining blue/green ECS deployments.
• You cannot use the AWS::CodeDeploy::BlueGreen hook in a template that includes nested stack resources.
• You cannot use the AWS::CodeDeploy::BlueGreen hook in a nested stack.

Are there plans / work in progress to handle these use cases? Or for more complex use cases like this would it be meant to be handled by the CodeDeploy DeploymentGroup support described above?

[Lasim]

Thanks for the analysis @mwarkentin.

it's meant to me that there is no sense yet to use this feature if these functions are not implemented.

[vinay-nadig-0042]

I see that the issue has been closed since the Cloudformation support has been announced. But, as @mwarketin pointed it out, the limitations are quite crippling. Is there a separate issue where these limitations and their fixes are tracked?

[zzenonn]

I see that the issue has been closed since the Cloudformation support has been announced. But, as @mwarketin pointed it out, the limitations are quite crippling. Is there a separate issue where these limitations and their fixes are tracked?

I opened this issue on the CloudFormation roadmap

aws-cloudformation/aws-cloudformation-coverage-roadmap#483