Aws-sam-cli: Nested stacks are difficult to build and deploy for several reasons

[ajhool]

Description:
There are a few closely-related issues. I'll start by describing our ideal deployment strategy for SAM and then explain the current pain points. Some of these pain points are when dealing with nested stacks. Having typed these out, it might make sense to split this into three issues although they are closely related:

1. "No Build Hook pain point"
2. "Nested Stack Packaging pain point"
3. "No Archive Upload Endpoint pain point"

Ideal flow:

1. Write a build command for the lambda functions.
2. Write the child and parent SAM templates.
3. Run sam build or sam package on the top-level SAM template. Recursively build and/or the child SAM templates.
4. Run sam deploy on the top-level SAM template. This command would deploy the results of Step 3.

Ideally, Step 3 and 4 could be combined so only one deploy command is needed.

No Build Hook pain point Currently, there is no build hook, so we have to build first and then run deploy. It's just an extra step but requires extra thought, especially when working with nested stacks.

Nested Stack Packaging pain point The second pain point with nested stacks is needing to recursively call package on each nested stack. This one is brutal! It forces us to keep track of where each stack is and write a custom script in each Nested stack to package it before the parent stack calls package -- then, even worse -- we need to make sure that the parent stack is pointing at the output of the child stack! This feels really clunky and makes it possible to use a stale nested child stack. That is the "Nested Stack Packaging pain point"

No Archive Endpoint pain point Finally, the third pain point is using S3 buckets for archive upload. It just feels weird. Why isn't there a cloudformation upload endpoint where we can send a zip file with all of the stacks (which would use relative references for each other). The Cloudformation backend already manages everything, why doesn't it manage artifact storage and relative template reference interpretation, too? You could choose to reference templates directly from S3 for reusability's sake, but you wouldn't be forced to.

[jlhood]

@ajhool Thanks for reporting these issues! These are SAM CLI related so I'm going to transfer this issue to that GitHub repo. However, regarding this:

The second painpoint with nested stacks is needing to recursively call package on each nested stack. This one is brutal!

sam package supports recursive packaging. If you have AWS::CloudFormation::Stack or AWS::Serverless::Application resources in your template that reference local template files, when you run sam package on the top-level template, SAM CLI should recursively package the referenced local template files and upload them to S3.

[daroot]

I just ran face first into several of these issues, so I'll add my use cases. It's soooo close right now that it's actually a bit more maddening than if it flat didn't work at all.

@jlhood my biggest issue is that SAM supports recursive packaging, but does not support recursive building beforehand. #1470 is related to this, as are #1443 and #1213

My scenario is as follows: I have a set of shared resource sub-stacks (one storage.yaml for dynamo tables/s3 buckets, one lambdas.yaml for SAM built lambda functions, etc). Some of these are separate stacks because their templates are meant to be modular and used elsewhere, some because they're on different lifecycles and all the various AWS best practices suggest that both cases should be separated. The "master" stack.yaml for this application contains AWS::CloudFormation::Stack and AWS::Serverless::Application references to the various resources stacks, plus some EventBus schedule Rules and a Step Function that uses the resources out of the sub-stacks. All of this resides in a single repo.

If I sam build on the master stack.yaml, it does not build on the included AWS::Serverless::Application included in stack.yaml. My build directory contains a single template.yaml and nothing else. If I sam package on this template, it _will_ package up the source for the lambdas.yaml, but not having done a build step for it, it lacks the appropriate additional libraries it should have included and the resulting deployment won't work as expected.

Doing a manual sam build and sam package on the lambdas.yaml works fine, but a subsequent sam build on stack.yaml wipes out those local copies and the resulting template does not correctly refer to them.

Even just doing sam build on lambdas.yaml before doing a build and package on stack.yaml does not work as some of the other issues suggest, because the second build command wipes the target directory before running.

So right now I'm left with either using CF Exports and separate deployments to move values between sub stacks or writing some custom scripting to upload the nested stacks resources with proper sam builds and then rewriting the including stacks to point to the correct templates.

What I would like:

• One repo containing multiple templates (some modular ones imported from elsewhere, some local ones with long lived resources like DynamoDB tables, S3 Buckets, or VPCs, some ancillary ones like a CI/CD pipeline and CodeCommit Repo).
• Allow templates at either the top or leaf levels to contain SAM resources like AWS::Serverless::Function.
• sam build on a the top level stack recursively builds any nested local stacks, outputting appropriate copies of the templates into the build directory, and doing dependency resolution for Lambdas, handling Lambda LayerVersions, etc.
• sam package handles uploading all nested templates and build resources into the appropriate S3 bucket with appropriate resources resolved and outputting a single master template which references these uploaded nested templates.
• Being able to run CloudFormation on the output master template via standard CI/CD process using CodePipeline. Using Create/Execute ChangeSet is fine in this case.

[kornicameister]

Generally bumping.
Our use case was to split the application into smaller piece so that we can deploy each sub-application separately in CI/CD for CI purposed.

You know like...we deploy backend.ypl.yml and we run set of integration tests. We can do the same for UI if UI was in fact modified. Problem is that nested applications or rather their dependencies (defined in given language) are not picked up.

However, might be that a way to work around it is to push all dependencies into a layer. Layers we have are complicated enough to say that building one simple one wouldn't exactly hurt ;-)

[rsgaikwad]

Trying to apply netsted stack on aws-serverless-shopping-cart , facing same issue. It creates child stacks, but when testing functionality, lambda is unable to import module because build directory just contains parent template file but not includes lambda and required dependent packages. Is anyone found recommended solution (without any custom scripts)?