Aws-sam-cli: sam deploy command option for different config file

[ericallam]

Describe your idea/feature/enhancement

I'd like to be able to have multiple different samconfig.toml files for deploying to different environments in my CI/CD process.

Proposal

Add an option flag to sam deploy that allows a user to specify a path to a samconfig.toml path, like so:

$ sam deploy --config-file ./samconfig-dev.toml

The guided mode should also be updated to ask for the path name, possible right after the Save arguments to samconfig.toml: input step.

[sanathkr]

This is a good feature request. We had made the explicit decision to skip this to support more automatic resolution mechanisms of the config file.

What do you think about an automatic resolution mechanism where you specify the env sam deploy --env dev and SAM CLI automatically chooses samconfig-<env>.toml?

[ericallam]

@sanathkr That would work for my use case 👍

[rowanu]

I like to have multiple stacks in my application repo (e.g. splitting front-end and back-end resources), so I don't think the --env approach works.

Really I just want to be able to support a custom configuration file name, so that I can specify where to load settings from, just like I would when specifying different template files - it feels like there is always going to be a 1:1 mapping between template file and config file, so why not support specifying them the same way?

[benkehoe]

I agree with @rowanu. I'd rather start with a simple generic mechanism, --config-file, that can enable all sorts of use cases. Then we can see what people use it for, and build an opinionated version (like --env) based on that.

[benkehoe]

Additionally, with the ability to use a separate configuration file should come the ability for "guided" mode to not actually deploy but write the values to a new config file location.

[benkehoe]

I need multiple configs for all of:

1. the same stack copied to multiple accounts
2. a slightly different stack in dev vs test vs prod
3. a set of different stacks together in the same place comprising an application

I can accomplish all of them with --config-file.

[jdonboch]

I would also be in favor of being able to define multiple profiles inside of a single samconfig.toml similar to AWS credentials file. There are already namespaces in there for [default], why not also support other profile names that you could choose to override the default.

Does this belong in a separate issue?

[darylounet]

It seems there is a hidden feature (or at least not yet documented) here with "--config-env" :

[default.validate.parameters]
template_file = "template-dev.yaml"

[prod.validate.parameters]
template_file = "template-prod.yaml"

$ sam validate
template-dev.yaml is a valid SAM Template

$ sam validate --config-env prod
template-prod.yaml is a valid SAM Template

:)

[jfuss]

@darylounet I would not rely on this at the moment. I think we initially had some design but pulled it out to start with the single config file. See https://github.com/awslabs/aws-sam-cli/issues/1615#issuecomment-563357713 above.

[darylounet]

@jfuss however I really think it's the good way to go. I have 5 environments, it's ridiculous to have 5 files, don't you think ?

[sbbowers]

@jfuss I'm not against adding a config-file option, but I'm totally in favor of the undocumented feature @darylounet points out. I think there are use cases for both, but the current --config-env is more elegant.

I will, however, point out that the label of "environment" is semantic to the application itself. What we are really talking about are stack labels.

I recommend implementing --config-file as requested (and not enforce a naming convention), and possibly making the existing --config-env cannon, except maybe call it --stack. Having both of these features will give developers a lot of versatility in how they address the many configuration they can have across many accounts.

[jfuss]

@darylounet We haven't made a decision on what is best here. There are defiantly case for both solutions.

@sbbowers We are planning on removing, as this was never meant to be available: https://github.com/awslabs/aws-sam-cli/pull/1769 Originally, we thought about this but didn't have enough to understand if this was the right choice. It was meant to be hidden in the code to be able to expand later not hidden and still available from the command line. This is why it is not documented, because it was never meant to be there.

[benkehoe]

It seems like the two cases are compatible. --config-file determines the file that configuration is loaded from. --config-env determines what elements to use from the config file that was loaded.

Separate quibble: it seems like because of the similarity with the AWS CLI config files, it should maybe be a "profile", not an "env". It doesn't _have_ to be a separate environment, it could just be a different bucket to use within the same environment, or even literally just using a different credentials profile.

[stojanovic]

I agree with @benkehoe. I would love to see this feature, and I think guided mode without an actual deployment would be helpful too.

Also, having a bit more docs explaining what are the allowed/expected params in this samconfig.toml file would be excellent.

[mariaines]

Has anyone come up with a workaround for this in the meantime? How do you manage pushing to dev / test environments? Having to use -g every time is annoying and fragile.

Edit: Oh, just found this:

If you are using SAM for a non-interactive deployment, it is possible to pass all required information as parameters. For example, for a continuous integration continuous delivery (CICD) pipeline:

from https://aws.amazon.com/blogs/compute/a-simpler-deployment-experience-with-aws-sam-cli/

I think I can work with that...

[darylounet]

It is surprising to remove a workaround possibility from your users before even proposing a solution ...
For those who wish to keep the workaround while using the latest sam cli versions you can revert the patch :

cd "$(dirname "$(which sam)" )/../lib/python/site-packages"
wget https://github.com/awslabs/aws-sam-cli/commit/1b7b4c9b11b5149b333d1dd20458293d5d4af6d4.patch
patch -R samcli/cli/cli_config_file.py 1b7b4c9b11b5149b333d1dd20458293d5d4af6d4.patch
rm 1b7b4c9b11b5149b333d1dd20458293d5d4af6d4.patch

[revolutionisme]

This option with different config file would be definitely good to have!

[ronaldour]

What is the issue with the --config-env ? personally I believe this behavior is perfect for this use case

[brysontyrrell]

I'm going to throw my support behind @benkehoe for having a file path option and not just setting an env.

[kirnberger1980]

@benkehoe We also would like to have this functionality. As we are working in a Landing Zone where we do have staging accounts (DEV, TEST, PROD) for each product, we would like to deploy a SAM application via CICD with a separate samconfig.toml for each account.

[benkehoe]

To be clear, I think SAM CLI should support both configuring the "env" that selects sets of content from within the samconfig.toml file, and configuring the location of the samconfig.toml file itself. One possibility is that, like the AWS CLI, the config file location would only be configurable through an environment variable.

[kirnberger1980]

From our personal perspective we would like to configure the location of samconfig.toml , since we would use a separate git repo where we want to place the samconfig.toml of the environment (dev1, dev2). So the support to configure an env section inside of the samconfig.toml is not neccessary in our use case.

[milesgranger]

It seems one of the points of having a .toml config file is to facilitate different settings for given build. For example Cargo.toml profiles allows Rust users to specify how to compile given certain profiles. Likewise, Poetry does similar with normal, dev, and extras dependencies.

Therefore, IMO, it appears to be an unexpected choice to have multiple .toml files for different environments. However, I can see the --config-file option in the cases such as @kirnberger1980 experiences, having config files in either unexpected locations or naming conventions as dictated by the project.

At this point, I just want _something_.

[awsjeffg]

This feature has now been released. Apologies, the SAM cli docs will not be available for this new functionality until tomorrow ( 9/24 ).