Aws-sam-cli: `sam build --parameter-overrides` fails to override

[diablodale]

Description

sam build --parameter-overrides fails to override SAM template yaml Parameters

Setup

• Ubuntu 18.04.3 x86_64
• Python 3.7.3
• aws-cli/1.16.265 Python/3.6.8 Linux/5.0.0-32-generic botocore/1.13.1
• SAM CLI, version 0.23.0
• empty requirements.txt
• arbitrary app.py
• below parampoof.yaml

parampoof.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: |
  Repo of param override failing to override

Parameters:
  GitTag:
    Description: description field on lambda version
    Type: String
    Default: defaultgittag

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      AutoPublishAlias: test
      VersionDescription: !Ref GitTag
      Handler: app.lambda_handler
      Runtime: python3.7
      Timeout: 10

Steps to reproduce

sam build \
    --template ./parampoof.yaml \
    --manifest requirements.txt \
    --parameter-overrides "ParameterKey=GitTag,ParameterValue=aabbccdd"
grep aabbccdd .aws-sam/build/template.yaml

Observed result

First...

Building resource 'MyFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Then...

Nothing. grep finds no value of aabbccdd in the file

Expected result

grep should find the value aabbccdd in the file. Likely in Parameters: section by overriding/replacing the default value defaultgittag. This is needed because the next steps sam package and sam deploy in the workflow depend on this .aws-sam/build/template.yaml transformed output of sam build.

Debug output of sam build

Using SAM Template at **redacted**/parampoof.yaml
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {'GitTag': 'defaultgittag'}
1 resources found in the template
Found Serverless function with name='MyFunction' and CodeUri='.'
Building resource 'MyFunction'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r **redacted**/requirements.txt --dest /tmp/tmpk1rvadh4
Full dependency closure: set()
initial compatible: set()
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: set()
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): set()
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): set()
Final compatible: set()
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 523, 'exitReason': 'success', 'exitCode': 0, 'requestId': '9f3a7fc1-3397-48a2-afe0-22848b8743b4', 'installationId': 'c648e689-e57c-4a3c-a26f-ca6d1fd13041', 'sessionId': 'd5aced2b-51ec-4288-aab8-a47317da031d', 'executionEnvironment': 'CLI', 'pyversion': '3.7.3', 'samcliVersion': '0.23.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Workaround

Don't use sam build --parameter-overrides. Instead use sed to search/replace the base template.yaml for a known value to replace/override.

Related

https://github.com/awslabs/aws-sam-cli/issues/572

[jfuss]

@diablodale I think you have a misunderstanding of the output of sam build. Passing --parameter-overrides into sam build is purely for the process, we will not replace those in the template we produce. Customers tend to use --parameter-overrides in build as an escape hatch to some things, like to get around a layer arn check we have. What you want to do is pass --parameter-overrides during sam deploy.

The template build produces will not write a mutated version of the passed in template, this is by design.

[jfuss]

Correction: We do mutate but this is to update the CodeUri part of the template to point at the new built code.

[diablodale]

Quite possible as I'm guessing in areas of ambiguity (to me).

What is a specific example (specific key and specific value) of using sam build --parameter-overrides? If it is not overriding the Parameters: in the Cloudformation (aka SAM) template, then what is it overriding? I ask because sam build --help writes "...CloudFormation parameter overrides".

I saw that some part of sam deploy is aws cloudformation deploy and I do see the parameter-override on it. However, this does not help me with local testing. I also see parameter-override on sam local invoke but in that scenario it is unknown to me what param it would override that is distinct from some mutation done by sam build --parameter-overrides

Perhaps this is all ambiguity in the docs and not a feature bug. If more clarify can be brought to --parameter-overrides in each sam build, sam deploy, sam local invoke and how they differ yet have the same parameter and usage...then issues like this one can be reduced. 🤔🙂

[jfuss]

One example would be passing in Layer ARNs through Parameters. If you Ref: LayerArnParameter, SAM CLI fails with "Not valid Layer Arn". To get around this, you can override the parameter at command execution time. Or if you set CodeUri/Code properties through parameters. In that case, we wouldn't know what to build.

It does override the value of the Parameter, but that does not mean we update the "Default" value or replace that parameter in the outputted template. The concept comes from CloudFormation, where you can set defaults in the template but override them at execution time. CloudFormation does not alter the template either just the 'in memory' version of the resource graph.

[diablodale]

Ooo, that's super-subtle. What just now clicked for me is 'in memory'.
Personally, I think the 'in-memory' of sam build should include mutation of Parameters:. Just like it does for the CodeUri, and other params (I've already see sam dramatically reformats the top-level Description).

Production CloudFormation mutates things 'on-disk'. Yes, it can receive a param that overrides the CloudFormation template default, but when it writes 'on-disk' to an EC2 instance, CloudWatch rule, or any other created resource that references that overridden param...that's on-disk. It's not just in-memory. And those on-disk mutations persist and may further propagate if the created resources do creation of their own.

Since the sam workflow is a series of steps (at minimum package and deploy are required because of the S3 package location mutation, build for safer dependency management), I suggest that the mutations of sam includes param-overrides of the top-level Parameters:. I acknowledge that it is a bit odd to mutate the default value. So perhaps an approach like...

• Spec new param Parameters.MyParam.RuntimeValue
• Update sam code so that if that RuntimeValue is present in the template, that it uses that value instead of the Parameters.MyParam.Default value.
• Update sam code to add Parameters.MyParam.RuntimeValue for any --parameter-overrides provided on the cli.
• The above allows sam build, sam local invoke, or sam deploy with --parameter-overrides "ParameterKey=MyParam,ParameterValue=value123". The first two commands are the updated sam code that writes the new RuntimeValue param and the 3rd you tell me already works with its alias to aws cloudformation deploy.

That brings the behavior of --parameter-overrides more in alignment and behavior for local and AWS deployments. This allows for scenarios like:

1. Environment variables of a lambda function based on Parameters: will now work both locally and on AWS. Without mutating the Parameters: with a value passed with --parameter-overrides, the local testing docker container won't receive the overridden value -> can't adequately test.
2. Dynamic git commit tagging of a build->package->deploy using the Serverless::Function.Prop.VersionDescription as a reference to an overriden param. It is not uncommon for a running service to know its own git commit.

[jfuss]

I don't think mutating and writing it to disk is the right thing to do for build. --parameter-overrides is more of an escape hatch to get the template into a state that the CLI can understand than something you should be using regularly. Ideally, build doesn't need for customers to use --parameter-overrides but right now there are cases for it.

Doing what you are suggesting, isn't ideal for how many customers setup pipelines. The recommended/best practice is to build -> package to produce a single template with all artifacts, then deploy that to regions/accounts. In these cases, you do not want build to set defaults because your template becomes very narrow. Yes, you could then do many builds but the CloudFormation way to do this is through Parameters and then overriding them at deploy time.

The path forward here would be to only use --parameter-overrides if you need to on build. For invoke, pass --parameter-overrides to get the template to a state you want to test. This is the current pattern which is all modeled on how CloudFormation APIs work.

[diablodale]

I verified --parameter-overrides works as expected on sam local invoke. That leads to the possibility to provide the same --parameter-overrides to both sam local invoke and sam deploy which should lead to the same expected behavior for both local and AWS infrastructure testing. ✔

What remains, for me, is the case for when metadata (like the git commit hash) is desired on the sam workflow (build, invoke, package, deploy). This type of metadata is stricted associated w/ code and not a dynamic runtime options. The typical place to label such is at a build or packaging phase.

That written, there is a technical workaround that can achieve most. I can pass my git hash with --parameter-overrides on the sam local invoke and sam deploy. Architecturally, that feels an odd place to do it. However, in practice, it will label the resources created by CloudFront, as well as the CloudFront stack itself, with my git hash...achieving my goal.

Unless you have comments on the above, I consider sam working as expected and this issue resolved.

[sanathkr]

@diablodale You have a really good point. sam build should capture the flags provided earlier and pass it along to the next commands. Your mental model makes sense and in fact is more powerful IMO.

I think #1503 is going to enable this capability. Please take a look at the design and provide comments. You have an interesting perspective and I think we can all benefit from it.

Closing this Issue because sam is working as expected

[diablodale]

I'm light on time this week. If not this, then I already booked some time on Monday 18 Nov to look at the linked doc.

[diablodale]

feedback in https://github.com/awslabs/aws-sam-cli/pull/1546

[lmayorga1980]

any workaround for https://github.com/awslabs/aws-sam-cli/issues/1948

[rpstreef]

This feature is extremely confusing, my expectations were that sam build is using the parameter overrides flag to create an output artifact that I can use for execution or deployment purposes.

[shane-js]

Note to those who do use CodePipeline's built in CloudFormation deploy option who were tripped up by this: Look in the advanced options in the Deploy stage. There is a JSON input field for parameter overrides that does what you want it to (if you don't resort to scripting the deploy).