Aws-sam-cli: Referencing Layer by parameter arn worked in Sam 0.19.0, after that it broke

[bmschwa]

Description

Had been using the most recent aws-sam-cli to build our projects. Version 0.21.0 & 0.20.0 have an issue when we refer to a layer param whereas version 0.19.0 did not.

Steps to reproduce

Using the following template with the command sam build --use-container --skip-pull-image --debug --template layer_ref_bug.yaml caused the error in versions 0.21, but not in 0.19.0

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Demo Template; can I reproduce this issue?

Parameters:
  aLayer:
    Description: Some Layer
    Type: String
    Default: arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:4
Resources:
  aFunctionResource:
    Type: AWS::Serverless::Function
    Properties:
      Timeout: 900
      CodeUri: .
      Handler: resource.handler
      Runtime: python2.7
      Layers:
        - Ref: aLayer

Observed

(from a code deploy build)

Traceback (most recent call last): 
  File "/usr/local/bin/sam", line 10, in <module> 
    sys.exit(cli()) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__ 
    return self.main(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main 
    rv = self.invoke(ctx) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke 
    return _process_result(sub_ctx.command.invoke(sub_ctx)) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke 
    return ctx.invoke(self.callback, **ctx.params) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke 
    return callback(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func 
    return ctx.invoke(f, obj, *args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke 
    return callback(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 94, in wrapped 
    raise exception  # pylint: disable=raising-bad-type 
  File "/usr/local/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 65, in wrapped 
    return_value = func(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/build/command.py", line 105, in cli 
    skip_pull_image, parameter_overrides, mode)  # pragma: no cover 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/build/command.py", line 138, in do_cli 
    mode=mode) as ctx: 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/build/build_context.py", line 65, in __enter__ 
    self._function_provider = SamFunctionProvider(self._template_dict, self._parameter_overrides) 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/local/lib/sam_function_provider.py", line 52, in __init__ 
    self.functions = self._extract_functions(self.resources) 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/local/lib/sam_function_provider.py", line 100, in _extract_functions 
    layers = SamFunctionProvider._parse_layer_info(resource_properties.get("Layers", []), resources) 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/local/lib/sam_function_provider.py", line 263, in _parse_layer_info 
    raise InvalidLayerReference() 
samcli.commands.local.lib.exceptions.InvalidLayerReference: Layer References need to be of type 'AWS::Serverless::LayerVersion' or 'AWS::Lambda::LayerVersion' 

Expected result

Working with old version, not with the new one

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10 (using powershell)
2. Python 3.7.4
   3a. sam --version: (when it worked): SAM CLI, version 0.19.0
   3b, sam --version: (when it didn't work): SAM CLI, version 0.21.0

[sriram-mv]

We had our integration tests pass: https://github.com/awslabs/aws-sam-cli/blob/develop/tests/integration/testdata/invoke/layers/layer-template.yml#L5 (But we over-ride our parameters with parameter over-rides), do you see this if you override parameters?

[daveykane]

When I try referencing by a param I get the following error:

Error: Layer References need to be of type 'AWS::Serverless::LayerVersion' or 'AWS::Lambda::LayerVersion'

[bmschwa]

We had our integration tests pass: https://github.com/awslabs/aws-sam-cli/blob/develop/tests/integration/testdata/invoke/layers/layer-template.yml#L5 (But we over-ride our parameters with parameter over-rides), do you see this if you override parameters?

Sorry for the delay. I drafted a response, and guess I never hit send. Even when I use the parameter overrides, I see the same thing.

I tried using !Sub & Fn::Sub, suggested in 1182, but also did not resolve the issue.

[jfuss]

@bmschwa I took the same template above and it built without any issues. The main difference looks to be you are on windows and I am on Mac, but I don't think OS should have anything to do with this. How have you installed SAM CLI? Do you have multiple versions installed by chance?

You are able to repro with the template above right (locally not in CodeBuild)? Want to make sure we have good repro steps, otherwise this is nearly impossible to know why a path is getting executed when it shouldn't.

[daveykane]

@jfuss I am seeing the same issue and I'm running on Mac, I have SAM CLI 0.21.0 installed via brew. I also had an issue when using !Sub, it didn't error but it also did not download the Layer.

I tried a lot of different options but could only get the Layer loading with SAM CLI 2 ways:

1. Hard code the Arn
2. Reference a local LayerVersion resource but the contentUri had to be a directory not a zip (which was an issue for me due to the size limits deployment the stack) and I believe that has been raised here https://github.com/awslabs/aws-sam-cli/issues/947.

[jfuss]

@daveykane Can you provide a template and any additional details needed to repro?

For the second point above, that is a completely different thing than this and would prefer to keep this issue scoped to the initial issue raised.

[daveykane]

@jfuss I have a very large template so I've put together an example which is consistently reproducing the error for me. This is consistently throws Error: Layer References need to be of type 'AWS::Serverless::LayerVersion' or 'AWS::Lambda::LayerVersion' for me when starting the template with sam local start-api

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Layer Test
Parameters:
  LayerArn:
    Default: 'arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:4'
    Description: Layer Arn
    Type: String
Resources:
  FunctionWithLayer:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      FunctionName: function-with-layer
      Handler: index.handler
      Runtime: nodejs10.x
      MemorySize: 512
      Timeout: 240
      Layers:
        - Ref: LayerArn
      VpcConfig:
        SecurityGroupIds:
        - Fn::ImportValue: !Sub ${VpcStackName}-AppSG
        SubnetIds:
        - Fn::ImportValue: !Sub ${VpcStackName}-AppSubnet1
        - Fn::ImportValue: !Sub ${VpcStackName}-AppSubnet2

Strangely if I commented out / remove the VpcConfig from this example the error disappears. Are you able to reproduce? If I remove the VpcConfig from my main template the error remains so I would guess there is some other setting, as well as the VpcConfig, in there which also causing an issue?

[jfuss]

@daveykane So what looks to be happening is the following (I stepped through this template with SAM CLI in debug mode):

• We run our 'resolve intrinsics' class on the template.
• Ref: LayerArn does get resolved to the default Parameter value
• We try to resolve the VpcConfig Property, which fails because we cannot resolve Fn::ImportValue.
• The FunctionWithLayer Resource remains unchanged
• We try and parse the values of Layers, which needs to be a arn and fail because we see a Ref instead.

We currently do not fail the resolving of intrinsics if we cannot resolve but I think leaving the whole resource unchanged is a bug. We should just set VpcConfig back to the original value and keep moving on, instead of the Resource. This also explains why our integ tests pass, because we don't have any additional values that are not resolvable.

Not sure how quick of a fix this is. There is lots of recursion I need to wrap my head around first.

[daveykane]

@jfuss that all makes sense, thanks for debugging. I have a workaround for now and will keep an eye out for the fix. Happy to do any testing you might need help with when a fix is ready.

[jfuss]

@daveykane I just submitted a PR for the fix (#1410). I added your example above into our smoke tests (which pass), so I am pretty confident this will resolve yours (and others) issue once and for all :). Feel free to pull and install the PR locally and give it a whirl.

[daveykane]

@jfuss I've tested the PR and I can confirm the updates fix the issue on both the example template above and my main project template.

The updates also have an added bonus of removing the noise of the unprocessed properties logs in terminal regarding properties (like Fn::ImportValue) which I am already aware SAM CLI can not process.

Thanks very much!

[danludwig]

@jfuss could you also please add #1407 and #1408 to the smoke tests? The first is giving us an error during sam build (!Ref), but the other is giving us an error during sam local start-api (!Sub)

[jfuss]

@danludwig I already have: https://github.com/awslabs/aws-sam-cli/pull/1410/files#diff-26e0eeea5414a362819f4d4a440c096b

The issue does not have relate to which intrinsic is used, so both issues you have are duplicates of this.

[sriram-mv]

Closing this issue. Latest Release v0.22.0 has the fix.

[wattry]

@daveykane So what looks to be happening is the following (I stepped through this template with SAM CLI in debug mode):

• We run our 'resolve intrinsics' class on the template.
• Ref: LayerArn does get resolved to the default Parameter value
• We try to resolve the VpcConfig Property, which fails because we cannot resolve Fn::ImportValue.
• The FunctionWithLayer Resource remains unchanged
• We try and parse the values of Layers, which needs to be a arn and fail because we see a Ref instead.

We currently do not fail the resolving of intrinsics if we cannot resolve but I think leaving the whole resource unchanged is a bug. We should just set VpcConfig back to the original value and keep moving on, instead of the Resource. This also explains why our integ tests pass, because we don't have any additional values that are not resolvable.

Not sure how quick of a fix this is. There is lots of recursion I need to wrap my head around first.

@jfuss Could you please elaborate on the statement: "which fails because we cannot resolve Fn::ImportValue". Someone has a similar question here. We are unable to pull layers with local testing or set VpcConfigs for local test. However, we are able to build and deploy. I am trying to use Fn:ImportValue to import from a Fn::Sub call as follows:

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: >-
  customer-portal

Parameters:
  AppNamespace:
    Type: String
  AppPrefix:
    Type: String
  AppEnvironment:
    Type: String
  CommonLayer:
    Type: String
  OracleDBClientLayer:
    Type: String

Resources:
  CustomerPortal:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: index.handler
      VpcConfig:
        SecurityGroupIds:
          - Fn::ImportValue:
              Fn::Sub '${AppPrefix}-${AppEnvironment}-LambdaSecurityGroupId'
        SubnetIds:
          - Fn::ImportValue:
              Fn::Sub '${AppPrefix}-${AppEnvironment}-PrivateSubnet1ID'
          - Fn::ImportValue:
              Fn::Sub '${AppPrefix}-${AppEnvironment}-PrivateSubnet2ID'
      Description: Customer Portal OracleDB API.
      Role: Fn::Sub arn:aws:iam::${AWS::AccountId}:role/${SecretLambdaRole}
      Layers:
        - Fn::Sub "${CommonLayer}"
        - Fn::Sub "${OracleDBClientLayer}"
      Events:
        dhcp:
          Type: Api
          Properties:
            Path: /dhcp/{proxy+}
            Method: any