Serverless-application-model: SAM policy templates for VPC don't seem to reliably work?

[bluepeter]

I have a Lambda function defined with VPC access as follows:

  get:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      CodeUri:
        Bucket:
          Fn::Sub: bigonion-deployment-helper-${AWS::AccountId}-${AWS::Region}
        Key: BigOnion_dummyFunction.zip
      FunctionName: BigOnion_get
      Policies:
        - VPCAccessPolicy: {}
        - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
      VpcConfig:
        SubnetIds:
          - Ref: VPCsubnet1
        SecurityGroupIds:
          - Ref: VPCsecuritygroup

The stack fails. Looking at the log, I can see that the Role is created with the VPC Access Policy:

        {
            "StackId": "arn:aws:cloudformation:us-east-1:xxx:stack/BigOnion/082d9e10-de40-11e7-9274-503acac41efd",
            "EventId": "getRole-UPDATE_COMPLETE-2017-12-21T03:01:29.349Z",
            "StackName": "BigOnion",
            "LogicalResourceId": "getRole",
            "PhysicalResourceId": "BigOnion-getRole-1WISQQA2H8J9K",
            "ResourceType": "AWS::IAM::Role",
            "Timestamp": "2017-12-21T03:01:29.349Z",
            "ResourceStatus": "UPDATE_COMPLETE",
            "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole\"],\"Policies\":[{\"PolicyName\":\"getRolePolicy0\",\"PolicyDocument\":{\"Statement\":[{\"Action\":[\"ec2:CreateNetworkInterface\",\"ec2:DeleteNetworkInterface\",\"ec2:DescribeNetworkInterfaces\",\"ec2:DetachNetworkInterface\"],\"Resource\":\"arn:aws:ec2:us-east-1:xxx:network-interface/*\",\"Effect\":\"Allow\"}]}}],\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":[\"sts:AssumeRole\"],\"Effect\":\"Allow\",\"Principal\":{\"Service\":[\"lambda.amazonaws.com\"]}}]}}\n"
        },

But then later the Lambda function fails with The provided execution role does not have permissions to call CreateNetworkInterface on EC2

        {
            "StackId": "arn:aws:cloudformation:us-east-1:xxx:stack/BigOnion/082d9e10-de40-11e7-9274-503acac41efd",
            "EventId": "get-UPDATE_FAILED-2017-12-21T03:02:09.752Z",
            "StackName": "BigOnion",
            "LogicalResourceId": "get",
            "PhysicalResourceId": "BigOnion_get",
            "ResourceType": "AWS::Lambda::Function",
            "Timestamp": "2017-12-21T03:02:09.752Z",
            "ResourceStatus": "UPDATE_FAILED",
            "ResourceStatusReason": "The provided execution role does not have permissions to call CreateNetworkInterface on EC2",
            "ResourceProperties": "{\"Role\":\"arn:aws:iam::xxx:role/BigOnion-getRole-1WISQQA2H8J9K\",\"FunctionName\":\"BigOnion_get\",\"Runtime\":\"nodejs6.10\",\"VpcConfig\":{\"SubnetIds\":[\"subnet-16a91239\"],\"SecurityGroupIds\":[\"sg-46534333\"]},\"Handler\":\"index.handler\",\"Code\":{\"S3Bucket\":\"bigonion-deployment-helper-xxx-us-east-1\",\"S3Key\":\"BigOnion_dummyFunction.zip\"},\"Tags\":[{\"Value\":\"SAM\",\"Key\":\"lambda:createdBy\"}]}\n"
        },

Any clues on how to get SAM policy templates for VPC working properly?

[george-ayris]

Just hit a similar issue and I got it working by:

1. Removing the VpcConfig from my template, but keeping the Policies
2. Deploying it
3. Adding the VpcConfig back in
4. Performing another deploy

Not sure why this worked, potentially a time lag in the IAM role being updated?

Hope this works and Merry Christmas!

[bluepeter]

Thanks, I may give that a try (I had previously tried all sorts of combinations using DependsOn with no luck). I got it to work by defining a Role outside of CF and explicitly granting it to each function in CF. Not ideal. (P.S., Don't get me started on DynamoDB DAX... what an absolute PITA that it lives in VPC.)

[horaceli]

I just have

      Policies: AWSLambdaVPCAccessExecutionRole

AWSLambdaVPCAccessExecutionRole has everything in AWSLambdaBasicExecutionRole plus:

• ec2:CreateNetworkInterface
• ec2:DeleteNetworkInterface
• ec2:DescribeNetworkInterfaces

VPCAccessPolicy has just the above ec2 actions + ec2:DetachNetworkInterface - it doesn't have the actions in AWSLambdaBasicExecutionRole (cloudwatch logging), but the transform automatically adds a separate AWSLambdaBasicExecutionRole so it doesn't need it explicitly.

[bigwheel]

I have same issue. Im new at SAM but this problem seems like it was included from latest or recent version.

[sanathkr]

Yeah this is definitely a propagation delay problem. If you are running into this too often, you can create your own IAM Role resource, and add a timer with a WaitCondition and chain the IAM Role, Wait Condition and Lambda Function together using DependsOn. This sucks I know, but if you are consistently running into this problem, this could be a workaround.

Btw, which region are you deploying this to?

[bluepeter]

As you mentioned, I was able to get around this issue by creating a IAM Role manually, and applying it to the Lambda function in CF template. Good idea on the WaitCondition.

For our use case, as the AWS DAX Node.js SDK turned out to not be reliable (persistent socket pooling issues), we no longer need VPC/NAT, and are fortunately able to avoid the many issues with Lambdas in VPC.

In terms of regions, prior to our pivot away from VPC, we deployed in multiple regions. Strangely, some regions always worked (us-east-1, us-west-2, eu-west-1), but others, despite having all of the resources we used, always failed with vague "internal server errors," including us-east-2 and eu-west-2.

[sanathkr]

Makes sense. The hard part is SAM can't do much here. This is coming from CloudFormation, which is the execution engine of the template, not waiting long enough for IAM resources to be propagated.

Closing this Issue

[Drag0]

You can create a IAM role in the SAM template using CloudFormation definition and add inline VPC policies to it. Then you can use that in the lambda and deploy all in one go.

  FunctionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      Policies:
        - PolicyName: FunctionPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'ec2:CreateNetworkInterface'
                  - 'ec2:DescribeNetworkInterfaces'
                  - 'ec2:DeleteNetworkInterface'
                Resource: '*'

[clakech]

hi, maybe if VPCAccessPolicy does not work by default and we have to use our own role, we could remove VPCAccessPolicy from the SAM policies.

BTW same issue here

BTW 2: As a workaround, no need of WaitCondition, just create a role with needed policies, add this role to the function, no dependson needed, cloudformation understand the dependency and the "The provided execution role does not have permissions to call CreateNetworkInterface on EC2" goes away

[justabuzz]

OK, so for those who use AWS CodeStar to generate a CI/CD pipeline for an API Gateway+Lambda application and stuck with this error when trying to set the lambda in a VPC - this is a problem in the lambda's execution role's Permission Boundary. The boundary is auto generated as part of CodeStar provisioning the necessary resources. This policy doesn't have the necessary permissions for allowing the CreateNetworkInterface action.

To fix I have manually updated that policy - I added the following statement:

{
    "Effect": "Allow",
    "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
    ],
    "Resource": "*"
}

I hope this helps someone! I spent hours figuring this one out!

[hoang-innomizetech]

We fixed this error by using the above @justabuzz . Thank you!