Copilot-cli: Pipeline DeployTo-prod failing with: Template format error: Unresolved resource dependencies [AddonsStack]

[fuji97]

Issue

I have an ECS cluster with two Load Balanced Web Services and two environments, test and prod, with a pipeline generated from copilot that deploys the application on the two environments, first on test and after on prod. When I deploy the service with two addons directly from the copilot CLI, the deploy is successful, but when I try to deploy using the pipeline, the deploy on test is successful, but the deploy on prod fails with the error:
Template format error: Unresolved resource dependencies [AddonsStack] in the Resources block of the template (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: xxx).
Only by removing all the addons seems to fix the error and let the pipeline deploy to prod successfully.

Environment

• ECS Cluster on Fargate
• Copilot CLI v1.2.0
• Node.js v14.x
• manifest.yml

name: backend
type: Load Balanced Web Service

image:
  build: backend/Dockerfile
  port: 80

http:
  path: 'api'
  healthcheck: '/api/v1/healthcheck'

cpu: 256
memory: 512
count: 2

• pipeline.yml

name: pipeline-name

version: 1

source:
  provider: GitHub
  properties:
    access_token_secret: github-token-aws-ecs-node-demo-aws-ecs-node-demo
    branch: main
    repository: https://github.com/user/repo

stages:
    - name: test
      requires_approval: true
      test_commands:
        - echo 'running tests'
        - cd backend/
        - npm ci
        - npm run test
        - echo 'test completed'
    - name: prod
      requires_approval: true

• addons/dynamodb.yml

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.
Resources:
  DynamoDbAccessPolicy:
    Metadata:
      'aws:copilot:description': 'An IAM ManagedPolicy for your service to access the demo-ecs db'
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Grants CRUD access to DynamoDB
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: DDBActions
            Effect: Allow
            Action:
              - dynamodb:BatchGet*
              - dynamodb:DescribeStream
              - dynamodb:DescribeTable
              - dynamodb:Get*
              - dynamodb:Query
              - dynamodb:Scan
              - dynamodb:BatchWrite*
              - dynamodb:Create*
              - dynamodb:Delete*
              - dynamodb:Update*
              - dynamodb:PutItem
            Resource: !Sub arn:aws:dynamodb:eu-central-1:user-id:table/ecs-${App}-${Env}-${Name}-*
          - Sid: DDBLSIActions
            Action:
              - dynamodb:Query
              - dynamodb:Scan
            Effect: Allow
            Resource: !Sub arn:aws:dynamodb:eu-central-1:user-id:table/ecs-${App}-${Env}-${Name}-*/Index/*

Outputs:
  DynamoDbAccessPolicy:
    Description: "The IAM::ManagedPolicy to attach to the task role."
    Value: !Ref DynamoDbAccessPolicy

I've removed personal information like User ID and repository's URL.

[huanjani]

Hello, @fuji97!

Hmm 🤔 , that's odd. Thanks for all of the info you included; are your test and prod environments in the same or different AWS accounts? We are going to try to reproduce the issue-- what region(s) are your envs in?

Thanks!

[fuji97]

Hi, @huanjani
Yes, both the clusters are under the same AWS account (sub-account with administrator rights) and everything is in the eu-central-1 region.

[huanjani]

From https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-template-validation/
Validate logical and physical IDs

1. Confirm that resource logical IDs are defined in your template.
2. Confirm that resource physical IDs exist in your environment.

For example, test is referenced for the ImageId property in the following JSON template and for the ImageID property in the following YAML template. Neither template includes a resource logical ID or parameter named test.

The resource ID isn't correctly defined in the following JSON and YAML templates. These templates return the following error: "Unresolved resource dependencies [test] in the Resources block of the template.” For more information on resource definitions and their syntax, see Resources.

JSON:

{
  "Parameters" : { ... },
  "Resources" : {
    "EC2Instance01" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : {"Ref": "test"},
        ...
      }
    }
  }
}

YAML:

Parameters:
Resources:
  EC2Instance01:
    Type: AWS::EC2::Instance
    Properties:
      ImageID: !Ref: test

[huanjani]

Did you do Step 1 in the sample template here? https://aws.github.io/copilot-cli/docs/developing/additional-aws-resources/

[fuji97]

Did you do Step 1 in the sample template here? https://aws.github.io/copilot-cli/docs/developing/additional-aws-resources/

Initially we was using the template created by copilot storage init, but we noticed that we preferred to create the table manually or by the application itself, so we decided to remove the table creation resource and edited the IAM Policy to allow access to all tables that matched a pattern.

By the way, we have 2 addons, one for DynamoDB and one for Secrets Manager and we noticed that removing only one of the two addons always triggers the error, we were able to deploy successfully only by removing both addons.

This is the secretsmanager.yaml

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.
Resources:
  smDemoecsAccessPolicy:
    Metadata:
      'aws:copilot:description': 'An IAM ManagedPolicy for your service to read secrets from Secrets Manager'
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Grants access to the Secrets Manager
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:ListSecrets
            Resource: "arn:aws:secretsmanager:eu-central-1:user-id:secret:*"
            Condition:
              ForAllValues:StringEquals:
                secretsmanager:ResourceTag/copilot-application: !Ref App
                secretsmanager:ResourceTag/copilot-environment: !Ref Env


Outputs:
  smDemoecsAccessPolicy:
    Description: "The IAM::ManagedPolicy to attach to the task role."
    Value: !Ref smDemoecsAccessPolicy

[efekarakus]

hi @fuji97 ! This is very strange, especially since the stacks get created successfully 🤔

@huanjani found this issue yesterday (https://github.com/aws/aws-cdk/issues/4406) from another tool which seems to be related to this error. So we'll investigate if adding a DependsOn: 'AddonsStack' for the task role will resolve this problem.

[CalogeroMandracchia]

Hi! I'm also following, is there any news? Thanks!

[efekarakus]

Hi @CalogeroMandracchia ! apologies for the lack of communication, we have tried placing DependsOn: 'AddonsStack' on the task role last week but that didn't help solve the problem.
We'll continue investigating the issue this week. We see the stacks get created in CloudFormation successfully but CodePipeline throws an error.

[huanjani]

Hello, @CalogeroMandracchia!

Sorry you're also having this issue. Can you please tell us more about your use case? Are you also deploying to two stages? Is the pipeline deployment failing for just one or both? If you don't mind, can you share your addons template? Is an IAM::ManagedPolicy your only resource?

Thanks in advance!

[huanjani]

@fuji97 and @CalogeroMandracchia:

A possible solution!

Try creating a new buildspec.yml by:

1. rm copilot/buildspec.yml
2. copilot pipeline init again to generate the buildspec anew; the existing pipeline manifest will remain
3. add, commit, push the new buildspec to your repo
4. copilot pipeline update

(Thanks, @vbarba!)

[fuji97]

@fuji97 and @CalogeroMandracchia:

A possible solution!

Try creating a new buildspec.yml by:

1. rm copilot/buildspec.yml
2. copilot pipeline init again to generate the buildspec anew; the existing pipeline manifest will remain
3. add, commit, push the new buildspec to your repo
4. copilot pipeline update

(Thanks, @vbarba!)

Ok, it worked for me!
The pipeline successfully deployed to production after recreating the buildspec.yml.

[huanjani]

That's so great to hear! We're still digging into this to try to prevent this blocker from recurring. Thanks for bringing this to our attention!

[gautam-nutalapati]

We saw similar issue and re-creating buildspec.yml helped. Thank you @huanjani .
In case useful for others, In our case, we moved our pipeline from one env to another, and I think we missed updating section of build script that uploads addons templates to respective env:

jq --arg a "https://stackset--infrastru-pipelinebuiltartifactbuc-*/manual/$timestamp/$workload.addons.stack.yml" '.Parameters.AddonsTemplateURL = $a' ./infrastructure/$workload-<ENV>.params.json > "$tmp" && mv "$tmp" ./infrastructure/$workload-<ENV.params.json