Amplify-cli: Support for custom Resources/CloudFormation templates

[johanneslanger]

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

There is no easy way to add and publish custom CloudFormation templates using the amplify cli workflow. This results in having to deploy AWS resources which are not supported by Amplify CLI currently with a custom script.

What is the expected behavior?

It would be nice to integrate custom CloudFormation templates/resources into the amplify cli workflow in order to allow deployment of features or services which are not supported by amplify-cli and streamline the deployment experience.

[SagarMhatre]

An example that I am working on where I need this is where I am modifying the Lambda Function's app.js (created by amplify cli) & calling some additional API methods which need additional permissions. Currently I am modifying the role through the web console to get it working, but it would be nice to have it as a part of the cloudformation template itself

[kaustavghosh06]

@johanneslanger @SagarMhatre

We're currently brainstorming and working towards providing a CI/CD workflow as a part of the Amplify CLI, and in doing so we want to work backward from the customer needs and we've opened up an issue (#286) as a part of the CLI repo to track this story.
Would be great, if you guys could take a look at it and respond back with some answers to the questions mentioned in the issue (#286) which would give us a better idea about your needs. Thanks!

[abdelhakim-atmani]

Hi,

Any update ? Without the customisation of the cloudformation template we are facing a big limitation. The current implementation is nice when you start the project but after some weeks of development it became difficult to use it. We are even not able to call 2 dynamodb from a lambda function and if I need to add some pre operation to my cognito user pool (like a pre-signup) I cannot do that.
It would be nice to be able to reference the created resources from amplify to another CF. Just by exporting the ARN.

[kaustavghosh06]

@abdelhakim-atmani Could you tell me a bit more about your use case and the feature that you would like as a part of the CLI to improve your workflow?

[abdelhakim-atmani]

For example I have one lambda which needs to connect to 2 dynamodb tables. And I need to have a lambda triggered by a dynamodb stream. Another use case would be to add a pre-signup operation to my cognito user pool.
The feature would be at least to export all ARN resources created by CLI in that case I can create a separate CF template pointing to them.

[chufgard]

We would like to use such a feature too. We have some resources (DynamoDB, s3 buckets) that are shared by two lambdas. Actually something that is pretty easy to implement with amplify. Just add the parameters to the cloudformation template and the sources as dependsOn to amplify-meta.json.

Unless you change something somewhere and amplify-meta.json is overwritten...

imho it should be pretty straight forward to implement a mechanism that allows to define the dependencies the way we use them in an additional file that is merged into amplify-meta.json.

[hoang-innomizetech]

Currently, we are using Serverless Framework to build and configure all backend functionals including Auth (Cognito User Pool), web hosting, API (Lambda + APIG), custom resources (s3 bucket, queue/topic, Dynamodb tables, etc.).

We are trying to evaluate Amplify in order to move all our shared backend stuff to Amplify if we can. We still use Serverless for our API development. Then I tried to figure out how can we add custom resources (almost we can use CloudFormation to provision and configure custom resources). However, I cannot see any document that can help us solve that problem.

I think we can utilize Amplify plugin to create an additional CF Stack file, then add it to the master/root stack for all custom resources like other categories that Amplify is currently supported. I worked likes custom category that allows developer to add any custom resource they want for their project

[zjullion]

Behold, a way to add custom resources (not via the CLI, but it works nonetheless):

Modify backend-config.json like so:

{
    "hosting": {
        "S3AndCloudFront": {
            "service": "S3AndCloudFront",
            "providerPlugin": "awscloudformation"
        }
    },
    "api": {
        "myapp": {
            "service": "AppSync",
            "providerPlugin": "awscloudformation"
        }
    },
    "ses": {
        "emailTemplates": {
            "providerPlugin": "awscloudformation"
        }
    },
    "sns": {
        "importantTopic": {
            "providerPlugin": "awscloudformation"
        }
    },
    "yetAnotherCustomResource": {
        "something": {
            "providerPlugin": "awscloudformation"
        }
    }
}

(IMPORTANT: note that you should put each custom resource under it's own 'category' - putting them all under one folder, such as 'custom', seems to cause issues)

Under amplify/backend folder, make a folder structure like so:

  \backend
    \api
      ...
    \ses
      \emailTemplates
        parameters.json
        template.json
    \sns
      \importantTopic
        parameters.json
        template.json
    \yetAnotherCustomResource
      \something
        parameters.json
        template.json
    \hosting
      ...

Just like under hosting, template.json is a cloudformation template, and parameters.json is a json file of parameters that will be passed to the cloudformation template. Additionally, the env parameter will be passed in to your cloudformation templates.

Viola! You can publish whatever custom cloudformation templates you like as part of your amplify stack.

Important note: ensure you do an amplify env checkout <env> before pushing / publishing, otherwise your new custom resources will not be published.

[CodySwannGT]

Is this a viable solution if you need to reference other areas of the CF templates?

For example, let's say I want to have a DynamoDB Stream attached to a table created by the @model directive that sends an email whenever a new record is created in that table.

How could I reference the DynamoDB table in my custom CF template?

[zjullion]

@CodySwannGT to reference outputs from other stacks in your custom CFN templates, follow my comment here: https://github.com/aws-amplify/amplify-cli/issues/1481#issuecomment-493603180

Unfortunately, you can't control the outputs provided by the api stack, but there are sometimes ways to use those outputs along with string interpolation to get the ARNs you need to reference various DynamoDB / AppSync resources.

To get a DynamoDB table stream ARN specifically, you can use the following ImportValue: ${apiId}:GetAtt:CompletedTransactionTable:StreamArn, where apiId is the GraphQLAPIIdOutput output of your API stack.

[kaustavghosh06]

Thanks, @zjullion, for helping out folks with the custom cloud formation stack issue. We'll make sure to document some of this as a part of our official docs.

[davidjhodge]

Spent a few hours debugging null parameter errors in custom functions. Turns out @zjullion hit the nail on the head, and that you must put each custom resource in a custom category.

I found this to be very counterintuitive, because the "function" category that's auto-generated for Lambdas supports multiple function resources within that category. Naturally, I followed this pattern and created my own "rule" category, that contained multiple custom IoT rules, each with a custom template.json.

After my rules kept failing to deploy, I tried many different variations, including hardcoding the ARN string, and creating a parameters.json, to get the Lambda ARN to be injected into the custom template.json without throwing a "Parameters: [xyz] must have values" error. Those erorrs are shown below:

CREATE_FAILED rulecreatedevice AWS::CloudFormation::Stack Mon Apr 20 2020 15:09:03 GMT-0400 (Eastern Daylight Time) Resource creation cancelled CREATE_FAILED ruleupdatedevice AWS::CloudFormation::Stack Mon Apr 20 2020 15:09:02 GMT-0400 (Eastern Daylight Time) Parameters: [functioncreatedeviceArn] must have values

As a feature request/enhancement to the existing behavior, it would be amazing if users could create custom categories that contain multiple custom resources, with full support for referencing existing parameters. As users create more robust Amplify applications with unique types of resources, I feel that simply allowing multiple custom resources in single custom category will save a few hours of debugging time and improve the Amplify development experience in general.

[tarun971]

Hi @zjullion - I followed your advice to add a step function template within my Amplify project. Thanks a lot for such a valuable pointer. One thing I am still trying to figure out is that if there is a way to keep the step function definition outside of the Cloudformation template? below are more details to explain what I am trying to do

Current behavior

1. Within /amplify/backend/backend-config.json I added by custom stack as follows
   "customstepfunction": {
   "myCustomStepFunction": {
   "service": "customStepfunction",
   "providerPlugin": "awscloudformation"
   }
   }

2. I created the following directory

3. /amplify/backend/customstepfunction/myCustomStepFunction

4. I added the following files

5. /amplify/backend/customstepfunction/myCustomStepFunction/parameters.json
   (with content in this file as follows)
   {}

• /amplify/backend/customstepfunction/myCustomStepFunction/template.json
  (with content in this file as follows)
  {
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description":"An example template for a Step Functions state machine.",
  "Parameters": {
  "env": {
  "Type": "String"
  }
  },
  "Resources":{
  "MyStateMachine":{
  "Type":"AWS::StepFunctions::StateMachine",
  "Properties":{
  "StateMachineName":"HelloWorld-StateMachine-123",
  "StateMachineType":"STANDARD",
  "DefinitionString":"{\"Comment\":\"A Hello World example of the Amazon States Language using Pass states\",\"StartAt\":\"Hello\",\"States\":{\"Hello\":{\"Type\":\"Pass\",\"Result\":\"Hello\",\"Next\":\"World\"},\"World\":{\"Type\":\"Pass\",\"Result\":\"World\",\"End\":true}}}",
  "RoleArn":"arn:aws:iam:::role/service-role/"
  }
  }
  }
  }

Desired Behavior

• Instead of writing the state machine definition within the template /amplify/backend/customstepfunction/myCustomStepFunction/template.json . I want to keep that outside of template is a file like /amplify/backend/customstepfunction/myCustomStepFunction/mystatemachineDefinition.json
• If there is a way "amplify push" can add my custom code (mystatemachineDefinition.json) onto S3 build location then I can potentially provide reference of S3 location within my template. Is there a way to do so?

[zjullion]

• Instead of writing the state machine definition within the template /amplify/backend/customstepfunction/myCustomStepFunction/template.json . I want to keep that outside of template is a file like /amplify/backend/customstepfunction/myCustomStepFunction/mystatemachineDefinition.json
• If there is a way "amplify push" can add my custom code (mystatemachineDefinition.json) onto S3 build location then I can potentially provide reference of S3 location within my template. Is there a way to do so?

I don't personally know a way to make either of these things work. We haven't explored any of these options on my team. Good luck to you, though!