Amplify-cli: RFC: @function directive

[mikeparisstuff]

Feature Request

Let's use this ticket to track the progress of the @function directive that will enable easily calling out to lambda functions from within your schema.graphql

Directive Definition

The goal of this directive to provide an easy mechanism to call a AWS Lambda function from a field in your AppSync API. Todo this, I propose we introduce a @function directive.

directive @function(name: String!, region: String) on FIELD_DEFINITION

You may use this directive to attach a lambda resolver by function name to a field in your schema.graphql. By default, the region assumes the same region as the stack (AWS::Region in CF) but this may be overwritten. You may use ${env} to enable multienv support from the CLI.

Usage

Let's assume you created an AWS Lambda Function named echofunction using the amplify add function command. You can easily call that lambda function from your GraphQL API with the following:

type Query {
  echo(msg: String): String @function(name: "echofunction-${env}")
}

The directive above would generate the following CloudFormation resources and resolver templates:

"QueryEchoResolver": {
      "Type": "AWS::AppSync::Resolver",
      "Properties": {
        "ApiId": {
          "Ref": "AppSyncApiId"
        },
        "DataSourceName": {
          "Fn::GetAtt": [
            "EchoLambdaDataSource",
            "Name"
          ]
        },
        "TypeName": "Query",
        "FieldName": "echo",
        "RequestMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.echo.req.vtl",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              }
            }
          ]
        },
        "ResponseMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.echo.res.vtl",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              }
            }
          ]
        }
      }
    },
    "EchoLambdaDataSource": {
      "Type": "AWS::AppSync::DataSource",
      "Properties": {
        "Name": "EchoFunction",
        "Type": "AWS_LAMBDA",
        "ApiId": { "Ref": "AppSyncApiId" },
        "ServiceRoleArn": {
          "Fn::GetAtt": [
            "EchoLambdaDataSourceRole",
            "Arn"
          ]
        },
        "LambdaConfig": {
          "LambdaFunctionArn": {
            "Fn::Sub": [
              "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:echofunction-${env}",
              {
                "env": {
                  "Ref": "env"
                }
              }
            ]
          }
        }
      }
    },
    "EchoLambdaDataSourceRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": "EchoLambdaDataSourceRole",
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "appsync.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "Policies": [
          {
            "PolicyName": "InvokeLambdaFunction",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "lambda:invokeFunction"
                  ],
                  "Resource": [
                    {
                      "Fn::Sub": [
                        "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:echofunction-${env}",
                        {
                          "env": {
                            "Ref": "env"
                          }
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          }
        ]
      }
    }

Along with these resolver templates:

# Query.echo.req.vtl
{
    "version": "2017-02-28",
    "operation": "Invoke",
    "payload": {
        "type": "Query",
        "field": "echo",
        "arguments": $utils.toJson($context.arguments),
        "identity": $utils.toJson($context.identity),
        "source": $utils.toJson($context.source),
        "request": $utils.toJson($context.request),
    }
}

# Query.echo.res.vtl
$util.toJson($ctx.result)

All lambda resolvers can be passed a standard payload that includes the type and field being resolved as well as useful context information like the args, identity, and source. This could also include HTTP headers and in the future the selection set. Are there other attributes required here?

With this setup, customers will be able to write lambda functions that resolve a number of different GraphQL fields. For example, you might have a single function GraphQLResolversFunction that looks something like this:

/**
* Implement any resolver logic here.
*/
const Resolvers = {
  Query: {
    echo: (ctx) => Promise.resolve(ctx.arguments.msg)
  }
};
export async function handler(event, context) {
  const typeResolvers = Resolvers[event.type];
  const fieldResolver = typeResolvers ? typeResolvers[event.field] : null;
  if (fieldResolver) {
    const result = await fieldResolver(event);
    context.done(null, result);
  } else {
    context.done(`No resolver found for ${event.type}.${event.field}`, null)
  }
};

You may then use this single function to resolve many fields:

type Query {
  echo(msg: String): String @function(name: "GraphQLResolversFunction")

  anotherField(msg: String): String @function(name: "GraphQLResolversFunction")
}

[marcato15]

Would it be possible to reference a function created via the amplify cli by a name/identifier such that you could specify that directly from the GraphQL definition, instead of having to rely on the CLI to prompt you for which function to connect?

[HieronymusLex]

@mikeparisstuff would you be able to post here once a branch is available to test this feature? I'm happy to contribute to the development of this feature if needed. great work on this btw!

[oxbits]

Someone suggested I post this idea here:

Feature Request
https://github.com/aws-amplify/amplify-cli/issues/188

Brief Synopsis:

1. $context.original_raw_graphql_query_string

   • original raw graphql query/mutation/subscription string
     
     

2. $context.parsed_graphql_query_json

   • entire graphql query/mutation/subscription parsed into json
     
     

* at the VTL resolver layer

[awsed]

I'd love to see this somehow detecting a function created by "amplify add function", populating the ARN in CFN with the function created by the CLI

[IljaDaderko]

This solves many server side validation use cases imho, was work on @function started already so we can follow / help you guys, or is this still at proposal stage?

[rayhaanq]

Just wondering if there's been any progress on this

[ryan-mars]

Is this something the community needs to do or something the Amplify team is going to do? This is a blocker for me using Amplify for anything serious. I'm willing to dig in and ship this but not if the Amplify team is already working on it. Some transparency might help.

[undefobj]

@ryanwmarsh we are going to do this in the future, but at the current time it could be a couple months out or so as we're working on things like multiple environments. We posted the feature here for transparency so that in the meantime we could get feedback on the functionality. If you want to take a crack at the design based on the above definition we're happy to give feedback on the PR and evaluate your implementation. If you do wish to do this please reply on this thread with any thoughts on both the interface as well as implementation. We have certain design principles around simple, declarative interfaces and such that we can guide you through in the process.

[ryan-mars]

@undefobj I dug into this last night. I looked at the two of the directive implementations and debugged them. This makes sense. Function would be one of the more simplistic transforms to build. The only thing I'm concerned about is how we want to handle the actual lambda code. Could we leverage the amplify add function command and do something like...

1. amplify add function
2. Give it a name ImportantFunction, and name the function important, answer the questionnaire
3. Add a function directive to an Object or Field using ImportantFunction or important not sure...
4. amplify codegen: the FunctionTransformer.ts generates DataSource and Request/Response resolver templates.

The only question I have is how do we get the function Arn from the template at amplify/backend/function/ImportantFunction/ImportantFunction-cloudformation-template.json

It could be as simple as adding an additional output to the Amplify function template using the "friendly name" ImportantFunction? IDK, thoughts?

SDL would look like

type Post @function(name: "important") {
    id: ID!
    title: String
}

or

type Post @model {
    id: ID!
    title: String
    computedValue: Int @function(name: "important")
}

[undefobj]

Ideally we would like to support two capabilities here:

1. Specifying existing functions (such as those created with amplify add function
2. Specifying a new function name and @function would create a new Lambda that has some boilerplate "hello world" style code in it.

The first one above is probably the MVP for the directive and then we could add more from there.

@mikeparisstuff @kaustavghosh06 can you give some guidance on getting the ARN from a Lambda created with amplify add function?

[chrisl777]

The ability to call existing Lambda functions would be huge! Any progress on this ability @undefobj @ryanwmarsh?

[ryan-mars]

@chrisl777 I have a PR related to this (#490) with a minor refactoring which has been hanging out for a month. I'd like to finish working on @function but it's not going to move to the top of my list until I see better governance and communication here.

Quite frankly, I'm treating #490 like a canary.

[chrisl777]

@ryanwmarsh I see your PR, thanks for submitting some work. My guess is that the Amplify team has been super busy... not only with the holiday season, but with re:Invent late last month, releasing the multi-environment solution, along with Amplify Console, along with a plethora of other bug fixes and updates. I would very much like to see the @function directive added, but I can understand if Dec has been a tough month for them to be super responsive with the high amount of interest in Amplify. Let's hope that team is getting more resources with all of that interest. I think in 2019 we'll continue to see even more great things for Amplify!

[austinamorusocfc]

This is a big need on our team is there any headway or suggestions for current work arounds?

[ryan-mars]

@austinamorusocfc The pull request I was waiting on (simple necessary refactoring) before investing more time on this was never merged and has been open so long it now has a conflict. The amplify team is obviously working very hard, I assume they're under-resourced. I thought I knew which direction they wanted to take Amplify but I'm not sure anymore. I'm going to hang back and see what happens. In the mean time I'm using SAM on the backend, and amplify-js.

[undefobj]

Hi @austinamorusocfc @ryanwmarsh @chrisl777 we are indeed working on some larger, foundational changes - specifically we want to merge the new multi environment & teams feature we want to merge into mainline (avoiding the separate branch) as well as addressing issues around template sizes for nested stacks (see https://github.com/aws-amplify/amplify-cli/pull/581 for more information). There are is custom resolver support that we're working on which will allow you to drop in templates & CloudFormation extensions straight into the Amplify directory, which is a nice escape hatch for any use case, including passing it through to a Lambda of your choosing (see https://github.com/aws-amplify/amplify-cli/issues/574 for more information).

While we do want to add @function in the future to make this even easier but the issues above were not only blockers for actually deploying in "regular" use cases, but they also required some underlying foundational changes to the code such as stack partitioning and even migration under the covers from older projects. We also use analytics on GitHub issues and customer reactions/responses/tickets to drive backlog and these have really been rated high.

So not ignoring this at all and we do plan on looking at it but just being transparent on the current efforts. We hope to get some of these done in the coming weeks.

[ryan-mars]

@undefobj Hi Richard, great to hear. I really appreciate the transparency. More please. 🙏🏻

Amplify has been growing quickly. It only makes sense it would require some refactoring to accommodate such big features.

What would help potential contributors is if project leadership could paint a roadmap for us, and perhaps even be clear where community help is welcome and where it may not be necessary. I understand this can be a little tricky given that this project is simultaneously strategic for AWS and open source.

I’m curious if you could clarify the community engagement model intended for this repo.

[undefobj]

We are looking at doing something like this, but I don't have an ETA at the moment because it would take some effort and tooling on our part to do it properly. @mlabieniec is tracking this.

My biggest concern with roadmap is that the Amplify project truly works in an OSS manner at the moment and directions are community driven - we have staff on-call rotations answering issues, addressing bugs, answering questions live on the Gitter channel (we're about to also add a Slack too). We do metrics on GitHub issues/reactions/trends and drive sprints & epics off of this. While we do have opinions on the direction of the project and what we will/won't add (e.g. the goal is to help Frontend developers build mobile and web apps) I also want to ensure that we can keep our teams working in an agile manner to address the biggest customer issues. I would hate to publish something and lose trust because something was "on the roadmap for a while" but didn't get addressed because the community rated other things higher. I'd also hate for some of this to drive Amplify in a waterfall direction on features because of some list that might not be as relevant when the mobile and web space is rapidly changing. That's not to say we won't do it we're just trying to figure out what the right way to do it is. Your point on knowing high level what assistance might be needed is well taken though, and we'll look to account for this too.

@ryanwmarsh

[YikSanChan]

@undefobj Do you want to make this post more publicly accessible? I believe this is what the whole community happy to read.

[undefobj]

Sure

[chrisl777]

@undefobj Hi Richard, exciting to hear about all the plans, #574 looks especially good as a utility knife for all kinds of situations. Thanks for keeping us in the loop!

[mikeparisstuff]

I have updated this issue with new details on how we are thinking about the @function directive. The pattern detailed above can be implemented manually after the work related to #581 has been merged, but the @function directive will replace ~one hundred lines of CloudFormation with one line of SDL.

[zjullion]

@mikeparisstuff, are there plans to integrate @auth with @function? Or will there be some other way to ensure that the lambda can only be executed by authenticated Cognito users?

[austinamorusocfc]

@mikeparisstuff our team could really use this we are having to manually work around this and its a pain.

[tjhowe]

@austinamorusocfc How did you get around it? I'm currently looking in to this as well.

[zjullion]

Until @function is implemented, y'all should check out: https://aws-amplify.github.io/docs/cli/graphql#add-a-custom-resolver-that-targets-an-aws-lambda-function

[davekiss]

This is going to be the greatest PR of the decade. I never want to touch a VTL file ever again in my entire life.

[ajhool]

If the team continues to add features like this, Amplify will be to serverless architecture as React is to frontend development IMHO.

The Amplify system still has gaps and flaws but it has the potential to make cloud/serverless development as easy as monolith development. Kudos to the team on these excellent ideas. It's hard to overstate how much complexity Amplify strips from app development

[ambientlight]

@mikeparisstuff:

What do you think about in relation to pipeline resolvers https://github.com/aws-amplify/amplify-cli/issues/1055

The directive naming @function might logically clash with AppSynch pipe resolver functions, given amplify is going to support pipeline resolvers and based on your proposal 2 in https://github.com/aws-amplify/amplify-cli/issues/1055 does it make more sense using @function as generic pipeline resolver building block? thus lambda names becomes a datasource in that function directive: I have provided more thoughts about this in https://github.com/aws-amplify/amplify-cli/issues/1055#issuecomment-483942329

[ambientlight]

I've created a PR that implements original @mikeparisstuff proposal. https://github.com/aws-amplify/amplify-cli/pull/1321.

Let me know what you guys think. (though It does not create a new lambda function if lambda with passed name does not exist, as proposed in https://github.com/aws-amplify/amplify-cli/issues/83#issuecomment-440437673)

[buggy]

I agree with the comment @ambientlight made about @function potentially being confused with pipeline resolver functions. It might be better to use @lambda just so that @function is available later for pipeline resolvers.

[mhrisse]

Hi! Our team is really excited about this. Thanks for doing this feature. Could be as significant as proxy was for Lambda.

I recommend to keep the naming of the Transformers platform agnostic, or is this (not longer) a goal of amplify?

To me personally, not being an expert on this and still learning amplify, but having a fair dose of serverless and serverless frameworks experience overall, the @function keyword makes much more sense for Lamda/FaaS integration.

[undefobj]

I agree with the comment @ambientlight made about @function potentially being confused with pipeline resolver functions. It might be better to use @lambda just so that @function is available later for pipeline resolvers.

We wouldn't want to do this as @lambda is implementation specific, and Amplify in general has declarative & category based approach to both design and naming conventions. This allows us to change implementations later, for instance we could use AWS Lambda functions as well as XZY functions in the transformer through configuration options. We do a similar thing with the Auth category in the JS library having no Cognito specifics which is what allows us to also use Auth0.

[hisham]

I see #1346 is merged which is great. Question: does the multi-env version function-name-${env} support the NONE env where -${env} should be excluded but only if env = NONE?

[mikeparisstuff]

@hisham Thank you for the comment. I will submit a new PR with the update.

[valeeum]

Is this feature working with amplify version 1.6.9? None of the CloudFormation resources and resolver templates are getting created.

I created a function named AdminManageUsers using the amplify add function command. My qraphql definition looks like:

extend type Query { getGroups( limit: Int nextToken: String ):[Group] @function(name: "AdminManageUsers-${env}") }

[davekiss]

@valeeum working great for me! Are you using it on a Query or Mutation field?

[valeeum]

@davekiss I'm using it on a Query field. I did just noticed that is you define your @function in the root Query, it works as expected. So this works:

type Query {
  getGroups(
    limit: Int
    nextToken: String
  ):String @function(name: "AdminManageUsers-${env}")
}

But if you extend type Query, resources are NOT created:

type Query {
  dummy : Boolean
}

extend type Query {
  getGroups(
    limit: Int
    nextToken: String
  ):String @function(name: "AdminManageUsers-${env}")
}

The reason I take this approach is to organize my schema across multiple files.

[davekiss]

@valeeum I also organize my schema across multiple files but was only able to make it work with a single Query.graphql file containing queries relating to multiple entities ie.

type Query {
  me: User @function(name: "AdminManageUsers-${env}")
  myPosts: [Post] @function(name: "SomethingElse-${env}")
}

[valeeum]

@mikeparisstuff @davekiss would this be considered a bug?

[johncantrell97]

Is there any example of using a custom lambda resolver for a graphql query/mutation where the authenticated cognito user and dynamodb is accessible in the lambda function?

[davekiss]

@johncantrell97 these docs would probably be helpful for you: https://github.com/aws-amplify/docs/pull/640/files

[kaustavghosh06]

We launched support for @function directive today. You can find docs for the same out here - https://aws-amplify.github.io/docs/cli/graphql#function
Please let us know if you're still not able to solve your problem through this solution and we'll re-open this issue for you.

[ajhool]

Awesome, the implementation design looks great! Could we open a separate issue to track: https://github.com/aws-amplify/docs/pull/640/files#r279216768 , which I suspect will be a common impediment to using a lambda function in practice?

As far as I can tell, the @function directive currently overrides the lambda's IAM role, so if you interact with a resource outside of the Amplify ecosystem (eg. Cloudwatch or an SQS queue or a dynamodb table), you'll need to manually update the lambda's policy on each deployment in the AWS console and you can't currently use a cloudformation template

[kaustavghosh06]

@ajhool No, the @function directive won't override lambda's execution role. It would just add a role (not the lambda execution role), for the AppSync service to interact with the lambda function.

[ajhool]

Brilliant, thanks, sorry for the misinterpretation.

[ajhool]

Is #1055 Proposal 1 required before @function works with the @auth directive, as @zjullion suggested? As far as I can tell, @auth needs to be implemented as a pipeline directive to enable that use case. Most of our use cases require authentication for custom resolvers.

[GeeeCoin]

Hi @mikeparisstuff, I've been making an effort to pick up the latest Amplify tools, in particular the GraphQL related environment (as the rest is mostly familiar). And I've had quite the time trying to set up a workflow with the function directive. My issue and mistake is that I've added the function as a table field like so:

type MyTable
@model(
  mutations: { create: "addSomething" }
  queries: {get: "getSomething"}
  subscriptions: null
)
@auth(rules: [{allow: owner, ownerField: "username" , operations: [create, read]}])
{
  id: String!  
  username: String!
  address: String!
  # validation input string is not required below. probably called in parallel with mutation.
  myReqValidation(input: String): String @function(name : "validateReq", region: "us-blah")
}

Expectation: The validation lambda function would query some other tables via doc client and throw if data mismatch, which would prevent the requested mutation from taking place.
Actual: Function is called after addSomething mutation executes.

I've tried modifying the request mapping template in my AppSync pipeline resolver to catch the error like so:

#if( $ctx.error )
  $util.error($ctx.error.message, $ctx.error.type)
#end

but that doesn't handle the problem.

Two possible solutions.
1) Add the function directive directly to the mutation inside our @ model directive. Is this possible and is there a specific syntax to add this to the addSomething auto-resolved method above?
2) The longer form solution is to create a Mutation object with a mutation followed by function directive. In this case, I don't know if it's possible to autogenerate a resolver to handle the mutation if lambda is successful - lambda might have to do all the validation and ultimate table write. Also, is it possible to include the auth directive in the mutation object as @ajhool inquired?

I can open this as a new issue as I wasn't sure where best to post. I know a friend of mine had the exact same issue and had to move towards a different solution. Many thanks!