Hello,
Can you help me with this?
I would like to use pipeline resolvers (otherwise, my logic cannot work).
I know that amplify console does not support this feature and that there is a ticket about that:
https://github.com/aws-amplify/amplify-cli/issues/1055
But my question is about how I can do this manually. I mean, what files and content do I need to put inside my amplify folder so when I use “amplify push”, the pipeline is uploaded?
Because if I don’t do that, if I use the amazon console or dashboard to create my pipeline, the next time I execute “amplify push”, everything done using the AWS console or dashboard will be lost.
Without pipeline resolvers, we only have a pair of files, for example:
Mutation.createNote.req.vtl
Mutation.createNote.res.vtl
And modify the cloud formation file 2 new entries.
That’s easy.
But when using pipeline resolvers...
Well, check this picture
https://miro.medium.com/max/1348/1*nj4pfiFzCzA9xdjn8TF8Sg.png
From:
https://medium.com/@crhuber/using-aws-appsync-pipeline-resolvers-for-graphql-authorization-d04bb7a8dc44
There we have 2 functions in the pipeline, I guess (not sure) I need 8 files. Something like:
I guess that createNote launches the pipeline, so, its data source is not a dynamo database, am I right?
How is this setup in that file in the JSON cloudformation file?
How do I setup BEFORE and AFTER files (if they are files), in the cloudformation template?
Can anyone share its json cloud formation file? If I see one, I can use that one to start.
Thanks a lot.
@Ricardo1980 I'm sorry I don't have time to put this down in more detail, I'm under some time pressure on a project right now. Hopefully these highlights will point you in the right direction. My project has a lot of custom pipelines, mainly to enable my authorization scheme, so if you have questions or run into trouble I can try to help.
The first thing to keep in mind is that this is a little fiddly and there is nuance in how amplify CLI behaves with custom stacks. You will need to line up your graphql schema with your custom pipeline and ensure that you provide the correct directives in your graphql to the amplify cli or it will step on your toes. That said, it's just a learning process, a bit of trial and error should give you the nuance. Also, read up on @key, it can be very useful in combination with custom pipelines!
I'm going to use a simple example here:
First create a graphql schema with a model and _manually_ specify the queries or mutations you want to pipeline like this:
@model(queries: null, mutations: null, subscriptions: null)
type MyModel {
id: ID!
name: String!
}
type Query {
getMyModel(id: ID!): MyModel
}
This goes into your %amplify-project%/amplify/backend/api/%api-name%/schema/schema.graphql file.
Step 2.
Look in %amplify-project%/amplify/backend/api/%api-name%/stacks
you should have a "CustomResources.json" in there, if not you can create one: (sorry it's been a while since I've done this part)
If it's not there: create a file in %amplify-project%/amplify/backend/api/%api-name%/stacks
name it: "CustomResources.json" but it can be anything you like.
Paste this into the file:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Custom AppSync Resources",
"Metadata": {},
"Parameters": {
"AppSyncApiName": {
"Type": "String",
"Description": "The name of the AppSync API",
"Default": "AppSyncSimpleTransform"
},
"S3DeploymentBucket": {
"Type": "String",
"Description": "The S3 bucket containing all deployment assets for the project."
},
"S3DeploymentRootKey": {
"Type": "String",
"Description": "An S3 key relative to the S3DeploymentBucket that points to the root of the deployment directory."
},
"AppSyncApiId": {
"Type": "String",
"Description": "The id of the AppSync API associated with this project."
},
"env": {
"Type": "String",
"Description": "The environment name. e.g. Dev, Test, or Production",
"Default": "NONE"
}
},
"Resources": { }
}
In the Resources sections is where your custom resources go. For each Custom Data Source, Function, or Resolver you will need a resource defined here.
Since amplify created our table with @model, we don't need a custom data source. We can use the exported information that is made available to us. This is easy because they use a convention: %typename%Table so "MyModelTable" is our datasource name for the type MyModel
Ok so for each function we want to use we need an entry in Resources like so:
Resources: {
"GetMyModel": {
"Type": "AWS::AppSync::FunctionConfiguration",
"Properties": {
"Name": "GetMyModel",
"ApiId": {
"Ref": "AppSyncApiId"
},
"DataSourceName": "MyModelTable",
"FunctionVersion": "2018-05-29",
"RequestMappingTemplateS3Location": {
"Fn::Sub": [
"s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Function.MyModel.req.vtl",
{
"S3DeploymentBucket": {
"Ref": "S3DeploymentBucket"
},
"S3DeploymentRootKey": {
"Ref": "S3DeploymentRootKey"
}
}
]
},
"ResponseMappingTemplateS3Location": {
"Fn::Sub": [
"s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Function.MyModel.res.vtl",
{
"S3DeploymentBucket": {
"Ref": "S3DeploymentBucket"
},
"S3DeploymentRootKey": {
"Ref": "S3DeploymentRootKey"
}
}
]
}
}
},
}
Now you must create the Function.MyModel.req.vtl and Function.MyModel.res.vtl and put your VTL code in there. A good trick is to put a regular model type into your schema and leave off the "queries:null" directive. Then run "amplify api gql-compile" and look in %amplify-project%/amplify/backend/api/%api-name%/build/resolvers for the vtl that is auto generated, you can copy that and make a few adjustments to create your files. Make sure you put the vtl files you create into %amplify-project%/amplify/backend/api/%api-name%/resolvers and NOT in the build folder, the build folder is overwritten by amplify on gql-compile or push.
Now, to create a pipeline resolver is similar, you are putting another entry into Resources like so:
Resources: {
"GetMyModel": { ... },
"GetMyModelResolver": {
"Type": "AWS::AppSync::Resolver",
"Properties": {
"ApiId": {
"Ref": "AppSyncApiId"
},
"TypeName": "Query",
"Kind": "PIPELINE",
"PipelineConfig": {
"Functions": [
{
"Fn::GetAtt": [
"GetMyModel",
"FunctionId"
]
}
]
},
"FieldName": "getMyModel",
"RequestMappingTemplate": "{}",
"ResponseMappingTemplate": "$util.toJson($ctx.result)"
}
},
}
Now obviously, you will want more than one function, so you would list each function you need in your resolver here. Ensure your "KIND" is "PIPELINE" and "FieldName" matches the query type you specified in the graphql schema.
Once this is done you should be able to push and test your configuration. Some errors are not detected until cloudformation gets the stacks so this can be a little frustrating because of the round trip time and rollbacks but once you get the hang of it and build up a stable of functions it's not so hard. Use the Amplify console to test your queries, and use cloudwatch to check the logs for errors, etc.
A couple other notes: read up on the AppSync VTL programming https://docs.aws.amazon.com/appsync/latest/devguide/resolver-util-reference.html
especially: $ctx.stash and $util
Amplify uses $ctx.source to pass ids on connections. It's worth it to create a named @connection and look into the build folder to see how the VTL is structured. Also, you can now create lambda functions in the graphql automatically and use those in pipelines also. Referencing the amplify team's vtl templates is super useful and copying them then modifying them to your own purposes even more useful.
Sorry I can't be more thorough, there is a deep well here and it is very flexible. With a little ingenuity you can build up some fast and intuitive API's. Hope this helps.
Hello @ryanhollander
Thanks a lot for your help, I appreciate it a lot.
I started understanding a lot of things with your post.
Some quick questions:
AWS::AppSync::FunctionConfiguration is to define functions
AWS::AppSync::Resolver is resolver that you attach to fields in a schema
So, when using pipelines, we have one Resolver and 2 (or more) AWS::AppSync::FunctionConfiguration, is that correct?
Resolver does not have a pair of files in S3, it is just a section in the cloud formation stack, right?
I thought BEFORE and AFTER mappings were 2 different files in S3, but I see you embedded them in the Resolver definition, which seems better. So, intead of RequestMappingTemplateS3Location/ResponseMappingTemplateS3Location, better to use RequestMappingTemplate/ResponseMappingTemplate because it seems to me those mappings are always the same, right?
And last question, how do you execute several functions? Here you are only executing GetMyModel? what is FunctionId?
"Functions": [
{
"Fn::GetAtt": [
"GetMyModel",
"FunctionId"
]
}
]
The doc does not say too much:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-appsync-resolver-pipelineconfig.html
Thanks a lot!
It is very annoying that Amazon does not explain this clearly.
@Ricardo1980 you're welcome. You're right, some of the documentation in this area is non-existent or hard to find and it combines a few services so the docs are spread out. I've found a lot searching this github too as other devs have run into and solved problems. There is a lot of info still stored in github issues which, while helpful, is not as easy as good documentation. Still, there is no substitute for experience, press on my friend. I struggled with a lot of this stuff over the last year but now I can whip these things out in a few minutes.
AWS::AppSync::FunctionConfiguration is to define functions
Yes.
AWS::AppSync::Resolver is resolver that you attach to fields in a schema
Yes.
So, when using pipelines, we have one Resolver and 2 (or more) AWS::AppSync::FunctionConfiguration, is that correct?
Yes, my example had one function (which will work) but typically you'd want a pipeline to have two or more functions. Also, you can use functions in multiple, different resolvers of course.
Resolver does not have a pair of files in S3, it is just a section in the cloud formation stack, right?
You can specify the VTL in a file if you want to. If it's complex I'd suggest that. If you look at the function definition you will see the template code you need to reference a file to switch the resolvers to use files for their mapping templates. A lot of resolvers don't need complex templates in their mapping slots as the action happens in the resolver or functions.
And last question, how do you execute several functions? Here you are only executing GetMyModel?
Just list them:
"Functions": [
{
"Fn::GetAtt": [
"GetMyModel",
"FunctionId"
]
},
{
"Fn::GetAtt": [
"SomeOtherFunctionName",
"FunctionId"
]
},
]
what is FunctionId?
The expression "Fn::GetAtt" tells cloudformation to lookup an attribute of a previously defined resource. In this case the resource named "GetMyModel" which is our function. cloudformation wants to reference the internal FunctionId for this resource so this expression just grabs the function id of the function we want to run in this position of the resolver.
You should review the cloudformation documentation for more on these expressions:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html
@ryanhollander thank you very much. Thanks to your explanations, finally I got this working!
You have no idea how much I appreciate your help, I was totally stuck on this.
Again, thanks :)
To Amazon ( @nikhname @UnleashedMind ): You should explain this in your documentation if you really want to create a good product and/or help users, given that amplify cannot create pipelines on its own.
@Ricardo1980 You're welcome. I'm glad I could help set you on track.
To be fair to the Amplify team, who are working hard everyday to bring us great new features given some pretty spectacular constraints, this system is meant as a sort of advanced feature for developers more familiar with Cloudformation, so it's a bit outside their core mission. That said, there are some examples in the amplify documentation, it's just that you have to go to the AppSync and Cloudformation documentation to get the whole story. I'm sorry, I did not mean to give the impression this is undocumented, I must have forgot to link you to this initially (or assumed you already saw it), if it helps here it is: https://aws-amplify.github.io/docs/cli-toolchain/graphql#custom-resolvers
Also, just to be pedantic. Pipelines with lambdas are already baked in, and AFAIK more better pipeline support is on the roadmap. The team tends to work on the features most demanded by the user base (or tries too) and pipeline support is complex. Amplify is open source, so if you have the time and energy you could also contribute to this.
Not to be dismissive of your criticism, at all, I think it is valid, the documentation can definitely be improved! I just mean to put your criticism in perspective to account for the hard work and good intent of the Amplify team, and recognize that we can all improve our software and documentation and give them the benefit of the the doubt: they are all working hard to bring us some great software that is both flexible and easy to use, no small task!
I'm glad I was able to help and set you on the right track, and as a bonus the blog post I've been meaning to write on this topic is now about 50% done. Good luck with your project!
@ryanhollander Problem is, I am using Aurora Serverless as a data source, and many features are not available when using Amplify and documentation is always about DynamoDB...
For example, you cannot add data models using the command line.
Anyway, the structure (cloud formation) when using pipelines is almost the same.
Again, thanks a lot.
Let me know when you write that post, it will be very useful :)
@Ricardo1980 will do. Don't hold your breath though, might be a while. :-)
@ryanhollander I was able to get this working. However, now amplify mock
does not work. Any idea how to fix that? Should I add the stack config from /bulid/stacks/MyModel.js
and make the updates there vs. CustomResources.json
?
Also on another note, do you have any clue how to add interfaces as a @model
? Something like:
interface User @model {
id: ID
email: String
created: AWSTimestamp
}
type ActiveUser implements User {
id: ID
first: String
last: String
email: String
created: AWSTimestamp
}
type InvitedUser implements User {
id: ID
email: String
created: AWSTimestamp
invitedBy: String
}
type Team @model {
users: [User] @connection
}
@jarkin13 I haven't ever used mock. I just tried it and I get an error when I run it. I'm sorry, I don't personally have time to figure that out right now. I suggest opening a new issue for the devs or another kind soul with more experience with that feature than I have.
Regarding interface, I don't think it's supported with @model and quick google search brings up these:
https://github.com/aws-amplify/amplify-cli/issues/1037
https://github.com/aws-amplify/amplify-cli/issues/202
I've been using the console to re-create my pipelines after every push, since I've never (despite many attempts) been able to correctly configure pipelines in CustomResources. Now I'm finally able to automate this. Big thank you @ryanhollander ! 🎉
@lissau Glad to hear it!
Hello @ryanhollander wow thank you so much for your explanation.
I spend a whole day trying to work this out, I didn't get it at all until I stumbled onto this post.
I have an idea to automate the process you described. I'd just like to describe why I think it's necessary and how I plan to do it here. Hoping you or anyone else might have some input or things I should watch out for.
I have an amplify graphql api for building competitions which is going to have a fairly complex authorisation logic (e.g: only the user who is set to the judge/admin of the parent competition can call updateScore
). There is a bunch of other checks that need to be done when updating/creating entities.
I don't want to do it in vtl
. I want to put all the logic into an authorisation lambda. My lambda will then authorise user actions (e.g: user A is trying to update Score with id 123 but he is not the judge or admin of the parent competition so return unauthorised). So I want to change every auto generated create/update/delete resolver into a pipeline resolver which calls my authorisation lambda first.
Note: I will also have a business logic lambda later that will do some other checks. Like createCompetitorCompetitionAllocation
can not be called if number of competitors >= capacity. I plan to incorporate this business logic lambda by also adding it to the pipeline.
I plan to write a bash script to automate the process outlined by @ryanhollander
Using your example, you can overwrite the default function names for mutations like so;
@model(mutations: { create: "auto_gen_createMyModel", update: "auto_gen_updateMyModel" })
type MyModel {
id: ID!
name: String!
}
type Mutation {
createMyModel(input: UpdateMyModelInput!, condition: ModelMyModelConditionInput): MyModel
updateMyModel(input: UpdateMyModelInput!, condition: ModelMyModelConditionInput): MyModel
}
I can then create a bash script to
amplify api gql-compile
vtl
files to /%api-name%/stacks
(and remove 'auto_gen_' from file name)jq
to add Functions and Resolvers to CustomResources.json
This will create 2 copies of my functions in the api (e.g: auto_gen_updateMyModel
and updateMyModel
) but thats fine during development. I just have to remember to change all the @model
mutations to null in the schema before going to production.
Any changes I have made to my custom vtl
file (e.g; adding $context.identity.username
as the default admin when creating a competition) will be overwritten. But that's fine I don't have many vtl
customisations (because I hate vtl
) I can just keep them at the top of my files (fenced with ## BEGIN CUSTOM VTL
and ## END CUSTOM VTL
) to make sure my bash script can find them and merge them into the new autogenerated files.
Any feedback on my plan would be greatly welcome. Of course I will share my script on here when I am done.
@ziggy6792 I'm glad my post helped you out! Depending on your use cases and other integrations, you might want to investigate @function and chaining @function, custom Lambda Triggers (especially if you are using cognito), and look into custom graphql transformers, which I can no longer find the documentation on but here is a breadcrumb: https://github.com/aws-amplify/amplify-cli/pull/1396 -and- https://github.com/aws-amplify/amplify-cli/issues/348 (disclaimer: I've never actually tried to build one of these). Using the built in features as much as possible will make your implementation less brittle over the long term. Good luck with your project!
I just took another look at the the graphql @auth
directive documentation. It seems there is some cool new stuff in there that wasn't in there when I first started this project. There is now dynamic-group-authorization. I can create a group for the judges and admins of my entities and only allow create/update/delete to members of those groups. Maybe what I want to do can be supported out the box after-all.
Dynamic-group-authorization should come with a big red warning sign. If you are using Dynamic-group-authorization you CANNOT use SUBSCRIPTIONS, and somewhat related you can't use DataStore for persistent offline cache. Also, there is a hard limit of 500 groups in Cognito.
Like most things, there are work arounds - put your groups in a DynamoDB table and use a pre-token-lambda to get groups from DynamoDB rather than Cognito. And put your Dynamic-group required behaviour in custom resolvers.
Hi @sacrampton thanks for your comment. Can you please explain what you mean by "CANNOT use SUBSCRIPTIONS" do you have a link to any issues?
Another option to me would be to protect all autogenerated mutations with @aws_iam
so that they can no longer be called with cognito auth.
E.g:
type Comptition @aws_iam {
#....
}
_Though I am not sure how I can set createCompetition
, updateCompetition
and deleteCompetition
to @aws_iam
and leave getCompetition
to `aws_cognito_user_pools. Does anyone know?_
Then I could create a pass through @function
(e.g publucCreateCompetition
) for each autgeneerated mutation that I want to expose to my front end.
The pass though @function
would do an authorisation check and then (if authorised) use an iam role to call 1 or more iam protected autogenerated functions.
What do you guys think?
In the link you provided (https://docs.amplify.aws/cli/graphql-transformer/directives#authorizing-subscriptions) it says...
"_Dynamic groups have no impact to subscriptions. You will not get notified of any updates to them._ "
Ok thanks. But I guess I can still use subscriptions that are not protect by @auth
right? Actually I only care about protecting my create, update and delete mutations. I don't about who subscribes to those actions.
Most helpful comment
@Ricardo1980 I'm sorry I don't have time to put this down in more detail, I'm under some time pressure on a project right now. Hopefully these highlights will point you in the right direction. My project has a lot of custom pipelines, mainly to enable my authorization scheme, so if you have questions or run into trouble I can try to help.
The first thing to keep in mind is that this is a little fiddly and there is nuance in how amplify CLI behaves with custom stacks. You will need to line up your graphql schema with your custom pipeline and ensure that you provide the correct directives in your graphql to the amplify cli or it will step on your toes. That said, it's just a learning process, a bit of trial and error should give you the nuance. Also, read up on @key, it can be very useful in combination with custom pipelines!
I'm going to use a simple example here:
First create a graphql schema with a model and _manually_ specify the queries or mutations you want to pipeline like this:
This goes into your %amplify-project%/amplify/backend/api/%api-name%/schema/schema.graphql file.
Step 2.
Look in %amplify-project%/amplify/backend/api/%api-name%/stacks
you should have a "CustomResources.json" in there, if not you can create one: (sorry it's been a while since I've done this part)
If it's not there: create a file in %amplify-project%/amplify/backend/api/%api-name%/stacks
name it: "CustomResources.json" but it can be anything you like.
Paste this into the file:
In the Resources sections is where your custom resources go. For each Custom Data Source, Function, or Resolver you will need a resource defined here.
Since amplify created our table with @model, we don't need a custom data source. We can use the exported information that is made available to us. This is easy because they use a convention: %typename%Table so "MyModelTable" is our datasource name for the type MyModel
Ok so for each function we want to use we need an entry in Resources like so:
Now you must create the Function.MyModel.req.vtl and Function.MyModel.res.vtl and put your VTL code in there. A good trick is to put a regular model type into your schema and leave off the "queries:null" directive. Then run "amplify api gql-compile" and look in %amplify-project%/amplify/backend/api/%api-name%/build/resolvers for the vtl that is auto generated, you can copy that and make a few adjustments to create your files. Make sure you put the vtl files you create into %amplify-project%/amplify/backend/api/%api-name%/resolvers and NOT in the build folder, the build folder is overwritten by amplify on gql-compile or push.
Now, to create a pipeline resolver is similar, you are putting another entry into Resources like so:
Now obviously, you will want more than one function, so you would list each function you need in your resolver here. Ensure your "KIND" is "PIPELINE" and "FieldName" matches the query type you specified in the graphql schema.
Once this is done you should be able to push and test your configuration. Some errors are not detected until cloudformation gets the stacks so this can be a little frustrating because of the round trip time and rollbacks but once you get the hang of it and build up a stable of functions it's not so hard. Use the Amplify console to test your queries, and use cloudwatch to check the logs for errors, etc.
A couple other notes: read up on the AppSync VTL programming https://docs.aws.amazon.com/appsync/latest/devguide/resolver-util-reference.html
especially: $ctx.stash and $util
Amplify uses $ctx.source to pass ids on connections. It's worth it to create a named @connection and look into the build folder to see how the VTL is structured. Also, you can now create lambda functions in the graphql automatically and use those in pipelines also. Referencing the amplify team's vtl templates is super useful and copying them then modifying them to your own purposes even more useful.
Sorry I can't be more thorough, there is a deep well here and it is very flexible. With a little ingenuity you can build up some fast and intuitive API's. Hope this helps.