Serverless: Support response/request templates for APIG in v1

[eahefnawy]

APIG Response/Request templates options are now missing in V1, we need to bring back this functionality. In v0.5.6 we let users define their velocity templates as strings in the s-function.json json file, which was very hard to maintain. We can still do that in v1 using our serverless.yaml file, but it'd still be really hard to maintain all the templates in all endpoints in all functions in a single file.

Someone suggested before that we should let users define their velocity template in a .vm file and just reference it in the APIG config object in serverless.yaml (or s-function.json). I think this will make it much easier to deal with APIG and will provide super flexibility.

Another option is for Serverless to provide an abstraction layer on top of velocity templates so that users don't have to provide velocity templates at all, just some yaml config options that will be compiled into valid velocity templates. This will probably be better UX, but will be much less flexible, and users with specific needs might get stuck.

Or maybe we can do something in between. Thoughts...? cc/ @flomotlik @pmuens @ac360

Parent Issue:
https://github.com/serverless/serverless/issues/1408

[ac360]

+1 to .vm templates that can be referenced w/ a single line in the endpoint event.

[flomotlik]

Agree on the vm templates file that gets set in a line in the serverless.yml config. Before we move into this implementation though what are use cases that currently aren't supported by our default template?

Imho we should make the default template better and document it thoroughly. It sounds to me like those templates are a huge PITA and the better and easier we can make the default template the easier time our users will have. No question we need to be able to support custom templates in the future, just think we should make our built-in ones better first.

Regarding response template as we're currently not using any we should come up with a default one that is equally powerful as the request template.

[ac360]

@flomotlik Great points. FWIW, I believe custom template support is pretty easy to configure:

• Read a file
• Add contents to the CF template

[flomotlik]

@ac360 yup agreed, implementation isn't hard by itself, just need to make sure we fully understand what we can do and how people want to configure it.

[activescott]

As I mentioned in #1640 I am trying to implement a standardized REST-based API and I just need access to incoming request headers with non-typical content-types and ability to set outgoing response headers and content types. On my case for SCIM 2.0, but I'm sure there are other scenarios.

I think the velocity templates are a moronic PITA, and if you searxh around about them nobody understands them, so while a way to abstract them is a laudable goal, I suspect that with APIG there won't be a way to accomplish even basic use cases like mine (again, SCIM) without the ability to supply the full template. I think being able to reference them from the yml is the right approach.

I think I have a feel for how to contribute the feature if we can agree on the yml syntax/UX here.

[flomotlik]

@activescott do you think we won't be able to do this through default templates? E.g. give you a way to set headers and content types through a specific syntax that we provide, so you don't have to mess with Velocity templates at all?

I do think that we should provide a way to set your own template as well, but if we can provide a standard syntax for your output that gets translated into the corresponding headers, content-types and whatever else we need that would imho be super helpful

[oriolgual]

I'm also interested in this! I'm trying to integrate with Slack's outgoing webhooks and the requests are always sent as application/x-www-form-urlencoded, so I need to set a template to transform them to JSON. Is there any way to do it now other than setting it with the AWS console?

[flomotlik]

@oriolgual could you provide an example Request that you're getting from Slack and what you want it transformed into? Imho from going through the docs I think we're able to provide a standard template that also makes your workflow possible (as it seems to be a pretty common one) and I want to make sure we fully understand it.

[oriolgual]

Sure! This would be an example request I receive from Slack:

curl -XPOST https://apigateway/foo/bar -d 'token=Mwxll7OrsLINUc4yuOz2JgaP&team_id=T0001&team_domain=example&channel_id=C2147483705&channel_name=test&timestamp=1355517523.000005&user_id=U2147483697&user_name=Steve&text=Hello!'

It would be OK if I could read it from the event as:

event.data.channel_name (and the same for the other properties).

[oriolgual]

I'm actually using a solution I found on StackOverflow: http://stackoverflow.com/questions/32057053/how-to-pass-a-params-from-post-to-aws-lambda-from-amazon-api-gateway

[oriolgual]

I've implemented a naive solution for it: https://github.com/oriolgual/serverless/commit/5ef828742c329fcfd675e53e0356d467993d77ce

I'm using it now and it works great and it also keeps the same structure as the original since it injects all the form params as the event body.

[flomotlik]

@oriolgual this looks pretty interesting, could you open up a PR with your changes so we can discuss them there? Even just being able to have a Form parameters mapping would be really helpful

[activescott]

@flomotlik I'll need to dive deep here to confirm, but I do believe serverless could provide some standard syntax without requiring the serverless user to deal with velocity that will work. I have some manual changes to APIG on top of the serverless deploy that, going from memory enabled me to accomplish the following:

1. Access all HTTP headers in the _request_ for a pre-defined set of content types (APIG requires you to specify the content type up front and add a velocity template).
2. Set HTTP status code successful and error responses (APIG makes a distinction between success and error responses).
3. Set HTTP response headers.

In order to enable this on APIG, I believe serverless could do the following:

1. Add syntax in serverless.yaml to specify supported content type values for incoming requests
   
   
   
   • SLS would define specify that event.headers is always passed into lambda (event.headers is what APIG's default application/json velocity template uses) and use a default velocity template (hidden from the serverless user) to map the headers for each of the supported content type.
   
   
2. Add syntax in serverless.yaml to specify the supported http response statuses.
   
   
   
   • Each of those would be mapped to a integration response mapping.
   
   
   • Does serverless need to know which is the default success and/or default error response?
   
   
   • Does serverless need to know which status codes map to lambda error results (error returned in function callback) and which map to lambda success results?
   
   
3. Similar to 1, serverless will should add serverless.yaml syntax to specify the list of possible response content types, and and specify that event.response.headers (??) maps to the actual http response headers.

Feels like severless should be consistent with swagger wherever possible (see produces, consumes of operationObject and responseObject).

Relevant APIG references:

• http://docs.aws.amazon.com/apigateway/latest/developerguide/request-response-data-mappings.html
• http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html

[flomotlik]

@activescott to support normal POST request templates we could potentially merge the template from @oriolgual and simply include standard templates for JSON and HTTP FORM data by default. The main question for me is how many people need more than those 2, and which ones those are specifically. My assumption is that those 2 cover at least 80% of the use cases for APIG (and we can make them nice and pretty but essentially provide those as defaults).

And for response codes wouldn't it possible to define a standard template for all HTTP response codes and then simply let the user take whatever they need? So we kind of over describe what a specific user might need, but in total we simply cover everything HTTP provides.

[oriolgual]

I'll try to submit a PR today or tomorrow @flomotlik!

[flomotlik]

@oriolgual awesome thank you, looking forward to it

[Smeb]

Hi! Our team ran into a similar issue with requests with Twilio sending callback requests as application/x-www-form-urlencoded, and we implemented the logic @ac360 mentioned - loading templates from a file: smeb@e767cdfc083b788a6cd7c4bb49b7a874eb21200e.

It loads templates from a templates/ directory in the project based on markup in the serverless.yml file, and supports the default json template which is stored in the apiGateway plugin folder, which is accessed like so:

- http: POST twilio/makeCall
  requestMappings:
    - application/json: default

Which means you can choose to not use a default (I'm not sure in practice if you'd ever want this though), since we have some requests that don't need mapping. I think as an end-goal it's definitely desireable to have both the default templates and the ability to provide your own - I'm not sure this is the best solution, but it works for a small project like ours.

[flomotlik]

@Smeb that looks really interesting, could you open a PR for it. It looks like @oriolgual opened a PR in #1685 as well, could you take a look at this one to see if that might solve your specific issue as well?

Generally what I'd like to achieve is having request/response templates that cover at least 80% of use cases and once we have those in provide a way for people to add more. Your implementation with the templates folder looks really interesting for that (though I would probably let them point to specific files so they could also pull them in through dependencies)

[activescott]

@flomotlik

...simply include standard templates for JSON and HTTP FORM data by default. The main question for me is how many people need more than those 2, and which ones those are specifically.

I think those two are fine to include. I still think adding a swagger-style consumes element/attribute to the serverless.yaml http event to allow supporting arbitrary content-types make sense. For each one specified, serverlesss would just apply the same default template. I'm not sure how to quantify it, but off the top of my head some content-types used by standardized REST APIs that I can think of: application/scim+json (SCIM - my case), application/atom+xml (OData), application/graphql, and I"m sure there are more such as text/html or multipart/mixed.

And for response codes wouldn't it possible to define a standard template for all HTTP response codes and then simply let the user take whatever they need?

Sure, why not.

[flomotlik]

@activescott yeah agreed totally in favour of having a way to define new content types and set specific templates for content types. Not sure yet about the exact implementation, but this is definitely an important feature.

[fridaystreet]

There seems to be a few requests for the ability to add more configuration options for resources such as request templates and authorizers and there's lots of other aspects to API gateway that can be configured.

I'm just wondering if it would be a good interim solution to change the way custom resources are applied. Currently it seems the custom resources are added to the cf object first in initializeResources.js. This creates a couple of issues. Firstly, if you want to define custom resources that reference any of the function / api resources that serverless creates, during the initial stack creation cloudformation fails as it can't find the references, because the compile functions haven't been run during stack creation. The only way around this seems to be to comment out any custom resources if you haven't deployed the stack yet. Then add them back and run deploy again.

The second issue, is that there's no way to override the default objects as the custom resources are add first and the default objects are merged over the top. So in the interim while all the bells and whistles for configuration are built and added to the serverless config, it would seem to make sense to change the custom resource functionality to merge the custom resources after the functions and events have been compiled.

This way people are able to customise all aspects of the resources and access all features of API gateway / lambda before functionality is implemented in serverless.

I'd be happy to submit a PR if anyone is interested

[flomotlik]

@fridaystreet yup agreed on all of this and we'd definitely like a PR on this.

The first issue you mentioned (custom resources being deployed at stack creation) is definitely a bug. Custom resources should only be deployed as an update.

We also discussed the second issue you mentioned before and we actually want people to be able to merge custom resources on top of what we create, so you can actually edit it in a way you want. So this would not just be a change for now, but actually the functionality we want.

A PR would be VERY much appreciated. Let us know if we can help in any way on this and if that offer still stands. /cc @pmuens

[fridaystreet]

@flomotlik awesome, yeah sure offer definitely still stands. I've already built it anyway as loving the serverless framework, but I couldn't really proceed any further with serverless until I could modify the core resources with my own stuff.

I'll just have to read through your PR guidelines and put everything in place to tick all the boxes.

[flomotlik]

nice, let us know if you have questions @fridaystreet

[bijoutrouvaille]

To avoid a possible future bug that can be hard to trace https://github.com/serverless/serverless/pull/1685#issuecomment-237153843.

[flomotlik]

@fridaystreet just wanted to check in if you have time to open up that PR? Happy to take over from the code you guys already have and make sure its in

[fridaystreet]

@flomotlik yep sorry been a busy week. Just on my way to office now I'll do it once I get there. I also have some templates for status codes. I might chuck that in too if everyone is happy with it.

[flomotlik]

@fridaystreet thanks, looking into it

[zachheine]

No feedback on the implementation but my use case (in #1811 [closed]) was for Twilio and the ability to access the parameters from the application/x-www-form-urlencoded POST and then send back an application/xml response.

[thenikso]

My use case would be having a lambda generate a text/html page. I ended up creating the lambda in the aws web console to test if it was even possible at all (and it is).

[ghost]

+1 to add the text/html functionnality to beta3

[seanohollaren]

+1 for the ability to specify integration response status code patterns in the serverless function yml.

[wxs]

Another common use-case is Stripe's checkout.js which, like Twilio, sends application/x-www-form-urlencoded, so also needs either custom integration request body mapping templates _or_ a standard body mapping template for formencoded POST data.

I would argue that _both_ would be useful. Handling form data is very common so a standard mapping of that content type would be ideal.

[Bilal-S]

Could this be implemented as convention?
vm Templates in separate files from core serverless definitions.
e.g. function: myFunction.js
could have templates: myFunction.req.vm and myFunction.res.vm
where .req.vm is the request template and .res.vm is the response template

The issue with an abstraction layer to vm would be that the implementation vs translations vs edge case could drive many to use the vm style directly anyways. It would also mean that you learn yet another markup. I would suggest have common templates as options people can use as starting blocks and copy in. Maybe build into cli,
serverless create --vmtemplate default-request
would generate all the files for each function if not present?

[lukas-gitl]

+1 for text/html functionality

[pmuens]

Following this discussion It hink we need to agree on two powerful default templates (one for the request and one for the response) and a way to define own content types in combination with a reference to a velocity template file which will be stored on disc and referenced in the serverless.yml file.

How should the default templates look like? What are the content types of those? And how should the syntax to define a custom content type and velocity template look like?

Here a different proposals (some are taken from the discussion above) so that we can discuss it:

Proposal 1:

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
          response:
            contentType: application/json
            template: ./response.vm

Proposal 2:

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
          response:
            application/json: ./response.vm

/cc @eahefnawy @flomotlik

[flomotlik]

@pmuens in addition to the strong default templates we also need to combine those default templates with various Content types and allowing users to add additional content types.

[Bilal-S]

@flomotlik @pmuens I would like to add for consideration that systems that require configuration are in general slower to learn and adopt then convention driven systems.
I like the idea of external template files in the native dialect of APIG. On the serverless offline plugin side, we're adopting a pure convention system for this part, but we will revert back if needed to be compatible.
Currently if a function has a request or response template it is in separate file with a similar name. It is simply auto-wired when needed and no extra configuration is needed.
For example:
adding a template file into your project named:
hello.res.vm
or
hello.res.get.vm

would achieve the same thing as suggested with this:

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
          response:
            contentType: application/json
            template: ./response.vm

[pmuens]

Thanks for the discussion so far.

Edit: Just updated the comment as the previous one was incorrect...

So how might the implementation look like?

I would propose this one as it's more expressive:

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
          response:
            contentType: application/json
            template: ./request.vm

and

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
          response:
            contentType: text/html
            template: ./response.vm

And the default template will be used (with the application/json content type) if no response or request is added.

Any thoughts?

/cc @eahefnawy @flomotlik

[tappantech]

First time serverless user here, so forgive me if this question/comment has already been asked/made! Based upon what I'm seeing, the default APIG request template created by Serverless (1.0.0-beta.2) _does not_ send through the resource path (ie.. $context.resourcePath), is that right? I'm attempting to have a single handler for multiple endpoints, and have my handler be able to determine the endpoint that was called which triggered the function. I don't believe the default template maps this information (resourcePath) into the event, and therefore cannot determine which endpoint was triggered form inside the lambda function. Should $context.resourcePath be added to the default template?

[lukas-gitl]

@tappantech You can deduce the path though, e.g. if you define it as {1}/{2}/{3}

[tappantech]

@lukas-gitl Thanks for the suggestion, much appreciated. While that would work to some extent, I think that would introduce new issues such as adding a custom authorizer to one endpoint (i.e. /user/create) while another should remain open (i.e. /user/login). Both of those would match a variable based definition of the endpoint (i.e. {1}/{2}) which would force you to authorize or not via logic in the handler. I feel that the better solution would be to add the $context.resourcePath into the request template on the APIG endpoint so that the exact path would be available to the Lambda function.

[lukas-gitl]

@tappantech Very much agreed. Depending on your situation it might have been a usable workaround.

[tappantech]

@lukas-gitl It's a great suggestion, so thanks again. I might be using it as a temporary workaround where applicable until the new vm templates are introduced and/or the default template is updated to include $context.resourcePath.

[thenikso]

Would it make sense to be able to have template also as an inline property? such as like:

response:
    contentType: text/html
    template:
        - "#set($inputRoot = $input.path('$'))"
        - "$inputRoot.html"

[flomotlik]

@thenikso if you really want to inline you could do this through custom resources, but imho supporting inlining in the config is just going to lead to lots of problems with people not getting it right.

Can be nice for some cases, but will lead to tons of problems and support requests and isn't that necessary imho as a feature.

[pmuens]

The implementation just started: https://github.com/serverless/serverless/pull/1968 feel free to comment on the PR!

[pmuens]

While implementing this I just wanted to ask two few questions to you guys:

1. What content types do you want support for?
2. Do you have any proposal for the default response template (Just found this gem but maybe you have more?!
3. Any other wishes on the list so far?

[flomotlik]

Imho:

Request:

• application/json
• application/x-www-form-urlencoded
• application/xml (optional, can be added later)

Response:

• text/html
• application/json
• application/xml (optional, can be added later)

We don't need multipart/form-data or any other file upload based content types for requests as you can't do this with API Gateway anyway

List of Media types: http://www.iana.org/assignments/media-types/media-types.xhtml

[mikelax]

To follow up on @flomotlik for the application/x-www-form-urlencoded request type we should be sure we wrap the body form parameters correctly to allow for handling of POST form data.

I would also add text/plain to the Response to allow for some flexibility.

[pmuens]

We just took a deep dive into API Gateway today and came up with this definition for request / response templates:

https://github.com/serverless/serverless/pull/1968#issue-173414321

Request:
You can define the content-type and the corresponding mapping template which should be used to transform the request according to your needs. We'll provide default mapping templates for application/json and application/x-www-form-urlencoded so you don't need to configure anything here. But you are also able to overwrite the default behavior if you setup a different mapping template there.

Response:
You can configure the header your response should have. Furthermore you can define a mapping template which will be used to extract the values from the response your lambda returns. A lambda function will always return application/json. This data can be transformend with the template definition you can setup. The Content-Type header you can setup will determine how the content the lambda returns will be displayed.
A default is setup here so that the response is always application/json.

/cc @flomotlik @eahefnawy

[kennu]

There has been some discussion about use cases not handled by the Serverless default template so I thought I'd share one.

In pretty much all my protected APIs, I use this template syntax to pass a JSON object from the custom authorizer to the Lambda functions as the principalId:

"principalId": $context.authorizer.principalId,

That means I can access various fields in the Lambda as

event.principalId.userId
event.principalId.email
event.principalId.role
etc.

The Serverless default template encloses principalId in quotes, preventing the JSON from being parsed by API Gateway:

"principalId": "$context.authorizer.principalId",

So that means I'd have to add tedious boilerplate JSON parsing and error checking code to hundreds of API endpoint Lambdas.

Anyway, I guess custom templates are coming, so I can hopefully handle this with them.

[flomotlik]

@kennu that sounds like a bug in the template. This happens with V1 or V0.5?

[kennu]

@flomotlik I don't know if I'd consider it a bug, since principalId is an opaque string field and people can use it for whatever they wish. API Gateway templating just makes it possible to use it like this, and having a default template doesn't cover this kind of use cases.

[kennu]

OTOH I also think that the default template is awesome for most use cases, and many people will adopt it. It makes me wish I could maybe override just some parts of it, like this principalId case. But that may be over-engineering.

[flomotlik]

But this is a bug right, e.g. we're setting it here in the new PR: https://github.com/serverless/serverless/pull/1968/files#diff-194d3970cc4b47abac5ae6c1a66059c1R47

And if we do "principalId": $context.authorizer.principalId shouldn't this solve this issue (and actually be more correct? I don't see any point of this part "principalId": "$context.authorizer.principalId" because if the "" are an issue with it getting properly evaluated thats a bug. I don't really see any point of enclosing it in quotes anyway or what could the potential use case of that be? Imho its a bug and we can fix it :D (although have to play with it myself as well)

[kennu]

Well, many people would probably be storing a normal string in $context.authorizer.principalId, and in that case the quotes are needed. It only works without quotes if the data is JSON-formatted.

[Bilal-S]

In serverless-offline plugin we externalized the default template. Its a separate file (offline-default.req.vm) that can be changed for each project. Would a scheme like that work here as well rather than attempting to come up with a template that will satisfy all use cases?

[kennu]

Btw, I just tested that the current 1.0 default template will break if you put JSON in principalId, because it does not escape the string:

Endpoint request body after transformations: { "body": {}, "method": "GET", "principalId": "{"id":"auth0|xxxxxxxxxxxxxxxxxxxxxxxx"
Endpoint response body before transformations: {"Type":"User","message":"Could not parse request body into json: Unexpected character ('i' (code 105)): was expecting comma to separate OBJECT entries\n at [Source: [B@116ae07a; line: 5, column: 34]"}

I don't know if it's possible to escape strings in API Gateway templates. There's no guarantee what might be in principalId, so this seems pretty problematic.

[flomotlik]

@kennu can you open up a separate issue for this?

[kennu]

At least for me, it would be pretty convenient to have the default request and response templates generated as files when I create the Serverless project. Then I could just modify them as needed and redeploy.

I realize though that they will contain a lot of complexity that would be nice to hide from new users.

[flomotlik]

Implemented and merged in #1968 and #1998