Serverless: Add support for AWS API Gateway Basic Request Validation

[vladgolubev]

This is a Feature Proposal

Description

AWS API GW officially supports this now - http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-method-request-validation.html

For example, it could be set as a new property somewhere in service.function.events.http.validation config object.

Blog post: https://aws.amazon.com/blogs/compute/how-to-remove-boilerplate-validation-logic-in-your-rest-apis-with-amazon-api-gateway-request-validation/

[schabert]

👍

[syang]

@pmuens This would be a great effort, and thanks for taking this as a feature (on April 12). And I am curious what the status/help-wanted label means here (April 12), does that mean you need some volunteer to implement this? (If so, I am interested further discussing this issue)

PS: here is a question I asked related this issue http://forum.serverless.com/t/can-serverless-offer-parameter-validation-capability/1919, where I listed why I feel this should be a great feature to add for Serverless team.

[DavidWells]

@syang help wanted means we would love this feature =)

Do you know if this is supported via cloudformation yet?

[iadknet]

This method uses swagger as the source, but I was able to get this kind of working without any serverless code/plugin modifications:

1. Add the API gateway extensions to your swagger doc:
   http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions.html

2. Add the following to the resources section of serverless.yml
   http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html

    ApiGatewayRestApi:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: ${self:service}-api
        Body: ${file(swagger.yml)}

This is not ideal in many ways. For one, you have to inject the lambda uri/arn into the swagger doc. Also, you have two conflicting api definitions in your cloud formation.... but it did work for me in a manual test.

Ideally there would be an option to have serverless reference the swagger doc directly for function definitinons, but dynamically inject the uri parameter when building the cloudformation.

[syang]

@iadknet The approach you listed above sounds a bit too heavy to me (not saying this is bad, maybe there is no way avoiding the heaviness). Let me elaborate my thoughts from a user perspective:

• Serverless attracted me because of it's simplicity (hide the unnecessary CloudFormation syntax and etc). For this kind of users, who prefer simplicity, learning a full Swagger + Amazon extension might not be ideal route for them. (again, I am not sure if there could be way to hide / abstract that learning curve)

[iadknet]

@syang I can see where you are coming from. We were attracted to Serverless for the way it simplified other aspects of creating lambda-based APIs. It does a lot of things that we would have had to build ourselves. This includes a large part of the delivery pipeline, a great set of local development helpers, and easy configuration of IAM roles.

On the other side we have been spending a lot of time fighting against some of the abstractions in Serverless that run counter to our established toolchains and development patterns.

Swagger is already built into our development process as we try to follow a 'design-first' development pattern, by writing the API specification first and then writing our code so it implements the design. There is a large ecosystem of swagger-centric tools that help enable this approach.

AWS also seems to be embracing Swagger/OpenAPI as the preferred method for defining API Gateway APIs.

So, in that aspect, Serverless is kind of cumbersome, because we are defining our APIs in two places, and have not been able to take full advantage of the Swagger feature set.

On the infrastructure side, we also prefer to use Terraform over CloudFormation (which I know is a whole separate philosophical battle that has been discussed to death).

So for us the Serverless project has been a bit of love/hate/frustration. The answer is we probably just need to build our own toolchain, learn how to write Serverless plugins, or hope someone else writes plugins that cover our use-cases :)

[HyperBrain]

@iadknet I fully understand your point of view. For me both ways of API definitions are valid, but different approaches.

IMO a Swagger integration to define the API can be integrated as plugin that would read the API definition from a Swagger definition and use that to create the Serverless service configuration internally.

Of course it is not an easy pick - there are lots of edge cases and general procedures that had to be defined how such a plugin would and could work, especially how to integrate lambda functions with endpoints. Writing plugins themselves has become quite easy with the latest few versions of Serverless in general.

I am sure, that such a plugin - including its planning, finding solutions, etc. can be designed, planned and created with help of the community in a quite reliable way as project/repo on GitHub. As long as there is enough demand and engagement.

[pmuens]

@syang thanks for asking!

@DavidWells already answered your question, but I just wanted to echo it again: We'd love to get some help with this feature. That doesn't necessarily mean that it needs to be in the shape of a feature complete PR. A discussion about the potential implementation is yet another way how this help can be expressed, so 👍 for chiming in and starting the discussion.

@iadknet thanks for providing a potential solution for it! 💪 And thanks a lot for the feedback in general. Feedback like this helps us to understand the pain points and guides the planning of the Framework.

It would be nice if we could get this working with core CloudFormation and w/o Swagger / OpenAPI. Our http event definition is heavily built atop of CloudFormation. Native Swagger support is not planned anytime soon (yet).

[iadknet]

I do still think an API Gateway Basic Request Validation plugin would be best served by by wrapping it in a Swagger plugin implementation.

To create a Request Validation plugin would require the following:

• Create a design that allows defining models/schema in serverless.yml
• Have each function define its response model?
• Have the ability to define a model and re-use it across multiple functions?

Once you have answered and implemented the above, you have essentially created a competing standard for Swagger, which is just a collection of API endpoint and model definitions in yml format.

Also, with AWS::ApiGateway::RestApi Swagger is valid CloudFormation... and it appears that Amazon is moving towards using Swagger to drive API definitions via CloudFormation/SAM. For example, I think Swagger might be the only way to enable request validation via CloudFormation (I may be wrong about that).

I do have some interest in tackling the creation of a Swagger plugin, but I am a pretty novice Node developer. I know enough Node to be able to write a simple Express app, or simple Lambdas, but I haven't done anything overly complex.

I did start to look into what it might take to write a plugin recently, but got bogged down by trying track down documentation around lifecycleEvents, and figuring out what I needed to hook into/override.

The functionality I was hoping to implement:

• Skip the creation of AWS::ApiGateway::Resource and AWS::ApiGateway::Method items
• Read in swagger.yml and inject the x-amazon-apigateway-integration: uri: xxxxxxx property to reference the arn for the corresponding AWS::Lambda::Function resource
• Add a Body property to ApiGatewayRestApi with the modified swagger.yml

As a bonus it would be nice to use swagger.yml as a source for local development functionality such as invoke local/offline. There are some other plugins that can take swagger.yml and generate function definitions in serverless.yml, which could be a workaround. Even nicer would be to have the plugin override the Serverless logic that reads from serverless.yml for those definitions, and reference swagger when building its internal data structures. I don't know how hard/brittle that would be.

Time and learning curve permitting, I would be interested in tacking a crack at at least some of the above functionality. Any pointers in where to start looking would be helpful.

[syang]

@iadknet @DavidWells @pmuens

Time and learning curve permitting, I would be interested in tacking a crack at at least some of the above functionality. Any pointers in where to start looking would be helpful.

Maybe some kind of meeting and design discussion session (whiteboard style) would help. I'd be happy to see how this may leads to.

[pmuens]

@iadknet thanks for the very interesting implementation proposal. Would love to see such an implementation as a plugin. We're currently looking into ways how we can enrich the discovery of plugin lifecycle events and aid plugin developer here (see #2821).

@syang

Maybe some kind of meeting and design discussion session (whiteboard style) would help. I'd be happy to see how this may leads to.

That sounds like a good idea. I'd be in for that 👍

[syang]

@pmuens I could provide a whiteboard discussion / conference facility if people are interested in this idea. If people in this thread are all in SF, we should have a casual in-person meeting + whiteboard discussion.

[pmuens]

I could provide a whiteboard discussion / conference facility if people are interested in this idea.

@syang that would work best for me 👍 . I'm not in SF right now so in-person meeting wouldn't be possible for me.

[sbkn]

Is there any further progress on this issue?

[pmuens]

Is there any further progress on this issue?

Thanks for commenting @sbkn 👍

AFAIK this issue is still in progress and needs some more discussion so that we get a final implementation proposal in place.

Do you have some feedback on the current comments and implementation proposals in this thread @sbkn?

[sbkn]

Do you have some feedback on the current comments and implementation proposals in this thread @sbkn?

I'm curious whether we can set a model defined in APIG as validation basis without using swagger at all. In the console you'd go to Method Request, set Request Validator to f.e. Validate body and add a model in the Request body - that's it.

The models needed you'd add f.e. w/ serverless-aws-documentation.

[pmuens]

Thanks for getting back @sbkn 👍

I'm curious whether we can set a model defined in APIG as validation basis without using swagger at all.

IMO only using raw CloudFormation (and not introducing Swagger / OpenAPI) would be the best solution here.

Does anyone know if API Gateway supports such a feature via CloudFormation?

[raids]

API Gateway has some extensions to Swagger (listed here: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions.html) of which request validators are included. I haven't experimented with those specific extensions, but have had success with others.

EDIT: Whoooooops - just realised you probably meant without using Swagger/OpenAPI. My bad!

[pmuens]

API Gateway has some extensions to Swagger (listed here: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions.html) of which request validators are included. I haven't experimented with those specific extensions, but have had success with others.

Thanks for the comment @raids 👍

EDIT: Whoooooops - just realised you probably meant without using Swagger/OpenAPI. My bad!

No worries! Yes, implementing this w/o Swagger / OpenAPI would be great!

[marvinpinto]

I think that request validation is now supported in Cloudformation via the Schema property: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-model.html

[0libre]

Btw do you guys know how to define and create a model an apply it to a method in the same cloudformation?

[RafPe]

@0libre I have done something like this and it creates the model perfectly fine in backend. But does not apply validators to my methods :(

resources:
  Resources:
    PetsModelNoFlatten: 
      Type: "AWS::ApiGateway::Model"
      Properties: 
        RestApiId: {Ref: ApiGatewayRestApi}
        ContentType: "application/json"
        Description: "Schema for Pets example"
        Name: "PetsModelNoFlatten"
        Schema: 
          Fn::Join: 
            - ""
            - 
              - "{"
              - "   \"$schema\": \"http://json-schema.org/draft-04/schema#\","
              - "   \"title\": \"PetsModelNoFlatten\","
              - "   \"type\": \"array\","
              - "   \"items\": {"
              - "       \"type\": \"object\","
              - "       \"properties\": {"
              - "           \"number\": { \"type\": \"integer\" },"
              - "           \"class\": { \"type\": \"string\" },"
              - "           \"salesPrice\": { \"type\": \"number\" }"
              - "       }"
              - "   }"
              - "}"

[mjmac]

I think the missing piece to this puzzle is that in combination with the model definition, request validation needs to be enabled for the method request: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-requestvalidator.html

"$schema": "http://json-schema.org/draft-04/schema#"
title: InputModel
id: "#InputModel"
type: object
properties:
  id:
    type: string
  state:
    type: string
    enum:
      - NEW
      - DOWNLOAD
      - VERIFY
      - CLEAR
      - INSTALL
      - REBOOT
      - SUCCESS
      - ERROR

After defining a model like the above, I went into the APIG console and manually turned on validation, then tested it. I got a 400 error to the client as expected, and in the logs:

(0ed9ced3-a2f2-11e7-b6eb-bbbc06515723) Request body does not match model schema for content type application/json: [instance value ("GRARFELB") not found in enum (possible values: ["NEW","DOWNLOAD","VERIFY","CLEAR","INSTALL","REBOOT","SUCCESS","ERROR"])]

[RafPe]

@mjmac Yes - the fact you can apply created resources to methods in APIG console I know. But I got quite complex API where doing it manually for every method would be pointless hence I do for now validation in code until I can figure out how to do that automatically

[mjmac]

@RafPe Sure, I wasn't suggesting that manually enabling validation was a solution - just pointing out that it does work with defined models and that there is CloudFormation support for this. I had the impression that there was some doubt that CloudFormation supported it, which is a requirement for the serverless framework to drive it. I have been exploring serverless-aws-documentation for model definition. Maybe extending it to optionally enable validation would be the path of least resistance?

[RafPe]

I would be really interested if it would be possible to hook this into cloudformation template - so the resources could be created and assigned to methods. That would make life so much easier in terms of validation and schema control :)

Will try to do some more research and report back if I find something

[mjmac]

https://forums.aws.amazon.com/thread.jspa?messageID=787077

So close! But we're not there yet. We can define validators via CloudFormation now, but there is still no way to attach a validator to an APIG method yet. The good news is that it seems like it should be possible sometime soon. When it is, perhaps the framework should support it with new directives like the following:

provider:
  name: aws
 ...
  validators:
    all:
      body: true
      parameters: true
    params:
      parameters: true
...
functions:
  function1:
    events:
      - http:
          path: /foo/bar/{baz}
          method: POST
          validation: all

This would result in generated CloudFormation like the following:

  RequestValidatorAll:
    Type: AWS::ApiGateway::RequestValidator
    Properties:
      Name: All
      RestApiId: 
        Ref: ApiGatewayRestApi
      ValidateRequestBody: true
      ValidateRequestParameters: true
  RequestValidatorParams:
    Type: AWS::ApiGateway::RequestValidator
    Properties:
      Name: Params
      RestApiId: 
        Ref: ApiGatewayRestApi
      ValidateRequestBody: false
      ValidateRequestParameters: true
  ApiGatewayMethodFunction1:
    Type: AWS::ApiGateway::Method
    Properties:
      ...
      RequestValidator: <-- DOES NOT EXIST YET, BUT SOON?
        Ref: RequestValidatorAll

[RafPe]

I would say we would need to have something a bit different. Look at the situation where we create multiple models by defining resources ( that we already can create ) but then in functions we specify the model name ( or reference to model ) so in different parts of API we can apply different schemas

...
functions:
  function1:
    events:
      - http:
          path: /foo/bar/{baz}
          method: POST
          request_model: ModelNameComesHere

btw - I think there is plugin for it - have not tested - https://github.com/9cookies/serverless-aws-documentation

[mjmac]

I think we're talking about two aspects of the same thing... I was coming at it from already having defined models via the serverless-aws-documentation plugin, but I see where you're coming from. It would be nice to have a canonical way of doing request validation, but I wouldn't want to lose the flexibility of being able to do it either way.

[RafPe]

I will try connecting those 2 and see if this will do what I need :D

[matttowerssonos]

@RafPe Any luck with using serverless-aws-documentation to attach validators? I'm looking at doing the exact same thing.

[adityamedhe-cc]

Any updates on this issue? Support for validation: all / params-only / body-only would be a great addition.

[simlu]

We'd be very interested in this as well

[risabhkumar-cc]

:+1:

[RafPe]

@matttowerssonos @mjmac Hey - been quite distracted with other projects so managed to test it today .... and it works awesomely

So in my serverless.yml I have added required variables as described on plugin page

  documentation:
    models:
      -
        name: "testus"
        description: "This is a test"
        contentType: "application/json"
        schema: ${file(apis/models/testus.json)}   

And then in method which I wanted to apply the model to

  register:
    handler: apis/user/register.register
    events:
      - http:
          documentation:
            summary: "Register user"
            description: "Registers new user"
            tags:
              - "user"
              - "create"
            requestBody:
              description: "Request body description"
            requestModels:
              "application/json": "testus"      
          path: user/register
          method: post
          cors: true
          private: true

Since I just need models that was all I added. As you noticed I need to specify JSON schema file. This one I got by using schema generator from https://jsonschema.net/

Then I just deployed my functions ... and everything was as expected. So this opens the doors for me to remove redundant validation code for post messages ;)

Hope it helps!

[db-beachbody]

@RafPe Did you find a way to attach the RequestValidator to the Method enabling the validation for this Model?

The Cloudformation for this now exists but I am unable to figure out how we can implement this using serverless because the model resource is created using serverless so I cannot add RequestValidatorId after the CF is built.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-method.html#cfn-apigateway-method-requestvalidatorid

[RafPe]

@db-beachbody I have found a way :) At the moment I have written simple plugin that will attach selected RequestValidator into selected methods. I will post the plugin repo URL once it will be in state of being usable by others ;)

But it works 👍

..... so it has not taken so long and plugin is here => https://github.com/RafPe/serverless-reqvalidator-plugin
Have been working for me without any problems. Enjoy ppl 👍

[db-beachbody]

@RafPe Thank you so much, this has saved me so much time and prevented me from having to create an ugly hack.

[cortopy]

@RafPe plugin works great, although it took me a while to discover it actually depends on documentation plugin

If someone gets to read this far, this is FULL example that works:

service:
  name: my-service

plugins:
  - serverless-webpack
  - serverless-reqvalidator-plugin
  - serverless-aws-documentation

provider:
  name: aws
  runtime: nodejs6.10
  region: eu-west-2
  environment:
    NODE_ENV: ${self:provider.stage}
custom:
  documentation:
    api:
      info:
        version: '1.0.0'
        title: My API
        description: This is my API
      tags:
        -
          name: User
          description: User Management
    models:
      - name: MessageResponse
        contentType: "application/json"
        schema:
          type: object
          properties:
            message:
              type: string
      - name: RegisterUserRequest
        contentType: "application/json"
        schema:
          required: 
            - email
            - password
          properties:
            email:
              type: string
            password:
              type: string
      - name: RegisterUserResponse
        contentType: "application/json"
        schema:
          type: object
          properties:
            result:
              type: string
      - name: 400JsonResponse
        contentType: "application/json"
        schema:
          type: object
          properties:
            message:
              type: string
            statusCode:
              type: number
  commonModelSchemaFragments:
    MethodResponse400Json:
      statusCode: '400'
      responseModels:
        "application/json": 400JsonResponse

functions:
  signUp:
    handler: handler.signUp
    events:
      - http:
          documentation:
            summary: "Register user"
            description: "Registers new user"
            tags:
              - User
            requestModels:
              "application/json": RegisterUserRequest
          method: post
          path: signup
          reqValidatorName: onlyBody
          methodResponses:
              - statusCode: '200'
                responseModels:
                  "application/json": RegisterUserResponse
              - ${self:custom.commonModelSchemaFragments.MethodResponse400Json}

package:
  include:
    handler.ts

resources:
  Resources:
    onlyBody:  
      Type: "AWS::ApiGateway::RequestValidator"
      Properties:
        Name: 'only-body'
        RestApiId: 
          Ref: ApiGatewayRestApi
        ValidateRequestBody: true
        ValidateRequestParameters: false

[RafPe]

@cortopy Thx! Great u got it working. I updated my docs on that topic as well

[jjkirby]

This is exactly what I was looking for! Does this support nested models where in swagger you have a property and you use the $ref: '#/definitions/myOtherModel' . If so what is the syntax?

[jjkirby]

Sorry. Me bad. I found the way in the serverless-aws-documentation plugin
$ref: "{{model: MyOtherModel}}"

[jjkirby]

Will this work with Java instead of Node? It looks like webpack is choking on my handler which is defined as a Java package like "handler: com.my.package.handler" . I get an error on deploy saying No matching handler found for 'com.my.package'. Check your service definition. Is this compatible or do I need some additional configuration to handle java packages?

[ShawnAukstak]

@jjkirby I'm using this in Java and haven't had any problems so far.

For example, with the package com.serverless.SampleHandler and serverless.yml:

functions:
  sample:
    handler: com.serverless.SampleHandler

Make sure you build the zip/package before deploy. This adds a buildZip task into build.gradle with the Zip name, and a package setting within serverless.yml. I'm using the Gradle template which added this for me. So I just have to use build gradle prior to sls deploy.

[jjkirby]

I figured it out. I didn't need serverless-webpack plugin because I believe that is for node based implementations. Removed that plugin and it went fine

[mcroker]

Folks, thanks for your efforts...

I am seeing behaviour where if the Content-Type is either not set or set wrongly in the HTTP request, the request is passed directly to the lambda function without validation. Where Content-Type matches the template (application/json) than validation occurs correctly.

The occurs in both lambda and lambda_proxy integration approaches.

What I am looking to achieve is to have all requests rejected (by validation) which do not correctly validate to an expected Content-Type (as defined in serverless.yml). I am using serverless-aws-documentation and serverless-reqvalidator-plugin

integration:lamba (with passThrough: NEVER)
Content-Type: application/json => validated correctly.
Content-Type: application/x-www-form-urlencoded => Passthrough (i.e. not validated)
Content-Type anything else => Unsupported Media Type (correct behaviour)

I can see from the serverless code that where integraton=lambda default RequestTemplates of 'application/json' and 'application/x-www-form-urlencoded' are added; there doesn't seem to be a way to suppress this. If this RequestTemplate is manually deleted via the console then I get the behaviour I want (i.e. all requests are validated).

I have also tried adding a requestModel for application/x-www-form-urlencoded which serverless seems to correctly load to AWSGateway but doesn't seem to correctly validate.

integration:lambda_proxy
Content-Type: application/json => validated correctly.
Content-Type anything else => Passthrough (i.e. not validated)

The lambda_proxy documentation is quite clear that anything which isn't matched is passed to the lambda function, so this isn't too surprising. I guess one approach could be to validate the Content-Type in the lambda function, but it would be a better solution for an invalid Content-Type to never reach the function.

Config files
serverless.yml
```service: sportsmgr

frameworkVersion: ">=1.1.0 <2.0.0"

plugins:

• serverless-plugin-typescript
• serverless-dynamodb-local
• serverless-offline
• serverless-aws-documentation
• serverless-reqvalidator-plugin

package:
include:
- config
- models

custom:
documentation:
info:
...cut...
models:
-
name: "ErrorResponse"
description: "This is an error"
contentType: "application/json"
schema: ${file(models/errorResponse.json)}
-
name: "ObjectID"
description: "Object ID"
contentType: "application/json"
schema: ${file(models/objectid.json)}
-
name: "TenantData"
description: "Tenant Data Object"
contentType: "application/json"
schema: ${file(models/tenant.json)}
-
name: "TenantDataUrlencoded"
description: "Tenant Data Object"
contentType: "application/x-www-form-urlencoded"
schema: ${file(models/tenant.json)}

commonModelSchemaFragments:
ErrorResponse400:
statusCode: "400"
responseModels:
"application/json": "ErrorResponse"
ErrorResponse500:
statusCode: "500"
responseModels:
"application/json": "ErrorResponse"

provider:
name: aws
stage: dev
region: eu-west-1
runtime: nodejs8.10

functions:
tenant_create:
handler: src/handlers/tenant.createHandler
events:
- http:
path: /tenants
method: post
cors: true
reqValidatorName: 'xMyRequestValidator'
integration: lambda
request:
passThrough: NEVER
documentation:
summary: "Create a new tenant"
description: "Create a new tenant"
requestModels:
"application/json": "TenantData"
"application/x-www-form-urlencoded": "TenantDataUrlencoded"
methodResponses:
-
statusCode: "200"
responseModels:
"application/json": "ObjectID"
- ${self:custom.commonModelSchemaFragments.ErrorResponse400}
- ${self:custom.commonModelSchemaFragments.ErrorResponse500}

resources:
Resources:
xMyRequestValidator:
Type: "AWS::ApiGateway::RequestValidator"
Properties:
Name: 'my-req-validator'
RestApiId:
Ref: ApiGatewayRestApi
ValidateRequestBody: true
ValidateRequestParameters: true

models/tenant.json
```{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "tenant",
  "type": "object",
  "properties": {
    "title": {
      "type": "string"
    }
  },
  "required": [
    "title"
  ],
  "additionalProperties": false
}

[saintberry]

@mcroker did you find a solution to the issue posted above?

[mudlabs]

Is this not what Custom Request Templates are about?