Middy: HttpResponseSerializer middleware

[lmammino]

I believe it would be awesome to have a HttpResponseSerializer middleware that would be able to use different serialization mechanisms based on the current content negotiation

e.g. if the user prefers JSON, the response object is automatically serialized to JSON, if the user prefers XML, then the response object is automatically converted to XML.

The middleware should be able to handle even error paths and serialize errors accordingly.

[noetix]

How would status codes work? Assuming you don't want 200 for every non-error response.

[MaxVynohradov]

Hi! I'm interested in this - what do you think about https://www.npmjs.com/package/fast-json-stringify ?
Fastify framework use this with success.
@noetix what do you think? Yes, it will not support XML, etc. but must be great for REST API.

[noetix]

@MaxVinogradov

How you transform data is probably the easier part to this issue.

If you want to use fast-json-stringify you could modify the middleware I wrote:

export default () => ({
  after: (handler, next) => {
    handler.response = {
      statusCode: 200,
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(handler.response) // change this
    }

    next()
  }
})

The bigger issues are the fact that you're always returning 200 and always JSON.

Some customisation should be added for status codes -- those that want 201 on POST success for example.

Data transformation should be pluggable (cause JSON, XML, PDF, CSV, etc, etc) and each would likely need its own configuration. Our use-case here is a PDF transformer that needs to point to a particular template to facilitate transformation.

[noetix]

I have some middy work coming up and may have some time to contribute.

Here is what I was thinking for a response serializer. The below is the @middy/http-content-negotiation example refactored using the proposed.

import middy from '@middy/core'
import httpContentNegotiation from '@middy/http-content-negotiation'
import httpHeaderNormalizer from '@middy/http-header-normalizer'
import httpErrorHandler from '@middy/http-error-handler'
import httpResponseSerializer from '@middy/http-response-serializer'

const handler = middy((event, context, cb) => {
  let body

  switch (event.preferredLanguage) {
    case 'it-it':
      body = 'Ciao Mondo'
      break
    case 'fr-fr':
      body = 'Bonjour le monde'
      break
    default:
      body = 'Hello world'
  }

  return cb(null, {
    statusCode: 200,
    body
  })
})

handler
  .use(httpHeaderNormalizer())
  .use(httpContentNegotiation({
    parseCharsets: false,
    parseEncodings: false,
    availableLanguages: ['it-it', 'fr-fr', 'en'],
    availableMediaTypes: ['application/xml', 'application/yaml', 'application/json', 'text/plain']
  }))
  .use(httpResponseSerializer({
    'application/xml': ({ body }) => `<message>${body}</message>`,
    'application/yaml': ({ body }) => `---\nmessage: ${body}`,
    'application/json': ({ body }) => JSON.stringify(body)
  }))
  .use(httpErrorHandler())

module.exports = { handler }

So the basics are that the response serializer:

• Configure it to handle specific content types
• Route responses to the correct serializer using the request's accept header
• 
  
  • If no serializer is found (like 'text/plain') the response object is passed through
  
  
• Executes the matched serializer
• 
  
  • If the result of the serializer is a string, it replaces the body
  
  
• 
  
  • Otherwise the result replaces the response object -- allow for header manipulation or things like isBase64Encoded on APIG
  
  
• Attaches the content-type header to the new response

The middleware is purposely dumb to the specifics of serialization because it would be impossible to offer all content types, every configuration, customisation, etc.

Unsure whether to call it a serializer or a transformer.

Thoughts? @lmammino

[lmammino]

Love the proposal @noetix!

Just a few comments:

• It would be nice to have an option to override the negotiated output format. Maybe if the response is already set and it has a content-type header defined, that should be used instead (handler logic or other middlewares are forcing the selection of a given content-type). With this approach this should be the decision logic for the format:

  • If a response with a content-type is set (and there's a serializer for it), use that
  • If no content-type but there's a preferredContentType in the request (and there's a serializer for it), use that
  • default case (or no serializer for the selected type), leave the response body unchanged.

• There are cases where the preferred content type is sent with additional parameters. Eg.:

  • Accept: application/json; version=1
  • Accept: application/vnd.myapplication.user.v1+json

  To address all the cases, we should probably do some smart matching (maybe a prefix matching).

[lmammino]

Also this library might be an interesting reference: https://github.com/fastify/fastify-accepts-serializer

They use regexes to do the accept header matching

[noetix]

Awesome feedback @lmammino.

Hinting

I would expect if content-type was already set that the content is already in that format. As in the handler did the serialization itself (or another middleware.)

How about this order?

• requiredContentType response parameter -- handler forces a content type
• Accept from the request
• preferredContentType response parameter -- handler prefers a content type over the default
• default serializer config

If there is a result and is not a valid serializer a 406 response will be returned.

If there is no result (none of the above are set) it passes through.

Accept parsing

I'm also in favour of following the style of fastify-accepts-serializer (and webpack). Even though its more verbose its more explicit, and I expect this configuration will be abstracted in most user-land code.

Example:

httpResponseSerializer({
  serializers: [
    {
      regex: /^application\/xml$/,
      serializer: ({ body }) => `<message>${body}</message>`,
    },
    {
      regex: /^application\/yaml$/,
      serializer: ({ body }) => `---\nmessage: ${body}`
    },
    {
      regex: /^application\/json$/,
      serializer: ({ body }) => JSON.stringify(body)
    },
    {
      regex: /^text\/plain$/,
      serializer: ({ body }) => body
    }
  ],
  default: 'application/yaml'
})

[noetix]

The other thing to discuss is serialization of error responses.

Would it simply be a matter of attaching the middleware after httpErrorHandler?

[lmammino]

@noetix, very good points. Great conversation.

Good point about content-type being already set and considering the body already serialized. I actually agree now on this approach.

I also like your selection order and error management proposal.

Regarding the syntax, I would prefer the slightly more verbose approach suggested by fastify.

On the last point (errors serialization), your proposal might just be enough, but I guess it's easier to validate once we have a first implementation to play with. I would include this case among the unit tests and also I would make sure is properly documented.

Thanks again for your efforts so far!

[lmammino]

Done by @noetix in https://github.com/middyjs/middy/commit/954f71fb13ff40be65fe8d80c27a3bd96435cfa4 !

THANK YOU 🎉

[roni-frantchi]

Hi @lmammino , thanks for pushing this middleware in, it's just what we're after.
I'm having issues getting it to work because of that skip when content-type is set bit -
Say I'm POSTing if my request looks like his:

POST http://localhost:8080/resources
Accept: application/json
Content-Type: application/json
###

It seems as if since the content-type is set on the request (since I'm POSTing sending a JSON body), it is kept on the response and then http-response-serializer skips serialization (even though I wish to also return JSON.

The header doesn't seem to be set by any other middleware, it's just that middy keeps it.
I can add a middleware to clear the headers... but I don't think that's the best approach.

[noetix]

@roni-frantchi This is definitely a bug on my part, apologies.

I have a PR in awaiting review: https://github.com/middyjs/middy/pull/335