Serverless-application-model: Add support for defining Binary Support to API Gateway

[ebekker]

It would be nice to be able to define the Binary media types for an API Gateway resource in SAM.

[dinvlad]

FWIW, it is configurable through inlined or referenced Swagger template:

x-amazon-apigateway-binary-media-types:
  - 'image/png'

[sanathkr]

@ebekker Do you want binary data type for implicit APIs?

[ebekker]

Thanks @dinvlad -- I haven't had a chance to try that out, but will shortly.

[ebekker]

@sanathkr -- I believe so, right now the default behavior is for all the generated plumbing (API GW endpoint, etc) assumes that the response content will be text-based, like JSON response, but if you want to have an function that returns binary data, like an image or a ZIP file or whatever, you need to define all the binary media type setup for API GW.

[dinvlad]

In addition to that, one of the major limitations is of API Gateway that the clients must specify an Accept header for their types, and more importantly, those headers must have a specific format: a binary media type supported by API Gateway must come _first_ in that header. We also have to explicitly add all of the possible binary types that a client can send, which is hard because we don't always have control over how different browsers mangle these headers. E.g. they may send Accept: image/jpeg image/png ... but if we only support image/png we're out of luck (the API integration will be broken).

Another problem is that even if we want to use JSON but would like to binary encode it (most commonly, for compression), we'd then need to add application/json to this list. And then the clients' Accept headers must start with application/json.. even for cases where they don't support compression (which should be optional!). Otherwise, they get a broken response that they are unable to process.

Our current workaround for all of that is to in fact set the binary types to */*, i.e. to treat all of our request/responses as binary. In this special case we don't have to specify an Accept header anymore (woohoo!), but then our requests come in forcefully base64-encoded, so e.g. in our Lambda we get a body that we then need to decode, along with isBase64Encoded flag set to true. This is easy enough to detect and handle however, at least in Node.

The downside with the current Lambda proxy integration is that this is the only way we can use it at the moment, i.e. unlike for some other integrations we cannot for example instruct API Gateway to CONVERT_TO_TEXT integration requests for these methods (we can in fact set that option using API Gateway management API or AWS CLI, but it has no effect; neither is CONVERT_TO_BINARY).

Not sure how all of these issues relate to SAM specifically, as they seem to be the current limitation imposed by APIG. It'd be best if the APIG team resolved them first before we need to do anything custom over SAM (even if it was possible).

[ebekker]

@dinvlad -- I agree, this is one area where Lambda + API Gateway is falling short, especially when compared to similar services in competing platforms. I imagine AWS will eventually improve the story for Web-based calls to Lambda.

For my use case, I've done the exact same workaround as you (*/*) and that's worked well enough for me for now, but it does feel like a very clumsy and kludgy way to approach this.

In my case I'm using the C# support, specifically the Amazon.Lambda.AspNetCoreServer support from AWS, and they've added some functionality in that package to better handle this specific situation.

The Binary media types can currently be defined on API Gateway via the console or API, but they cannot be described using the SAM application model definition, which translates to a CloudFormation script and that's what I was asking for in this ticket.

[rpgreen]

Thanks for the feedback!

@dinvlad - it's not recommended to use API Gateway to serve content directly to web browsers. API Gateway is designed to be used with a client that you have control over and can set headers (i.e. Accept) as are appropriate for your use-case. You may be able to avoid this workaround by implementing a simple static site that makes requests to your API Gateway endpoint.

@ebekker - Currently you can define binary media types in Swagger, which can be used in conjunction with SAM. Curious, how would you like this to work?

[dinvlad]

@rpgreen - I'm not serving 'regular' HTML/images content, it rather serves JSON request/responses which I can call from JavaScript. This is perfect for SPAs where the front-end can consist entirely of static files (served e.g. from CloudFront). Or is this use case not appropriate either (I'd be puzzled)?

So for this case, the only reason for enabling binary support for us is to serve compressed requests/responses. That is, it is JSON that gets compressed. Or rather, we compress only responses as it is not possible to control the Content-Encoding header from JavaScript (it is one of the 'forbidden' headers), nor it is necessary (our requests are usually disproportionally smaller than responses).

I've only talked about image/png above as an example, sorry if it lead to confusion.

Also, the workaround with

x-amazon-apigateway-binary-media-types:
  - '*/*'

in Swagger has worked well for us so far, but I'd be curious if a better solution exists (and I'd prefer to communicate directly with API Gateway from my client, or it is not serverless anymore!)