Serverless-express: update to nodejs10.x brings "Malformed Lambda proxy response" error

[adrai]

I updated an existing lambda function runtime from node 8 to 10 now the api gateway logs say: "Malformed Lambda proxy response"

[adrai]

When testing via API-GW UI:

Tue May 14 20:32:59 UTC 2019 : Received response. Status: 200, Integration latency: 759 ms
Tue May 14 20:32:59 UTC 2019 : Endpoint response headers: {Date=Tue, 14 May 2019 20:32:59 GMT, Content-Type=application/json, Content-Length=4, Connection=keep-alive, x-amzn-RequestId=99b0d427-cfe9-4bc4-9a1a-b4d4c392db06, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=276, X-Amzn-Trace-Id=root=1-5cdb25fa-4edb801e05b7f89b19909fa2;sampled=0}
Tue May 14 20:32:59 UTC 2019 : Endpoint response body before transformations: null
Tue May 14 20:32:59 UTC 2019 : Execution failed due to configuration error: Malformed Lambda proxy response
Tue May 14 20:32:59 UTC 2019 : Method completed with status: 502

[adrai]

Seems like the body gets not back to the api-gw... it's always null did the interface for api-gw proxy integration change?

[adrai]

My findings so far:

// still works with node 10 lambda runtime
exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context, 'CONTEXT_SUCCEED'); };
exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context); };
exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context, 'PROMISE').promise;

// does NOT work with node 10 lambda runtime anymore
exports.handler = (event, context, callback) => { context.callbackWaitsForEmptyEventLoop = false; awsServerlessExpress.proxy(server, event, context, 'CALLBACK', callback); };

[adrai]

The question now is: how to set callbackWaitsForEmptyEventLoop = false?

[adrai]

side effect with (both: node 10 or node 8 runtime) because callbackWaitsForEmptyEventLoop can't be set to false:

exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context); };

now I "sometimes" get:

ERROR   Uncaught Exception:  { Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }

But it seems to occur after the lambda function has ended (see: END RequestId....)
... and seems like the request is getting sent twice (like this would be executed twice)
_reproducable example for double execution on lambda nodejs10.x_

have to switch back to node 8 with callback signature and callbackWaitsForEmptyEventLoop set to false:

context.callbackWaitsForEmptyEventLoop = false;
awsServerlessExpress.proxy(server, event, context, 'CALLBACK', callback);

[adrai]

Seems there are more “bugs” in node 10 runtime for lambda: https://forums.aws.amazon.com/thread.jspa?messageID=901128#901128

[adrai]

fyi: I can reproduce it with: https://github.com/lambci/docker-lambda

[adrai]

I know what the problem is...

The problem is in the Runtime.js:
when callbackWaitsForEmptyEventLoop is set to false, it just calls context.succeed(result) (to early) itself with the result being null because the real result will occur a bit later itself by calling the original callback with callback(null, result). If I remove callbackContext.succeed(result); in my test docker image it works.

      if (_isPromise(result)) {
        result.then(callbackContext.succeed, callbackContext.fail);
      }
      else {
        if (callbackContext.callbackWaitsForEmptyEventLoop) {
          process.prependOnceListener('beforeExit', () => { callback(); });
        }
        else {
          callbackContext.succeed(result);
        }
      }

[adrai]

read this for more info: https://medium.com/@hichaelmart/reviewing-the-aws-lambda-nodejs10-x-runtime-84de1e500aac

[JamesKyburz]

@adrai Thanks for posting this!

I found out this today too when debugging a test lambda with https://github.com/jameskyburz/sandbox-debugger

Not what people would expect that the new official node 10 runtime doesn't support async lambdas using the callback signature.

[thomasmichaelwallace]

FWIW I did flag a related issue to this to one of the more senior lambda team (https://twitter.com/tomincode/status/1131567879659425793)- but there's never any visibility with issues like this- so it's anyone's guess if/when it gets fixed.

[adrai]

Seems AWS updated their nodejs10.x runtime... closing for now.