Apollo-server: Resolved Lambda Issue Regarding Async Callbacks

Created on 4 Jan 2019 · 14Comments · Source: apollographql/apollo-server

Since I saw the line, context.callbackWaitsForEmptyEventLoop = false inside of the lambda code, I thought to myself, "Hm, I don't need to add that, cool!".

After deploying it, I found that any other method request to the server would hang if my lambda function timeout is long enough since my database connection stayed open. I'm able to simultaneously execute direct GraphQL queries, but anything else locks up (playground, options, etc).

My solution was to wrap the handler and manually set context.callbackWaitsForEmptyEventLoop:

const apolloHandler: lambda.APIGatewayProxyHandler = server.createHandler({
  cors: {
    origin: true,
    credentials: true
});

export const handler: lambda.APIGatewayProxyHandler = (
  event: lambda.APIGatewayProxyEvent,
  context: lambda.Context,
  callback: lambda.APIGatewayProxyCallback
) => {
  context.callbackWaitsForEmptyEventLoop = false;

  return apolloHandler(event, context, callback);
};

Perhaps "callbackWaitsForEmptyEventLoop" can be an option in createHandler so this hack doesn't need to be required.

🖇️ lambda

Source

Most helpful comment

cheers @braidn, following setup has worked for me. Although very weird that we have to use this hack 🤔😬

const server = new ApolloServer(config);

function runApollo(event, context, apollo) {
  return new Promise((resolve, reject) => {
    const callback = (error, body) => (error ? reject(error) : resolve(body));
    apollo(event, context, callback);
  });
}

export async function handler(event, context) {
  const apollo = server.createHandler({
    cors: {
      origin: true,
      credentials: true,
      methods: 'GET, POST',
      allowedHeaders:
        'Origin, X-Requested-With, Content-Type, Accept, Authorization',
    },
  });

  return await runApollo(event, context, apollo);
}

davidalekna on 20 Sep 2019

🚀4 ❤1

All 14 comments

@j do you have more examples of how you are doing this in your setup? Below is kind of what I am doing and it's an async callback, much like you are using. I am running into the same Lambda/hang timeout issue as you.

exports.handler = (event, context, callback) => {
  console.log("remaining time =", context.getRemainingTimeInMillis());
  console.log("functionName =", context.functionName);
  console.log("AWSrequestID =", context.awsRequestId);
  console.log("logGroupName =", context.logGroupName);
  console.log("logStreamName =", context.logStreamName);
  console.log("clientContext =", context.clientContext);

  context.callbackWaitsForEmptyEventLoop = false;

  startServer(event, context)
    .then(handler => {
      return handler(event, context, callback);
    })
    .catch(err =>
      callback(null, {
        statusCode: err.statusCode || 500,
        headers: { "Content-Type": "text/plain" },
        body: "Handlers Failed to Start"
      })
    );
};

I haven't tested GraphQL queries at the Lambda but, the playground and introspection will consume the entire timeout and do nothing. Not sure how to take your solution and fix my issue. Interested though!

braidn on 5 Jan 2019

@braidn Hmm, that's interesting.

You might have a CORS issue? Make sure you have the cors config as well as cors setup in api gateway.

Example Sam config (cors part):

Globals:
  Api:
    Cors:
      AllowMethods: "'DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT'"
      AllowHeaders: "'Content-Type, Authorization, X-Amz-Date, X-Api-Key, X-Amz-Security-Token'"
      AllowOrigin: "'*'"

And your apollo handler:

const apolloHandler: lambda.APIGatewayProxyHandler = server.createHandler({
  cors: {
    origin: true,
    credentials: true
});

Make sure you're on latest versions too.

Ever since I figured this out, Apex Ping is showing 100% uptime (before it was 0%)

j on 9 Jan 2019

@j I actually have these very settings / love that you all write out TypeScript in code examples.

As a _resolution_ here (for my issue): My code above (the async startServer code) does work! However, there is either something on the firewall of the server that's being introspected during the stitchSchema async phase or a problem with Lambda not being able to communicate with the outside world (port blocking). These issues cause the Lambda to timeout and have nothing to do with Apollo Server or using async in a handler.

Thank you for the response Jordan!

👍

braidn on 9 Jan 2019

👍1

@braidn mine was most likely due to a MongoDB connection and that since the connection was "Lambda Cached" throughout it's lifetime, that any route that doesn't use context.callbackWaitsForEmptyEventLoop = false will fail.

j on 15 Jan 2019

👍1

There was a fix for callbackWaitsForEmptyEventLoop on OPTIONS requests which landed in #2638. Anyhow, I'm curious if this issue still needs investigation and should be left open or if we can close it? 😄

abernix on 16 Jul 2019

@abernix it would still be nice if you could easily attach async bootstrap code to a lambda handler.

j on 2 Aug 2019

👍1

Hey guys, I've been trying to solve my implementation with callbackWaitsForEmptyEventLoop but it just doesn't work for me... Everything works fine except rds (db) calls, take about 10 seconds to execute. I've tried context.callbackWaitsForEmptyEventLoop = false; variations but I always get same result... Not sure if anything changed but seems like callbackWaitsForEmptyEventLoop is not taking effect on my lambda function :/

davidalekna on 20 Sep 2019

@davidalekna what happens if you setup something like:

const runHandler = (event, context, handler) =>
  new Promise((resolve, reject) => {
    const callback = (error, body) => (error ? reject(error) : resolve(body));

    handler(event, context, callback);
  });

const main = async (event, context) => {
    server = await someNewApolloServer(event, context);
    handler = server.createHandler(serverOptions);
    const response = await runHandler(event, context, handler);
    return response;
};

and in the async someNewApolloServer function set the context of the server using something like:

    context: async ({ event, context }) => {
        someContext....
        context.callbackWaitsForEmptyEventLoop = false;
    }

This should calm down your RDS call times (or it does for me)

braidn on 20 Sep 2019

🚀2

cheers @braidn, following setup has worked for me. Although very weird that we have to use this hack 🤔😬

const server = new ApolloServer(config);

function runApollo(event, context, apollo) {
  return new Promise((resolve, reject) => {
    const callback = (error, body) => (error ? reject(error) : resolve(body));
    apollo(event, context, callback);
  });
}

export async function handler(event, context) {
  const apollo = server.createHandler({
    cors: {
      origin: true,
      credentials: true,
      methods: 'GET, POST',
      allowedHeaders:
        'Origin, X-Requested-With, Content-Type, Accept, Authorization',
    },
  });

  return await runApollo(event, context, apollo);
}

davidalekna on 20 Sep 2019

🚀4 ❤1

Any updates on this? I faced the same problem and using the davidalekna trick works, but of course a more elegant solution is preferred :/
Also, I noticed that the default Playground scheme url isn't right when using stages in lambda. For instance, if I publish a lambda in "dev" staging, than the urls for the playground and the graphql server will be something like /dev/graphql. However the playground tries to fetch schema from /graphql. So I have to change that manually every time i access the Playground if is not cached. Not a big deal, but still annoying ¯_(ツ)_/¯