Middy: Middy cause a timeout with async lambda (but only on AWS, not on my machine)

[marianoc84]

I had a problem with Middy 1.0.0 and NodeJS 12.x. I was trying to connect to Redshift using dbManager, like this:

const middy = require('@middy/core')
const dbManager = require('@middy/db-manager');

const lambda = middy(async (event, context) => {
    const { db } = context;
    const records = await db.distinct('XXX').from('YYY');

    // print result
    console.log(records);

    // return result
    return {
        'statusCode': 200,
        'body': JSON.stringify(records)
    }
});

lambda    
    .use(dbManager({
        config: {
          client: 'pg',
          connection: {
            host: 'XXX',
            port: '1111',
            schema: 'public',
            user: 'XXX',
            password: 'XXX',
            database: 'XXX'
          }
        }
    }));

module.exports = { lambda }

This was causing Lambda to get stuck, until timeout kill it. I start to searching through the issues of this repo and I've found that set context.callbackWaitsForEmptyEventLoop to false. So I introduced the WaitsForEmptyEventLoop middleware and it works but... it only works on my machine! :-)

Yep, joke apart, if I'm executing the sam local invoke FunctionName everything works: the local Lambda invoke correctly the remote redshift and so it ends successfully. But, once I run sam deploy the same code, the same build still produce a timeout when real Lambda is executed.

Why this problem?
Any idea?

Bye

[tyvdh]

I too am wrestling with this just now.

Exploring this package atm:
https://github.com/middyjs/middy/tree/master/packages/do-not-wait-for-empty-event-loop

Looks like it works when set to:

.use(doNotWaitForEmptyEventLoop({
  runOnBefore: true,
  runOnAfter: true,
  runOnError: true
}))

[tyvdh]

I forgot to close some postgresql pools which caused the event loop to stay full. Adding the do-not-wait-for-empty-event-loop package solved the issue but I really need to ensure I'm closing my pg pools 😄

[marianoc84]

I forgot to close some postgresql pools which caused the event loop to stay full. Adding the do-not-wait-for-empty-event-loop package solved the issue but I really need to ensure I'm closing my pg pools 😄

How did you close the pool? I've searching for this, but I haven't found the command. Anyway this doesn't explain why it blocks on a real lambda and not locally. I mean the pool is the same.

[lmammino]

Hey @sdomagala can you have a quick look at this? Maybe we can improve the documentation of the middleware to address common issues...

[marianoc84]

I'm trying to figuring out how to destroy the connection pool and db, now my try blocks looks like this:

const { db } = context;
console.log("context.callbackWaitsForEmptyEventLoop: " + context.callbackWaitsForEmptyEventLoop); // false

const records = await db.distinct('XXX').from('XXX');
db._context.client.pool.destroy();
console.log("db._context.client.pool.destroyed" + db._context.client.pool.destroyed); // true
db.destroy();

When I run it using sam local invoke I have false for context.callbackWaitsForEmptyEventLoop this execute the query then I call pool.destroy()but when I run sam deploy and execute on the real Lambda platform it still gets timeout.

[willfarrell]

There is a forceNewConnection option that will trigger the destroy function of the client. I've also added a not in around event loop handling to the docs.