Serverless-express: The server crashes with EMFILE NetworkingError

[janaz]

Hello,

I'm running an express.js app using the aws-serverless-express adapter. My app is making some requests to AWS Cognito API using the aws-sdk module. It's been in production for many months and from time to time it's crashing with the following stack trace:

{ Error: listen EMFILE /tmp/server-ck6wudgsnfo.sock
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at Server.setupListenHandle [as _listen2] (net.js:1350:19)
at listenInCluster (net.js:1408:12)
at Server.listen (net.js:1503:5)
at startServer (/var/task/node_modules/aws-serverless-express/src/index.js:152:17)
at Object.proxy (/var/task/node_modules/aws-serverless-express/src/index.js:205:14)
at exports.handler (/var/task/src/serverless/index.js:31:33)
code: 'EMFILE',
errno: 'EMFILE',
syscall: 'listen',
address: '/tmp/server-ck6wudgsnfo.sock',
port: -1 
}

It's caused by the following error in the app. EMFILE is the error code if there's no more file descriptors available.

{ 
  message: 'getaddrinfo EMFILE cognito-idp.ap-southeast-2.amazonaws.com:443',
  code: 'NetworkingError',
  errno: 'EMFILE',
  syscall: 'getaddrinfo',
  hostname: 'cognito-idp.ap-southeast-2.amazonaws.com',
  host: 'cognito-idp.ap-southeast-2.amazonaws.com',
  port: 443,
  region: 'ap-southeast-2',
  retryable: true,
  time: '2019-05-28T06:04:43.973Z' 
}

Error: getaddrinfo EMFILE cognito-idp.ap-southeast-2.amazonaws.com:443
    at Object._errnoException (util.js:1022:11)
    at errnoException (dns.js:55:15)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)

[brettstack]

I've never seen this error. Are you initializing aws-serverless-express inside or outside the handler? The only other known reason aws-serverless-express will create multiple sockets is when your application crashes (unhandled exception)

[janaz]

I checked the module that creates the lambda handler and it has been changed recently. The server creation has been moved from the global scope to the lambda handler. Unfortunately, the change hasn't been noticed in the code review and we didn't have any automated tests to detect it. I guess it's time to write a test for it.

So basically every lambda invocation was creating a new listener and after some time it reached the limit of opened file descriptors (for lambda functions the limit is 1024)

[brettstack]

Awesome. You should see some perf improvements from that change also. I have gone ahead and made a fix in v4 to clean up sockets also.

[codingyourlife]

I frequently receive the same error. Message is identical down to the line numbers.
aws-sdk: 2.419.0
serverless-webpack: 5.2.0

@janaz I don't really understand your answer. Could you be more specific on how to fix it please?

Is this an error in a framework (if yes what framework) or is the error in usage?
There altogether isn't much information about this issue but possibilities range from OS issue to DNS caching issue to this here...

[janaz]

@codingyourlife

The reason I had the issue was initializing the internal aws-serverless-express server inside the handler function like this:

const express = require('express');
const awsServerlessExpress = require('aws-serverless-express')

const app = express();
app.get('/', (_, res) => res.json({hello: 'world'}));

const lambdaHandler = (event, context) => {
  // the server is created inside the handler function
  const server = awsServerlessExpress.createServer(app)
  awsServerlessExpress.proxy(server, event, context)
}

The solution to my issue was to move the server initialization to the global scope so that it's shared across lambda invocations:

// move the server to the global scope
const server = awsServerlessExpress.createServer(app)

const lambdaHandler = (event, context) => {
  awsServerlessExpress.proxy(server, event, context)
}

[codingyourlife]

@janaz ok thank you!

I've found the issue and I guess the root was the same/similar. There were too many concurrent web3js calls and there seems to be a low socket limit on AWS Lambda (couldn't reproduce it locally, I think here it says it's just 10 concurrent non-aws requests https://docs.aws.amazon.com/de_de/lambda/latest/dg/limits.html).

But limiting the concurrency to 10 via the npm library called bottleneck fixed the issue.

The learning here was that EMFILE can be related to sockets and not just to files which made this error particularly hard to find...

[emilionavarro]

@codingyourlife

The reason I had the issue was initializing the internal aws-serverless-express server inside the handler function like this:

const express = require('express');
const awsServerlessExpress = require('aws-serverless-express')

const app = express();
app.get('/', (_, res) => res.json({hello: 'world'}));

const lambdaHandler = (event, context) => {
  // the server is created inside the handler function
  const server = awsServerlessExpress.createServer(app)
  awsServerlessExpress.proxy(server, event, context)
}

The solution to my issue was to move the server initialization to the global scope so that it's shared across lambda invocations:

// move the server to the global scope
const server = awsServerlessExpress.createServer(app)

const lambdaHandler = (event, context) => {
  awsServerlessExpress.proxy(server, event, context)
}

I had no clue it worked this way...

[JoeNyland]

@brettstack I've just got caught out by this too as I had the server initialisation in the handler function. I wonder if this needs to be highlighted as a gotcha in the README 🤔