Serverless-express: ERROR: aws-serverless-express connection error with every requests

[hoangsetup]

"aws-serverless-express": "3.2.0",
[email protected]

It working fine in yesterday.

CloudWatch log
a0c895f5-fe7c-11e8-b3ae-4f8962debec2 ERROR: aws-serverless-express connection error
1544667186190,/aws/lambda/function,2018-12-13T02:13:06.190Z a0c895f5-fe7c-11e8-b3ae-4f8962debec2 { Error: socket hang up at createHangUpError (_http_client.js:299:15) at Socket.socketOnEnd (_http_client.js:391:23) at emitNone (events.js:111:20) at Socket.emit (events.js:208:7) at endReadableNT (_stream_readable.js:1064:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9) code: 'ECONNRESET' }

[brettstack]

Did you change anything on your end? e.g did you modify your API Gateway or Lambda Function?

[hoangsetup]

@brettstack No, nothing change. We have 2 stages on aws, both gets the same issue.

[keitakn]

The same error is occurring in our service.

"aws-serverless-express": "3.3.5",
"serverless": "1.32.0",

[brettstack]

Thanks for reporting. If you could provide additional information such as how you're implementing the handler (e.g is it wrapped in a promise?); Which version of Node.js runtime you're using, that might help.

[hoangsetup]

@brettstack
this is my handler, LambdaNode ver 8.10, I am using Typescript

exports.handler = (event, context, callback) => {
  try {
    context.callbackWaitsForEmptyEventLoop = false;
    context.succeed = (response) => {
      callback(null, response);
    };
    console.log("START REQUEST", event);
    return awsServerlessExpress.proxy(server, event, context);
  } catch (e) {
    console.info("===> Handler Error: ", e);
  }
};

[keitakn]

@brettstack
We are using the runtime of Node.js 8.10.

Here is our handler.

We also use TypeScript.

const server = awsServerlessExpress.createServer(app, undefined, binaryMimeTypes);

export const expressHandler = async (
  event: lambda.APIGatewayEvent,
  context: lambda.Context,
  callback: lambda.Callback,
) => {
  context.callbackWaitsForEmptyEventLoop = false;

  return new Promise((resolve) => {
    awsServerlessExpress.proxy(
      server,
      event,
      {
        ...context,
        succeed: process.env.IS_OFFLINE ? context.succeed : resolve,
      },
    );
  });
};

[hoangsetup]

Only request include Authorization in header has been crash.
my authorize function is working fine.

[brettstack]

Thanks for the additional details. Is it 100% failure rate for those?

[hoangsetup]

@brettstack Yup! it is 100%

[brettstack]

Which region(s)?

[brettstack]

I notice you both use context.callbackWaitsForEmptyEventLoop = false; and are overriding context.succeed. Might be relevant. We are investigating.

[hoangsetup]

@shwetaskatdare My region is ap-northeast-1

I am going to try remove callbackWaitsForEmptyEventLoop setting

[keitakn]

@brettstack Our region is ap-northeast-1.

We are using aws-serverless-express in multiple applications.

Some applications do not use context.callbackWaitsForEmptyEventLoop = false;.

The following is one of them.

import lambda from 'aws-lambda';
import awsServerlessExpress from 'aws-serverless-express';
import next from 'next';
import app from './app';

const appNext = next({ dir: './src', dev: false });
const server = awsServerlessExpress.createServer(app(appNext));

module.exports.handler = (event: lambda.APIGatewayEvent, context: lambda.Context, callback: lambda.Callback) => {
  appNext.prepare().then(() => {
    return awsServerlessExpress.proxy(server, event, context);
  });
};

This application is running with Next.js and aws-serverless-express.

This application also worked until yesterday, but aws-serverless-express connection error came to occur.

[fukuiretu]

Hi!

In our environment we run Nuxt.js but it stopped working with a similar error.

2018-12-13T03:29:15.120Z    4411feb2-fe87-11e8-a498-f1e00a009a2b    ERROR: aws-serverless-express connection error
2018-12-13T03:29:15.120Z    4411feb2-fe87-11e8-a498-f1e00a009a2b    { Error: socket hang up

Since erroneous response did not occur when accessing by erasing Cookie, I guess that the HTTP Request Header is involved.

region: ap-northeast-1
runtime: nodejs8.10
aws-serverless-express: 3.3.5

Partial snippets of code

// app.js
const express = require('express')
const { nuxt } = require('./core/nuxt')

const app = express()

app.use(nuxt.render)

module.exports.app = app

// handler.js
const awsServerlessExpress = require('aws-serverless-express')
const { app } = require('./app')

const server = awsServerlessExpress.createServer(app)

module.exports.handler = (event, context, callback) => {
  if (process.env.NODE_ENV !== 'production') {
    console.log('[info]', 'Event', JSON.stringify(event))
  }
  awsServerlessExpress.proxy(server, event, context)
}

I hope it becomes helpful 🙏

[hoangsetup]

@brettstack You can take a test on AWS env with a request what include a long header like that

{
"Content-Type": "application/json",
"testheader": "suntinculpaquiofficiadeseruntmollitanimidestlaborum.eyJzdWIiOiJlMTg4YTJiZC0zNzAzLTRhMDYtODA2YS0yYzU2OTNjODlhZjQiLCJldmVudF9pZCI6IjRiYTgxN2UzLWZlYWItMTFlOC1hZTA2LTIzMTY1ZWVkMmYzYSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXdzLmNvZ25pdG8uc2lnbmluLnVzZXIuYWRtaW4iLCJhdXRoX3RpbWUiOjE1NDQ2ODcyMjksImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC5hcC1ub3J0aGVhc3QtMS5hbWF6b25hd3MuY29tXC9hcC1ub3J0aGVhc3QtMV9FNXJuZ1RGZXYiLCJleHAiOjE1NDQ2OTA4MjksImlhdCI6MTU0NDY4NzIyOSwianRpIjoiNTFkMzU0YzItZDMyZC00NWRmLTlhYzUtYzU4MjJjOTc3ZmNlIiwiY2xpZW50X2lkIjoiMWhqa3A0ZGRqZTN1aWVjbDcxcDU1amIyNDYiLCJ1c2VybmFtZSI6InRoaW5oYmcifQ.PzbFkSa11PvIbM-bVQ94YGpqr1U2vu5H2lWeknUSGrhad2ptwKKm_nj6lOHdmSjgkrzEGRcd8mvHsjmKHidq68bC5tG_R3wNBstbV9luu_SeM-3WMIZ3PyXzIpszbQKdzbz3o3pXj1in6E1RUb7C-wgob9r6GbmVIVqtND1Bbxk4RFopt4nUU4VmfizOB0e_anKq8qMKNS6FMYVMxr_6xB-E0TB1fraDLur1pAP9ilpJALgfYX1nSzEMnSi6VgtDrXcuGjEfIUSG49kf6LOsWCJUD2-qF_X3xoS9TCIIY_Jmsv_Qe1CicI6A-i0n0XbziYcMQFC6tt3Mhfz6TsNtAg"
}

testheader header value is a string with length is 937

[tobinbc]

It is also 100% failure for us as of today (13/12/2018), no code changes since yesterday. Using Amplify library to create the lambda function / express.

I am using eu-west-1 region for this api gateway, lambda, dynamodb.

My current 'workaround' is switching all lambdas to use Node 6.10 (they are now working).

[SaMnCo]

Same here, with nodejs 8.10, version 3.3.5 in eu-west-1, also no code change.

100% failure rate, app down. We are going to test a few things from now and report if we find a work around.

[j0k3r]

Same issue here since few hours.
nodejs 8.10, eu-west-1, 3.3.5 too.

Are you guys using your Lambda behind API Gateway (and using a JS frontend, like React)?
It looks like here when we send a request using Chrome (version 70 or 71) it fails. Firefox is fine.

Looks like when cookies are defined, errors appears.

Can't this be a change related to AWS Lamba itself? :disappointed:

[0xR]

Same here in EU-WEST-1

[cipriancaba]

we have 3 stages deployed in eu-west-1.. only staging works.. Using API Gateway with Cloudfront

[kkarimi]

Same here with 3.3.5 in eu-west-1. TypeScript -> node 8.10

[asiniscalchi]

Same here, with nodejs 8.10, version 3.3.5 in eu-west-1.

[RossCasey]

Same here, with nodejs 8.10, version 3.3.5 in eu-west-1 as well

[peterservisbot]

Same, with nodejs 8.10, version 3.3.5 in eu-west-1! 👎

[licalac]

Same here eu-west-1b !

Ionic 3 =>API Gateway => Lambda = DynamoDB was working like a charm :(

[cipriancaba]

Is everyone here using context.callbackWaitsForEmptyEventLoop = false ?

[0xR]

So I guess we'll receive service credit for this outage? This is a long time for a major outage.

@brettstack Is anyone investigating what changed on AWS side?

[cianclarke]

Minimally reproducible testcase - upload this lambda:
Archive.zip

With this invoke payload:
https://gist.github.com/cianclarke/7a2d284747f6323e348f0012401f566b

Also reproducible locally, but using Node 8.14. Run testcase.js in same repo:

Cians-MacBook-Pro-2:deleteme cianclarke$ nvm use 8.14
Now using node v8.14.0
Cians-MacBook-Pro-2:deleteme cianclarke$ node testcase.js
9781
h1
h2
h3
h4
h5
h6
h7
h8
h9
h10
h11
reqerr
{ Error: write EPIPE
    at WriteWrap.afterWrite [as oncomplete] (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }
ERROR: aws-serverless-express connection error
{ Error: write EPIPE
    at WriteWrap.afterWrite [as oncomplete] (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }
succeed
{ '0': { statusCode: 502, body: '', headers: {} } }

[david4City]

Same here with a TS +Babel => node 8.10 config. Running on eu-west-1.

[ylynn]

We involved AWS Lambda to investigate. They are working on a fix.

[johnrellis]

anyone start working?

[johnrellis]

so looks like things are starting to come alive in eu-west-1 again for us anyway

[0xR]

Seems to be resolved? Anyone have any information what happened? Does this count as an official outage? Will it happen again?

[brettstack]

Thanks everyone for reporting and providing instructions which helped Lambda diagnose the problem. We'll provide a detailed response after further investigation.

[yokkoyokko]

working in ap-northeast-1. Thank you.

[asendia]

We had similar issue with lambda at (UTC+7) 03.38-06.13 AM and 09.49-10.13 AM today.
Node version: 6.10
Region: ap-southeast-1

[westmark]

Looking forward to the response. The "socket hang up" error seems to stem from the UNIX domain socket between aws-serverless-express and the actual Express app.

[gpolaert]

Same here, eu-west-1.

[SaMnCo]

It's resolved for us, thanks for fixing the issue, we are looking forward to the report about the root cause.

[david4City]

Same here, no report from AWS yet either.

[Morisf]

any news regarding this issue?

[slaat]

@brettstack Are there any news regarding this issue? I just want to make sure how I can avoid this in the future

[fgcui1204]

@brettstack any update for this issue? we have a similar issue now.

[keetonian]

@fgcui1204 what's the similar issue? Is it a new regression?

[fgcui1204]

@keetonian

ERROR: aws-serverless-express connection error
{ Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:66:16) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }

[jiankafei]

@all Has everyone's problem been solved yet? How to solve it?

[wallzero]

I think I am having the same issue - but at the moment I am only running locally. My application works fine with express alone, but when I run [email protected] with [email protected] and [email protected], it fails when my cookies total around ~1426 bytes or total header size of 1.771 kilobytes. At that size, it seems to fail intermittently. Any larger, it always fails; any smaller it seems to always run.

Have the fixes in AWS been reflected in the mentioned libraries? For now I just have to try and reduce my header size via cookies.

[cianclarke]

@Schlesiger What version of node are you running locally? 8.14 and greater seems to cause this, but IIRC only with far larger headers
https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V8.md#notable-changes-2

[wallzero]

@cianclarke I am running locally with my distro's provided version 11.7.0.

[zoellner]

running into similar issues on local development. Have isolated it to be related to the cookie size in our case (maybe overall header size?)

[zoellner]

I'm now able to reproduce the error consistently

After line https://github.com/awslabs/aws-serverless-express/blob/cb3b17d5fcfad911669e33433c191bb558945ed9/src/index.js#L136
i.e. when the request is initialized in forwardRequestToNodeServer, I watch the size of the header. Whenever JSON.stringify(req.getHeaders()).length >= 8180 the request fails with one of the errors below.

It might depend on the request itself a little bit, but the result seems to be consistent with https://nodejs.org/docs/latest-v8.x/api/http.html#http_http_maxheadersize which defaults to 8KB.

Most of the header size comes from headers['x-apigateway-event'] and headers['x-apigateway-context'], i.e. the hack to attach the event context to the request.

Possible Errors:

{ Error: socket hang up
    at createHangUpError (_http_client.js:342:15)
    at Socket.socketOnEnd (_http_client.js:437:23)
    at emitNone (events.js:111:20)
    at Socket.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9) code: 'ECONNRESET' }

{ Error: write EPIPE
    at WriteWrap.afterWrite (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }

[neo-headz]

I get this error intermittently after switching my API Gateway's custom domain to a regional endpoint. This is to support a multi-region, latency-based routing policy. Maybe the headers are different between an edge optimized and a regional endpoint?? @zoellner I'm using your fork and it seems to fix this issue. Thanks!

[asurinsaka]

02/22/2019 [email protected], no serverless module used.
I get this error today.
After I tried to change the header in API Gateway from json to html and updated the zip file in lambda, then I got the error.

[willwile4]

Also got this error. I was able to work around it by changing my node version from v8.15 to v8.10. Not exactly the best fix, but it will allow you to get your work done.

[mbenedettini]

I've run into this with serverless-offline, trying to upload a file via a POST. Ended up switching to https://github.com/dougmoscrop/serverless-http with which this error is not showing.

[dpurrington]

This is still happening to me ~ .5% of requests. Not good. Using serverless-http fixed the issue for me as well.

Maybe load-related. I'm load testing using artillery:

   3   │   http:
   4   │     pool: 40
   5   │   phases:
   6   │     - duration: 10
   7   │       arrivalCount: 100
   8   │   defaults:
   9   │     headers:
  10   │       "Content-Type": "application/json"
  11   │ scenarios:
  12   │   - flow:
  13   │     - loop:
  14   │       - post:
  15   │           url: "/v1/events/query"
  16   │           json:
  17   │             uid: 1
  18   │       count: 200

[adrai]

Someone already tried nodejs10.x ? https://github.com/awslabs/aws-serverless-express/issues/234#issuecomment-492559413

[david4City]

Hi, sorry to up this again, I'm running a TS +Babel => node 8.10 config on eu-west-1 and I'm currently getting this error :

ERROR: aws-serverless-express connection error
{ Error: socket hang up
at createHangUpError (_http_client.js:299:15)
at Socket.socketOnEnd (_http_client.js:391:23)
at emitNone (events.js:111:20)
at Socket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickDomainCallback (internal/process/next_tick.js:218:9) code: 'ECONNRESET' }

Am I the only one concerned ?

[seeheee]

I try to link the dialog flow and AWS using the lambda function, but an error occurs.
Tell me how to fix it.

{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'aws-serverless-express'",
"trace": [
"Runtime.ImportModuleError: Error: Cannot find module 'aws-serverless-express'",
" at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
" at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
" at Object. (/var/runtime/index.js:36:30)",
" at Module._compile (internal/modules/cjs/loader.js:776:30)",
" at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)",
" at Module.load (internal/modules/cjs/loader.js:653:32)",
" at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",
" at Function.Module._load (internal/modules/cjs/loader.js:585:3)",
" at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)",
" at startup (internal/bootstrap/node.js:283:19)"
]
}

[adrai]

@seeheee probably your uploaded lambda function does not include the aws-serverless-express dependency

[seeheee]

How do you include it?

[adrai]

@seeheee how do you deploy your lambda function?
via AWS UI?
Claudia.js?
serverless?
CloudFormation?
SAM?

[seeheee]

I deploy your lambda function via AWS UI

[adrai]

so probably you haven’t included node_modules/aws-serverless-express

[seeheee]

I don't know how to include a moduleㅠㅠ

[adrai]

Create a zip of your whole app

Il giorno 29 lug 2019, alle ore 14:22, seeheee <[email protected][e-mail]> ha scritto:

I don't know how to include a moduleㅠㅠ

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/awslabs/aws-serverless-express/issues/199?email_source=notifications&email_token=AAIJF4UVWYYBL5AEYOZBRRLQB3ORXA5CNFSM4GKDTF42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3ARCKQ#issuecomment-515969322, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAIJF4VSEHHY5DCZ5V5FFFLQB3ORXANCNFSM4GKDTF4Q.

[svarughese91]

any resolutions for this? I am using node 10.X and I am getting this issue as well. I am using serverless deploy to deploy the lambda function. Any suggestions?

[dineshrawat11]

Same issue!! Any resolution?

[emmanuel-omegaalexis]

I'm seeing the same issue with node10. No code changes just move the lambda function from 8.10 to 10. 8.10 is EOL soon. Does anyone have a resolution?

[emmanuel-omegaalexis]

I'm seeing the same issue with node10. No code changes just move the lambda function from 8.10 to 10. 8.10 is EOL soon. Does anyone have a resolution? Thanks

[adrai]

perhaps give serverless-http a try....
Seems also more performant: https://github.com/fastify/aws-lambda-fastify#%EF%B8%8Fsome-basic-performance-metrics

[zoellner]

@emmanuel-omegaalexis you mentioned previously you're having issues with my fork. Can you let me know what the issue is and a snippet showing how you're calling the library?
I'm using my fork with node 8/10/12 without any problems. Note that my fork doesn't have any of the legacy resolver modes but just the promise mode.

[emmanuel-omegaalexis]

Hi @zoellner, I have seen others have had success with your fork so I assume it is my implementation.

const awsServerlessExpress = require('aws-serverless-express');
const app = require('./app');
const server = awsServerlessExpress.createServer(app);

 exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context, {
     apiGatewayEventWhitelist: ['requestContext']
     });

I changed to serverless-http and it is working @adrai Thank you.

[mickey2012ex]

i encountered this issue when running sls offline

finally resolve it by installing aws cli
After installation go to cmd, type "aws configure"
enter AWS Access Key Id,AWS Secret Access Key, region
And the error is gone, the key value can be fake

See if it can help

[barakd]

Had the same issue, for me the issue was quite a tricky one, so I hope I'll save someone a bit time,
I had typescript export miss match, so my export default app;
had to be used as const server = awsServerlessExpress.createServer(app.default) and not const server = awsServerlessExpress.createServer(app)
the latter caused the socket timeout.