Node: https request ends OK but app crash due to ECONNRESET after 10 minutes

[xl-yang]

• Version: 6.10, 7.10.0, 8.1.3
• Platform: Ubuntu(Docker), , 64-bit (Windows 7)

The issue was originally posted here.

I used https.request to visit an intranet server, and response is OK. But after 10 minutes no action on my side, still got ECONNRESET error and process crashed. req.on('error') not working at all.

response data received
response end
!!!!!!!!!!!!!!!!!!!!!! uncaughtException !!!!!!!!!!!!!!!!!!!
{ Error: read ECONNRESET at exports._errnoException (util.js:1050:11) at TCP.onread (net.js:582:26) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }

I also tried using keep-alive agent/default globalAgent/no agent, and even listened 'error','end' of 'socket' event, but none of these prevented application from crashing. Only process.on('uncaughtException') can catch this error but obviously it's not the recommended way for error handling.

Code:

    var options = {
      method: 'GET',
      hostname: hostname,
      path: url,
      auth: username + ':' + password,
      ca: [fs.readFileSync(path.resolve(__dirname, '../cer.cer'))],
    };

    const req = https.request(options, function (res) {

      res.on('data', (resdata) => {
        console.log('data');
      });

      res.on('end', () => {
        console.log('end');
      });
    })

    req.on('error', (e) => {
      console.log(e);
    });

    req.end();

    process.on('uncaughtException', function (err) {
      console.log('!!!!!!!!!!!!!!!!!!!!!! uncaughtException !!!!!!!!!!!!!!!!!!!')
      console.log(err);
    })

[mscdex]

Did you try adding an 'error' event handler for res?

[josephrocca]

I may be experiencing this same bug downstream in the request module. See my comment for code that reproduces this bug fairly reliably (still takes 5 to 10 minutes for it to occur though).

[xl-yang]

@mscdex I did try res.on('error'), not working.

[xl-yang]

@mscdex this issue might need enchancement in code or API document modification to show the correct behaviors when using https.request. So I would suggest remove the 'question' label to draw more attention, thanks.

[josephrocca]

Some people (1, 2) have suggested this sort of thing:

req.on('socket', function(socket) {
  socket.on('error', function (error) {
    reject(error);
    req.abort();
  });
});

And response.on('error', ...) as suggested above, but the error (read ECONNRESET) still occurs and is uncaught in the reproducible example I posted above.

[Trott]

@nodejs/http

[BridgeAR]

@nodejs/http PTAL

[rtw]

@nodejs/http Any updates on this?

[bnoordhuis]

A full stack trace with the latest v8.x or v6.x would be helpful. I.e., without the process.on('uncaughtException') handler.

[Juul]

I believe I am seeing this same problem in node v8.9.4. I am using http.createServer and the ws module and passing the http server instance to ws like so:

var WebSocketServer = require('ws').Server;
var ws = new WebSocketServer({server: server});``

I have added error handlers everywhere I can think of:

server.on('connection', function(socket) {
  socket.on('error', function(err) {
    console.log("Client socket error:", err);
  });
})

server.on('error', function(err) {
  console.error("Server error:", err);
});

server.on('clientError', function(err) {
  console.error("Client connection error:", err);
});

ws.on('error', function(err) {
  console.error("WebSocket error:", err);
});

Two of the error handlers receive the ECONNRESET:

Client connection error: { Error: read ECONNRESET
    at _errnoException (util.js:1022:11)
    at TCP.onread (net.js:615:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
Client socket error: { Error: read ECONNRESET
    at _errnoException (util.js:1022:11)
    at TCP.onread (net.js:615:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }

However I still get an uncaught exception:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at _errnoException (util.js:1022:11)
    at TCP.onread (net.js:615:25)

That appears to be the entirety of the available stack trace. Since the emitted errors are exactly the same as the error from the uncaught exception it seems reasonable to assume that a bug is causing node.js to wrongly believe no error handlers are set on the socket.

I can reliably reproduce the problem by connecting to the websocket server with chromium (firefox doesn' trigger this) and repeatedly hitting reload to cause rapid client connects and disconnects. This issue is sporadic but usually shows up after less than 10 reconnects.

[bnoordhuis]

@Juul Can you reproduce without ws? I.e., using just built-in modules?

[lpinca]

@Juul it looks like you didn't add an error listener on the WebSocket instance on the server? In your case ws is the WebSocket server so you should add this:

ws.on('connection', function (socket) {
  socket.on('error', handleError);
});

From the above snippet you are only adding a listener on the WebSocketServer instance. See https://github.com/websockets/ws/issues/1256 for more details.

[Juul]

@lpinca Yay! Thanks for the suggestion! This turned out to be the problem + a similar problem at an even higher layer.

This is not a node.js issue. My apologies for taking up people's time.

[jwlbjtu]

Dose anyone have a solution or work around on this issue? Have the logic below, seems ECONNRESET
E exception did not get into the error handling block.

request(options, function(error, response) {
    if (error) {
         console.log('Caught');
    }
});

[joyeecheung]

@jwlbjtu Have you tried to add an 'error' event handler to the request? This is tested in the code base.

[joyeecheung]

Also, if anyone bumps into this and is sure that they have tried attaching an error handler to req, please try to reproduce the bug with only Node.js built-in APIs (that is, without any third-party modules). It's best if you can provide a test case that also sets up the server locally instead of relying on one that we could not test against.

If you can only provide the client code to reproduce this, please try running the client with the environment variable NODE_DEBUG=net,stream,http,tls and post the output to stderr here (ideally provide a gist instead of pasting them directly in this thread) - that environment variable would trigger node to log internal details to stderr to help us debug. Note that

• Please remove any sensitive data you can find in the log before posting it
• Be aware that it logs A LOT of stuff to stderr so ideally the client should only request once.

[gireeshpunathil]

Issue has been resolved - by adding an error handler to the right object upon which the event was emitted.

Closing.

[Waog]

I had this Error too and was able to solve it after days of debugging and analysis:

my solution

For me VirtualBox (for Docker) was the Problem. I had Port Forwarding configured on my VM and the error only occured on the forwarded port.

general conclusions

The following observations may save you days of work I had to invest:

• For me the problem only occurred on connections from localhost to localhost on one port. -> check changing any of these constants solves the problem.
• For me the problem only occurred on my machine -> let someone else try it.
• For me the problem only occurred after a while and couldn't be reproduced reliably
• My Problem couldn't be inspected with any of nodes or expresses (debug-)tools. -> don't waste time on this

-> figure out if something is messing around with your network (-settings), like VMs, Firewalls etc., this is probably the cause of the problem.

[kvelaro]

Didnt get any useful answers, some wrote about request, someone about ws, socket, etc.
I also have that error, and listening on (error, () => {}) does not help.
Whats problem, if it is solved/closed, please give me answer, thx
P.S node 8.10 ubuntu

[iesiyok]

For my case it was because of mysql "wait_timeout" value. Somehow if you open a mysql pool and didn't close it for 8 hours. Look here for more info: https://stackoverflow.com/questions/22900931/mysql-giving-read-econnreset-error-after-idle-time-on-node-js-server/22906189#22906189