Pm2: Graceful Reload - In flight transactions don't complete

[joysonvd]

Hi,

I am using pm2 as a process manager. We started using pm2 a couple of weeks back and it is great. The one issue we faced is regarding the down time we experience. We use Apache Jmeter for sending transactions in. While doing a graceful reload there is about 5 - 6% of transactions which never reach the server and around 10 to 15 transactions which reach the server but the processing is not complete and they stay incomplete in the logs. What we need is that all in flight transactions should complete the processing before the old process is restarted.

Can you please provide me an example code which I can refer while handling graceful reloads in my code.

Regards.

[jshkurti]

The idea behind gracefulReload is to let the user gracefully exit his app before killing it.
Here's how it works, PM2 sends a 'shutdown' message to the process which you can intercept this way:

process.on('message', function(msg) { 
  if (msg === 'shutdown') { 
    //do your stuff before exiting
  } 
});

then it waits till the process exit itself. If it doesn't, PM2 kills it after 8seconds.
This is configurable via env variable PM2_GRACEFUL_TIMEOUT.
You have to set it the first time you launch the PM2 daemon like this PM2_GRACEFUL_TIMEOUT=15000 pm2 start app.js sets the timeout to 15seconds.

Now the basic sample code is the following :

process.on('message', function(msg) { 
  if (msg === 'shutdown') { 
    server.close();
    process.exit(0);
  } 
});

The thing is, server.close() will just stop accepting new requests from this point but sadly Node.js doesn't offer any way to know when all the requests have finished.
So, for now I found a workaround which works fine but it's just a hack. Maybe there are other ways of doing this.

var http = require('http');

var server = http.createServer(function(res, req) {
  setTimeout(function() { // simulate a request which takes time (1sec) to finish                              
    req.end('hallo');
  }, 1000);
}).listen(8080);

server._requests = 0;
server.on('request', function(req, res) {
  server._requests += 1;
  res.once('finish', function() {
    server._requests -= 1;
    if (server._requests === 0)
      server.emit('no_more_requests');
  });
});

process.on('message', function(msg) {
  if (msg === 'shutdown') {
    server.close();
    if (server._requests === 0)
      process.exit(0);
    server.once('no_more_requests', process.exit);
  }
});

It works because the same object req and res is sent to all of the listeners for 'request' event.
Basically, for each request, we increment the value and decrement it as soon as the request has finished.
When the counter equals 0 we fire a custom event which I called no_more_requests.
So in the 'shutdown' listener we stop accepting new requests with sever.close() and then wait till all the running requests have finished before exiting the process.

If you combine this sample code with a big PM2_GRACEFUL_TIMEOUT and run your app in a cluster (let's say -i 4 or -i 8) you won't have dropped requests issues anymore.

I also posted on stackoverflow : http://stackoverflow.com/questions/31077421/node-js-know-when-all-http-requests-are-finished

[joysonvd]

Thanks for the detailed explanation. I will try this right now and post my observation.

[joysonvd]

Preliminary tests showed great results. We are not getting any incomplete transactions. Thanks for the help.

[jshkurti]

Glad I could help ;)

[thiagocaiubi]

@jshkurti @joyson-17

From: https://nodejs.org/docs/latest/api/net.html#net_server_close_callback

Stops the server from accepting new connections and keeps existing connections. This function is asynchronous, the server is finally closed when all connections are ended and the server emits a 'close' event. The optional callback will be called once the 'close' event occurs. Unlike that event, it will be called with an Error as its only argument if the server was not open when it was closed.