Bluebird: Bluebird Causes HTTP Module to Drop Requests

[DanielRLevine]

I have a really, really simple app, but when it experiences a high load of POST requests, it resets the connection for a few of them. I have tried replacing BPromise (my variable holding Bluebird's promise class) with Promise (i.e. native JavaScript promises) and the problem goes away. This leads me to suspect that Bluebird is involved in causing the error.

Here is the app code:

var http = require('http');
var BPromise = require("bluebird");

function my_route(req, res) {
    var array = [];
    for (var j = 0; j < 1000; ++j)
        array.push(Math.random());
    array.sort();

    BPromise.resolve(true)
    .then(function(){
        res.writeHead(200);
        res.end("true", 'utf-8');
    });
}

http.createServer(function (req, res) {

    my_route(req, res);

}).listen(1390);

console.log('Server running at http://127.0.0.1:1390/');

I have this code running in cluster mode (via PM2) behind a very run-of-the-mill NGINX reverse proxy. To test the configuration, I am hammering it with enough POSTs to fill CPU utilization from a separate server running apache bench:

ab -n 600000 -p body.json -c 32 [my public ip address]

Out of the 600000 requests sent, maybe 1-4 come back with non-2xx responses every couple of trials. The Nginx logs have the following error:

recv() failed (104: Connection reset by peer) while reading response header from upstream

The error that Nginx is providing tells me that the problem lies not with Nginx, but rather with Node or Bluebird. As I stated above, changing BPromise to Promise eliminates the problem.

Can anyone help me figure out why the HTTP module is hanging up the phone when Bluebird is used? Thank you so much for your help!

Here is my precise configuration:

• App Server: Linux 4.4.0-112-generic x86_64; Ubuntu 16.04.3; 8 Cores w/ 15 GB of memory
  
  
  
  • Nginx: 1.13.8
  
  
  • Node: 8.9.4
  
  
  • Bluebird: 3.5.1 (The problem also occurs in 2.11.0, but does not seem to occur in 1.2.4. I am happy to try other versions if that is helpful.)
  
  
• Test Server: Linux 4.4.0-112-generic x86_64; Ubuntu 16.04.3; 4 Cores w/ 4 GB of memory
  
  
  
  • Apache Bench: 2.3
  
  

Nginx Files
/etc/nginx/nginx.conf:

user                nginx;
worker_processes    auto;

error_log           /var/log/nginx/error.log warn;
pid                 /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include                 /etc/nginx/mime.types;
    default_type            application/octet-stream;

    log_format              main    '$remote_addr - $remote_user [$time_local] "$request" '
                                    '$status $body_bytes_sent "$http_referer" '
                                    '"$http_user_agent" "$http_x_forwarded_for"';

    access_log              /var/log/nginx/access.log  main;

    keepalive_timeout       0;

    underscores_in_headers  on;

    include                 /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/[my public ip address].conf:

upstream api_upstream
{
    server      localhost:1390;
    keepalive   64;
}

server
{
    listen          80 default_server;
    server_name     [my public ip address];

    location /
    {
        proxy_set_header    Host $host;
        proxy_http_version  1.1;
        proxy_set_header    Connection "";
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_pass          http://api_upstream/;
    }
}

Thank you again!

[benjamingr]

Hey, very interesting!

For isolating the issue - can you try running:

• with NODE_ENV=production?
• With Promise.setScheduler(fn => process.nextTick(fn))?

Thanks for the report!

[DanielRLevine]

Hi @benjamingr,

Thank you so much for responding!!! I tried all combinations of the two settings that you asked about. BPromise.setScheduler() fixed the problem; changing NODE_ENV had no effect either way.

Specifically, I added the following two lines to my program:

process.env.NODE_ENV = 'production';
BPromise.setScheduler(fn => process.nextTick(fn));

And I ran six trials:

• NODE_ENV undefined; no setScheduler() call -- BUG OCCURRED
• NODE_ENV="development"; no setScheduler() call -- BUG OCCURRED
• NODE_ENV="production"; no setScheduler() call -- BUG OCCURRED
• NODE_ENV undefined; setScheduler() call -- NO BUG
• NODE_ENV="development"; setScheduler() call -- NO BUG
• NODE_ENV="production"; setScheduler() call -- NO BUG

Thank you again for your help! Please let me know what next steps I should take.

[benjamingr]

@DanielRLevine this is because bluebird is scheduling tasks _behind_ the event queue and native promises do it before the event queue.

So the native implementation starves new requests while serving whereas the bluebird one doesn't.

It's a tradeoff, and you should be free to use .setScheduler as the above to switch to microtick semantics which should be working fine.

[DanielRLevine]

That is a great explanation of what is going on and is incredibly helpful! Just to clarify, does the .setScheduler() call cause Bluebird to schedule tasks before the event cue?

[benjamingr]

@DanielRLevine yes, that's exactly what it does!

[DanielRLevine]

I was able to answer my own question: using the .setScheduler() call you presented will move Bluebird promises from the immediate queue to the next tick queue, which will make them a higher priority than the IO events queue. For anyone else reading this in the future, there was a very helpful series of articles here: https://jsblog.insiderattack.net/promises-next-ticks-and-immediates-nodejs-event-loop-part-3-9226cbe7a6aa

Thank you, @benjamingr again for your help. You aid went a long way in solving a question that had been puzzling me for a few weeks.

[enGMzizo]

I have a question about this issue , Should I use the setScheduler function once on my entire code like on main file or it should be used whenever I require bluebird ?

[benjamingr]

The latter.