Pm2: Socket.io with pm2 reload duplicates sockets

[MyMomSaysIAmSpecial]

I am running node.js in cluster mode with socket.io and pm2 now like this (default socket.io chat "hello world" example)

var
    probe = require('pmx').probe(),
    app = require('express')(),
    server = require('http').createServer(app),
    io = require('socket.io')(server),

    port = 3131 + parseInt(process.env.NODE_APP_INSTANCE),
    counter = probe.counter({
        name: 'User online'
    });



io.on('connection', function(socket) {
    counter.inc();
    socket.on('disconnect', function() {
        counter.dec();
    });

    socket.on('message', function() {
        socket.emit('message', port);
    });
});

server.listen(port, function() {});

and default html client (all who visits my page, executes this)

var socket;
window.addEventListener("load", function() {
    socket = io.connect('http://localhost:3000/', {
        'reconnection': true,
        'reconnectionDelay': 500,
        'reconnectionAttempts': 3
    });

    socket.on('message', function(message) {
        console.log(message);
    });

    setInterval(function() {
        socket.emit('message', {});
    }, 2000);
});

this goes with default nginx sticky sessions configuration (http://socket.io/docs/using-multiple-nodes/ found here, one difference, i skipped last redis part, only nginx config is used)

upstream io_nodes {
    ip_hash;
    server 127.0 .0 .1: 3131;
    server 127.0 .0 .1: 3132;
    server 127.0 .0 .1: 3133;
    server 127.0 .0 .1: 3134;
    server 127.0 .0 .1: 3135;
    server 127.0 .0 .1: 3136;
    server 127.0 .0 .1: 3137;
    server 127.0 .0 .1: 3138;
}

and pm2 config like this:

{
    "apps": [{
        "name": "server",
        "script": "server.js",
        "instances": 8,
        "exec_mode": "cluster",
        "max_memory_restart": "2G"
    }]
}

And i have several problems:

1. First is user online count is highter when whole my project online users count, for example (real case!) if i have 3800 users online, my keymetrics (or simple io.eio.clientsCount) shows 4000 or even 5000 total clients connected.

2. Follows from the first, if i use pm2 reload or pm2 restart, i have case when first node duplicates or even multiplies several times sockets count, and gets 100%+ cpu load. Sometimes all nodes, not only first multiplies sockets count. To avoid this i need to kill pm2, wait a minute, and when start again.

Situation after pm2 reload (in real there is only 3868 users online):

[vmarchaud]

When you shutdown your socketio server, socketio must not be calling the .on('disconnect'), thats why the counter doesnt decrement.

[MyMomSaysIAmSpecial]

When you shutdown your socketio server, socketio must not be calling the .on('disconnect'), thats why the counter doesnt decrement.

@vmarchaud Sorry, do you mean it must not call, or its not calling?

Edit: As far as i understood from your post my problem at this moment is what if i restart/shutdown/reload server, sockets are not beeing closed. Because of this even io.eio.clientsCount displaying abnormal sockets count.

[vmarchaud]

Its not calling

[MyMomSaysIAmSpecial]

Small addition to topic, i noticed what when i try to reload/restart my pm2, it says what some processed cant be killed.

[vmarchaud]

Because the process might have a handler that doesn't exit on SIGINT signal, so pm2 force kill them after a certain timeout.

[MyMomSaysIAmSpecial]

About connection count what is rising constantly - it's not pm2 problem, i turned down nginx, pm2, cluster and left only one instance which i run on pure node.js with "node server.js" command without any parameters. So who will read this in future, my connection count problem is NOT because of PM2

Update: problem is mobile clients, they ignore socket.io connection/reconnection settings, and despite server was offline for 3 days, they still pingin it, and then server goes online, they reconnects. So if port was public once, and there is still open clients with this port, you will have my situation.

[bladefist]

In my opinion this is a better way to do what you want to do.

var metric = probe.metric({
name : 'Realtime users',
value : function() {
return io.engine.clientsCount;
}
});

[blueror815]

Hello @MyMomSaysIAmSpecial
So did you find a solution?
I am faced same issue on Digital Ocean.
When i restart my project usng pm2 restart all, i have 100 % overloaded CPU usage and my droplet has gone.