Pm2: Email notifications on crashes

[medikoo]

It'll be great to be notified immediately when process crashes due to unhandled exception.

[julianlam]

Give stack trace, too!

[rlidwka]

<irony>and also Jabber notifications, and phone calls on weekends except for midnight</irony>

Guys, just pipe ~/.pm2/logs/*.err to your favourite mailer, all data are there.

PS: by the way, why aren't ~/.pm2/pm2.log entries separated by newlines? It confuses some programs (i.e. tail).

[soyuka]

+1 it's not pertinent, and not usefull !

[medikoo]

@rlidwka no need for irony. This feature would be very helpful

_*.err_ logs are mainly about handled exceptions and eventual notifications of them should be solved within process logic.

Unhandled exceptions are in most cases result of critical bugs that's best when handled asap, silent restart is very imperfect solution.

Of course I would expect email notifications to be an option, and not a must.

[rlidwka]

*.err logs are mainly about handled exceptions

*.err logs are all that your application sent to stderr. I have only crashes there (not really readable by the way because of lack of ending newlines after stack traces).

Handled exceptions should probably be sent to stdout instead so they'll appear in *.log's, not in *.err's.

[Unitech]

@medikoo For now I don't plan to integrate this feature on the core of pm2.
BUT if we do an HTTP interface for pm2 as explained here https://github.com/Unitech/pm2/issues/91, we could build modules who interacts with pm2 and create module like a web interface or sending mail if a process hang up. So we keep the pm2 core lean.

As Alex Kocharin said console.log are output to *.out file and console.err to *.err log files.

PS: by the way, why aren't ~/.pm2/pm2.log entries separated by newlines? It confuses some programs (i.e. tail).

I just looked at my pm2.log every lines are separated by newline, maybe I haven't catch something

[rlidwka]

I just looked at my pm2.log every lines are separated by newline, maybe I haven't catch something

Nevermind, I looked at the top of the file, and it was generated by an old version 3 months ago. Yep, it works fine now.

[accessdev]

I m surprised pm2 dont write callstack on crash & dont restart app when a crash occurred. I agree than developers should fix crashes in their app but due to permissive nature of javascript on complicated apps errors like undefined is not function can easily happen, and pm2 dont seem to provide any feature to handle that.

[soyuka]

It does, your app restart when the process exits, and it logs stack informations.

[accessdev]

strange. I ve built a simple crash test running with express :

var crashCount = 5

router.get("/crash",function(request,response,next)
{
crashCount--;
if (crashCount<1)
{
setTimeout(function(){crash.me()},10) // Reference Error
}
response.json({"crashCount":crashCount})

            })

On pm2 0.12.5 on windows, pm2 auto restart the app after each crash

On pm2 0.12.3 on centos 6 , the app is still flagged as online (pm2 status) and the app is not responding anymore to http requests. not sure yet it the issue is related to our architecture (netscaler + 2 fronts servers) or pm2 itself, but I m curious why the app is still marked as online . there is also a uptime mismatch between id 0 and 1 (cluster mode bug? )

[ravi]

I am planning on implementing this in a very simple form in PM2: if an email configuration is specified, a crash notification email will be sent using the nodemailer module (config therefore has to be node mailer digestible). I do not want to compete with keymetrics capabilities, so I'd like to keep the implementation very simple. If anyone has any thoughts/comments before I implement, please let me know.

[jshkurti]

var pm2 = require('pm2');

pm2.connect(function() {
  pm2.launchBus(function(err, bus) {
    bus.on('process:event', function(data) {
      if (data.event === "exit") {
        // Do your stuff here ...                                                                                                                  
      }
    });
  });
});

[ravi]

That's not bad but it means the user has to write a program to monitor/manage a program that is monitoring/managing other programs. At some point, the mind tires of the indirection. The above however can be one implementation of a standalone program (pm2-notifier?) -- if one doesn't already exist -- to do the email notification, though I think I would still prefer this as a core function.

[liesislukas]

++ It would be great to get email with info about critical errors.

[soyuka]

I've done a prototype here

[liesislukas]

done catching all console.error with proxy pattern. So no need for pm2 stuff.
If some errors happens they are sent to email every 120s. I use nodemailer to send emails: https://www.npmjs.com/package/nodemailer

It works with any nodejs js. It would work even with browser, just email sending logic would be different.

const HelperEmail = require('./helpers/email.js');

var err_messages_to_send = [];

(function (proxied) {
    console.error = function () {
        if (process.env.NODE_ENV !== 'production') {
            return proxied.apply(this, arguments);
        }
        var err = new Error();
        var error = arguments[0];

        if (typeof arguments[0] === 'string') {
            arguments[0] = 'CRITICAL ERROR: ' + arguments[0];
            error = error.replace(new RegExp('\r?\n', 'g'), '<br />');
        }

        var today = new Date();
        var message = "<div><div><strong>TIME:</strong>"
            + today.toUTCString()
            + "</div><div><strong>Error:</strong> "
            + error
            + "</div><div><strong>Backtrace:</strong> "
            + err.stack.replace(new RegExp('\r?\n', 'g'), '<br />')
            + "</div></div>";

        err_messages_to_send.push(message);

        return proxied.apply(this, arguments);
    };
})(console.error);

var logs_interval = setInterval(function () {
    if (err_messages_to_send.length > 0) {
        var messages_to_send = err_messages_to_send;
        err_messages_to_send = [];
        var message = messages_to_send.join('<br /><hr /><br />');
        HelperEmail.sendEmail(null, 'Errors @ClientApi', '[email protected]', message, null, function (err) {
            if (err !== null) {
                console.warn('#anjhaej email ERROR not sending error notifications')
            } 
        });
    }
}, 120000);

[rameshvinfo]

var pm2 = require('pm2');

pm2.connect(function() {
pm2.launchBus(function(err, bus) {
bus.on('process:event', function(data) {
if (data.event === "exit") {
// Do your stuff here ...
}
});
});
});

note able to catch the process event