pm2 integration with logstash

[jaromirmuller]

Hello, we're using intensively logstash+elasticsearch (http://logstash.net/) for managing our logs. Is there any way how to integrate pm2 with logstash?

As I understand pm2 allows to log into file and also have real-time view in console with pm2 logs command. It's useful for development but I'm thinking about usage pm2 in production and ability to browse our logs via kibana interface is much more user friendly.

Is such integration new feature or could I use any module?

[Unitech]

All the logs are written by default into ~/.pm2/logs can you do something to use these files ?
And have a look to the section log management https://github.com/Unitech/pm2#a9, there is also the --merge-logsoptions that can be helpful.

Tell us if you found something to share !
Thanks

[michaelBenin]

Here is a relevant link to formatting logs for logstash in nginx:
https://blog.pkhamre.com/logging-to-logstash-json-format-in-nginx/

Any stdout from the app running would go to pm2's logs, make sure that format is in json that abides to logstash. I agree as this would be helpful if it came out of the box with pm2. Currently it doesn't format it to json.

[Unitech]

Would it be interesting to output the logs of application managed by PM2 in JSON?

[michaelBenin]

Is it possible to set the format like nginx does?

[atomantic]

Yes, JSON format output would be good. I'm also looking for logstash support.
Can we get this reopened for discussion?

I'd like to be able to configure it via process.json like:

"log_format": "json"

[michaelBenin]

Ping.

[solshark]

Ping.

[jlsjonas]

pong?

On a serious note...
generally supporting JSON-formatted logs would improve programmability a lot, so I would also like to recommend supporting an option to log to json instead of plaintext, for improved machine parseability.

Would a PR regarding this be accepted? If so I could look into it when I have time for it (soon).

[soyuka]

What do you mean by "JSON-formatted logs"? If you handle logs from your application, it works well.

Take a simple example:

// log.js
setInterval(function() { console.log(JSON.stringify({hello: 'world'}) })

pm2 start log.js
pm2 stop log.js
jq . ~/.pm2/logs/log-out-0.log
// Outputs json

If you want to pipe out JSON from pm2 logs add the --raw option so that pm2 doesn't add anything.

I don't see how PM2 could improve the logging workflow by implementing JSON output. BTW, should we respect a particular schema? How do we handle user context? This seems big work for no real improvement to me but I'm sure you'll have interesting arguments :).

[jlsjonas]

Looks like I didn't press save (correctly?) last time...
in short:

• handling them in-app might still cause some external/3th party (modules f.e.) logs to bleed through (without json)
• multiple apps logged together
• example scheme: { timestamp:"##timestamp##", app:"appname", msg:payload }
  the payload would be the message (as string) or if json already, passed along (not "double" stringified)

[bragatto]

if @jlsjonas is willing to implement this, by all means, please let him. That would be an excellent way to import logs into an ELK stack more efficiently. +1 for this request, and my compliments for the initiative.

[nidhhoggr]

@jlsjonas Any luck with implementation. Anyone?

[chrishelgert]

any news for logfile in json-format?

[jlsjonas]

Sorry have been preoccupied, I should be able to find some time to implement the changes next week, I'll let you know ;)

[Unitech]

The solutions to implement this would be:

• create an external pm2 module that listen to pm2 log:err and log:out events (these emitted events also contains all application metadata) then log this into a unique file that could be used by logstash
• edit pm2 internals to write json instead of raw lines (cluster mode: here and here fork mode here and here) so you will get all the logs separated

I guess the first solution would provide an easier way to integrate with logstash, as you do not need the log to be splitted by application name/id/error-out,

Maybe we can directly send the log to logstash? It looks like there are some modules for that (https://npms.io/search?term=logstash)

[Unitech]

You can refer to the pm2-syslog module that output app logs into /var/log/syslog

https://github.com/pm2-hive/pm2-syslog/blob/master/app.js#L1

Quite straightforward, 24 lines long!

[Unitech]

$ npm install Unitech/development -g
$ pm2 update

Now the pm2 logs can take the --json argument to ouptut json,
hope you find this helpful for logstash integration

[jlsjonas]

Maybe we can directly send the log to logstash? It looks like there are some modules for that (https://npms.io/search?term=logstash)

That should be perfectly possible yes, it is how I've been handling my internal communications to ELK so far. Given the --json argument to pm2 logs a pipe should be able to handle most usecases, albeit possibly in a hackish way. I might still make a submodule for it if we decide on using ELK again for our next project. The main project we used ELK for became obsolete so this doesn't have a high priority to me anymore (+ pm2 logs --json gives a workaround).

However I would recommend keeping the issue open for now, since the work that remains should be pretty trivial; in case anyone else wants to tie their pm2 processes to ELK

[Unitech]

$ pm2 logs --json

Available since PM2 2.x