pm2 logs & bunyan

[iliakan]

With bunyan logger, logs come in JSON.
Usually I watch them with tail -f my.log | bunyan -o short, so that bunyan CLI utilify parses and nicely shows JSON for me.

But PM2 adds a prefix to each record, so bunyan can't parse it any more.
Is there a way to use pm2 logs with bunyan filtering?

[soyuka]

https://keymetrics.io/2014/07/10/pm2-interface-1-build-your-own-logger/ there is a bunyan example, hope that helps.

[soyuka]

tail -f ~/.pm2/log/my.log | bunyan -o short is working to me on console.log(json)

node 0.10.31, pm2 0.10.8

[iliakan]

Doing almost the same ATM.

The original post was about pm2 logs command.

[iliakan]

If I use the example from here: https://keymetrics.io/2014/07/10/pm2-interface-1-build-your-own-logger/ and the application itself logs with bunyan, how do they make friends together? Will it be a "bunyan object inside a bunyan object" log?

[soyuka]

Oh I misunderstand

[soyuka]

Actually there is no pm2 logs command that outputs json but that could be a feature.

What are you trying to do if you're using bunyan inside the application? pm2 logs doesn't fit your needs?

Will it be a "bunyan object inside a bunyan object" log?

You'll need to send your logs through bunyan in the app with a json format and then send it to your logging app through pm2 events so that bunyan can format it. Or simply use tail (file if you need json), pm2 ilogs or pm2 logs

[iliakan]

I use bunyan to log in JSON of course :)
The problem with pm2 logs is that it prepends every record with a process name, so that it's not JSON any more.

How does pm2-interface fit in? Will it make pm2-specific log records JSON? Is it the suggested purpose?

[soyuka]

The file is not in a JSON format if you log through bunyan? It's only the pm2 logs interface that adds format for quick reading, isn't it?

[iliakan]

The file is JSON. Yes it's only pm2 logs that adds prefix.

[soyuka]

So, I don't see the issue here, make your own logging file to store pure json if you need too.

[aishwar]

May be a little late, but for people who want to use "pm2 logs" and still see the buynan formatted output, you could do this:

pm2 logs <app> | grep -o '\{.*' | bunyan

That strips out the pm2 appended appname in front of each of the log line, so the output becomes valid JSON per line and the bunyan CLI can parse and print a formatted line.

[raine]

pm2 logs could have an option to not prefix logs?

[hertzg]

@raine I agree that options would be nice

[raine]

Currently it's difficult to get application's logs out of pm2 logs without decoration (timestamp or app name etc.). I couldn't figure out a trivial sed or grep pattern that would do the job.

I think an option like -r --raw would be very useful.

[MichielDeMey]

+1 for the --raw option as proposed by @raine.

[purgna]

+1 for --raw please.

[GDownes]

+1 for --raw

[GDownes]

jshkurti confimed in the feature request issue #1189 thread. "Feature request: PM2 logs raw format" that a raw format will be provided in the next release.

[jshkurti]

Do you wish to keep PM2 logs showing when the --raw param is passed ?

PM2: 2015-04-21 16:00:40: App name:server id:0 exited with code 1
PM2: 2015-04-21 16:00:40: Starting execution sequence in -fork mode- for app name:server id:0
PM2: 2015-04-21 16:00:40: App name:server id:0 online

[raine]

For bunyan use case it doesn't matter because you can say --strict to suppress lines unintelligible to bunyan.

(And sometimes you might want to see both pretty bunyan output and PM2 logs.)

[GDownes]

I agree with raine, keeping them showing wouldn't affect the bunyan use case.

[jshkurti]

Here we go https://github.com/Unitech/PM2/pull/1201

Can you give it a try ? npm i -g git://github.com/Unitech/PM2#raw_logs

It will be included in the next release ;)

[astephensen]

Works perfectly here running pm2 logs --raw | bunyan :+1:

[astanciu]

My app outputs color coded logs and this --raw option seems to strip that away. Any chance we can really get the raw logs, so originating colors will still work?

[iamsebastian]

Same question here, as @astanciu already mentioned. Any possibility to keep colors from debug/chalk package?

[barkthins]

how about a small hook so that logs will display via bunyan without stripping anything or removing colors?

[jackishere]

pm2 logs --raw |bunyan -o short

makes the different

[shuperry]

I tried, this command works:

pm2 logs --raw --name kingnoshop-api| bunyan

kingnoshop-api is my app name.

[zamrokk]

pm2 logs --raw |bunyan 👍

[bidadh]

pm2 logs --raw |bunyan works

[tinybug]

great!

[jchallenger]

Does pm2 logs --raw |bunyan not work anymore?
PM2 is prefixing the raw output with a timestamp, and I'm not sure how to stop this.

[raine]

@jtchallenga I am not getting timestamps on version 2.4.2.

[jchallenger]

Ah, figured it out.
I am loading my app via an ecosystem.json, where I needed specify "log_date_format": "" instead of another value.

[mwakerman]

Is it possible to set raw: true in process file to have pm2 logs to the console without the processId and name prefix? Our use case is that we are using pm2-docker and logspout to ship our logs into logstash and the pure json logs are much easier to parse.

[vmarchaud]

@mwakerman You can use log_type: json : http://pm2.keymetrics.io/docs/usage/log-management/#directly-output-json-logs

[mwakerman]

Thanks @vmarchaud, unfortunately that doesn't remove the id and App name prefix. So we still get something like:

0|api      | {
0|api      |   "level": "info",
0|api      |   "message": "::ffff:172.19.0.6 - - [30/Aug/2017:09:30:23 +0000] \"POST /api/v1/auth/logout HTTP/1.1\" 204 - \"https://domain.tech/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36\" -\n",
0|api      |   "timestamp": "2017-08-30T09:30:23.566Z"
0|api      | }
0|api      | {"message":"{\n  \"level\": \"info\",\n  \"message\": \"::ffff:172.19.0.6 - - [30/Aug/2017:09:31:28 +0000] \\\"POST /api/v1/auth/login HTTP/1.1\\\" 200 155 \\\"https://domain.tech/login\\\" \\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36\\\" 3c285e4b-575c-4044-9d41-ed7e5a881f8e\\n\",\n  \"timestamp\": \"2017-08-30T09:31:28.312Z\"\n}\n","timestamp":"2017-08-30T09:31:28.313Z","type":"out","process_id":0,"app_name":"api"}
0|api      | {"message":"{\n  \"level\": \"info\",\n  \"message\": \"::ffff:172.19.0.6 - - [30/Aug/2017:09:31:28 +0000] \\\"GET /api/v1/document/previews HTTP/1.1\\\" 200 496 \\\"https://domain.tech/document\\\" \\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36\\\" 3c285e4b-575c-4044-9d41-ed7e5a881f8e\\n\",\n  \"timestamp\": \"2017-08-30T09:31:28.364Z\"\n}\n","timestamp":"2017-08-30T09:31:28.365Z","type":"out","process_id":0,"app_name":"api"}

Note the 0|api | at the beginning of every line.

[vmarchaud]

@mwakerman Can you share your process file ?

[mwakerman]

It's an express js and I'm using winston for console logging with json: true. To get the above, my process file looks like:

$ cat ecosystem.config.js
module.exports = {
  apps : [
    {
      name      : 'api',
      script    : 'bin/www',
      exec_mode  : 'cluster',
      instances: 2,
      log_type: 'json'
    },
  ],
};

[vmarchaud]

@mwakerman And which binary are you using to start this proces flie (also the flags you are giving to it) ?

[mwakerman]

I'm using pm2-docker and pm2-docker start ecosystem.config.js to start the process (no other flags).

pm2 --version
2.6.1

[vmarchaud]

@mwakerman You should use pm2-docker start ecosystem.config.js --raw --auto-exit. --raw to delete the prefix and --auto-fix so PM2 can exit when there is no more processes running

[vmarchaud]

Yes, its the pm2-docker binary that need to have the option to stream the logs without prefix, see https://github.com/Unitech/pm2/blob/master/lib/binaries/Runtime4Docker.js#L21

[blakew]

Output for

pm2 logs --raw | bunyan

is suppressed on Windows Powershell. Any help would be appreciated!

[felipekm]

I ended with (had to install bunyan globally):

pm2 logs <app_name> | bunyan -o short