Netdata: Nodejs server monitoring

[kevincaradant]

Hi

I read the documentation and find monitoring only for apache, nginx server. Is it exist something for nodejs ? Like web-log, maybe a roadmap about it ?

I would be happy to have this feature in the netdata monitoring because I have some applications which run in nodejs.

Thank you

[ktsaou]

@kevincaradant could you please seek some documentation or alternative methods for implementing this?

I am not aware of what is available. I have seen a few systems that provide a node.js library which should be imported into your code, which uses a statsd implementation to send APM metrics. Are there any alternatives?

You will greatly help towards this if you can find information on how we can do it.

[kevincaradant]

@ktsaou, OK I will do some research.
To manage my node servers, I use PM2 from here: PM2
Very awesome tool. We can have a logs about error or info. I don't know if we can do something with that but it's that I'm using.

You say: "which uses a statsd implementation to send APM metrics".
I have to find any library which use statsd ? Sorry to be a noob about this subject. I don't know really how metrics works.

[ktsaou]

library which use statsd

Check this: http://docs.datadoghq.com/libraries/

[ktsaou]

pm2 looks nice. If there is a statistics interface we can poll to get metrics, we could use that.

[luserx0]

I'm interested in taking this feature on me in terms of GSoC 2017.

[kevincaradant]

@luserx0 That could be awesome if you have the time to do it. For the moment and at least for one / two months, I will be busy so I can't help...

@ktsaou has a huge roadmap for this year I guess so if you want to start on it, please feel free to think about it :)

I will be happy to try your new feature if you need some tests from users of course. And if you need some informations about nodeJS during your dev, I will try to answer you.

[luserx0]

Well, I was not accepted in netdata for GSoC, but I got into GNU-Kawa. So, for the time being I won't be able to create the plugin.
Looking forward to contribute in the near future.

This issue remains unassigned :)

[cadeon]

+1 for pm2 support. It works great, it would make a great addition to netdata.

[mateusbadalotti]

+1 for pm2 support, it is such a great proccess manager :)

[matheusgrieger]

+1 for pm2 support, we use it and love it. Also we love Netdata, would be amazing to see them both working together!

[ktsaou]

Guys correct me if I am wrong, but I think pm2 has a commercial service: keymetrics.io.

Do they publish an API or something to query for metrics or they push everything to keymetrics.io?

I have contacted them on Mar 20 for the possibility of integrating pm2 with netdata, but I never received a response.

If the folks at pm2 are trying to have a living out of it, via keymetrics.io, I think we should not go against their efforts, especially since (as you say - I don't use it) pm2 is a nice, useful, tool.

[vmarchaud]

Hello guys, i'm the backend developer at Keymetrics and also the current maintainer of PM2.

First, indeed keymetrics.io is the paying platform that allow us to have time to work on PM2.
Secondly, our agent embed in PM2 is only pushing data to Keymetrics, in a non-standard protocol so i don't think it would be easy for you to use it.
However, the thing that a lot of people had done in the past is to create module to retrieve information from PM2.
Anyone can do a module as long as the code is published under AGPLv3 (same license as PM2).

I will try to discuss about it with our team to see what's they want to do and i'll come back to you when its done. (Feel free to ping me here if you think i forgot, hint: i might forget)

Edit: not working @ Keymetrics anymore, feel free to go forward if someone want to

PS : My view here doesn't necessarily reflect the view of Keymetrics nor Unitech

[Akuli]

I'm interested in this feature too, all I care about is how many requests/second the server makes. Is there any (hacky) way to do this currently, e.g. writing a plugin?

[antoinerousseau]

hello @vmarchaud, any update on this?

[Leeaandrob]

any update on this?

[rouing]

+1

[jdelvecchio]

Would love netdata <--> pm2 support!

[bsord]

+1 for PM2 support

[jonfairbanks]

Hey @vmarchaud ... any update on the PM2 support? This would be great! 🙏

[IngwiePhoenix]

PM2 would be great, but if we at least had a way of delivering information to netdata from our node applications, that'd be pretty cool. :) Any update here would be apreciated - if with, or without PM2.

[stale[bot]]

Currently netdata team doesn't have enough capacity to work on this issue. We will be more than glad to accept a pull request with a solution to problem described here. This issue will be closed after another 60 days of inactivity.

[ilyam8]

Ok, this is PM2 support issue.
I see @paulfantom doesn't like it for some reason - i see a bunch of downvotes from him.

@paulfantom ?

@vmarchaud

I will try to discuss about it with our team to see what's they want

Any updates on this?

However, the thing that a lot of people had done in the past is to create module to retrieve information from PM2.

Please give some good examples of such modules. I haven't found how to retrieve information from PM2. I read monitoring page but still no luck.

[paulfantom]

Idea is ok, I just downvote people who don't use GitHub reactions and instead add comments which doesn't add any value to the discussion.

[cakrit]

Hi all,

It seems like there is sufficient interest for this collector. However, to get started, we need a user that will act as a sponsor during the implementation.

He/she will assist us with the following:

• Specify the metrics required and how they should be presented in charts.
• Specify the alarms that would make sense for each metric.
• When the implementation passes QA, test the implementation in production.
• Use the charts and alarms in their day to day work and provide additional feedback after the collector is delivered.

Would any one of the people who have expressed interest here be willing/able to play this role?

[IngwiePhoenix]

@cakrit I am working a lot with Node these days, getting started on a project of mine - this would be a nice opportunity to work towards a NodeJS implementation as well, since planning and plotting for what metrics I would like to gather could serve as an - opinionated - basis for what may become a general collector. So - I'd be up for it! :)

[ilyam8]

@IngwiePhoenix

However, the thing that a lot of people had done in the past is to create module to retrieve information from PM2.

Please give some good examples of such modules. I haven't found how to retrieve information from PM2. I read monitoring page but still no luck.

^^ :cry:

Let's start from the metrics. The whole list of metrics we can retrieve from the PM2

[MrJacz]

Any update on this?

[ilyam8]

i googled a bit and found

https://github.com/saikatharryc/pm2-prometheus-exporter/blob/master/exporter.js

it seems that exporter just executes pm2 list and parses the output.

https://www.npmjs.com/package/pm2#managing-applications

Per process metrics

https://github.com/saikatharryc/pm2-prometheus-exporter/blob/cf05e139ffae45ad73900d2823618f66a4d25fe1/exporter.js#L12-L20

It looks like we can use apps.plugin to collect cpu/mem per process. Am i missing something @MrJacz ?

---

But if parse the output from pm2 list is all we need it is very easy to implement. I just have no info on this (help wanted label)

[MrJacz]

Using pm2 list doesnt seem like a terribly bad idea but it tends to be semi incorrect in terms of memory usage. Now it would definately be the easiest way to monitor node applications but if we wanted to drive deeper into the rabbit hole of monitoring node applications we would have to go a fair bit deeper.
When I say deeper im talking about creating a npm module that monitors all the stats for us then exposes some kind of API to gather the information for netdata to use. This would be a lot of work and would take time but the cool thing about doing that would be having the ability to properly monitor the application.

I have never made a netdata plugin and I dont really know how, but i think starting with pm2 list might not be a bad idea

[gc]

pm2 monit - opens a terminal view to monitor the processes in real time
pm2 jlist - shows the current process list in JSON format
pm2 logs - shows the logs of a process (or all processes) in real-time

These commands might be of use?

[jonfairbanks]

I recently found express-status-monitor which installs with minimal config and helps monitor CPU/RAM/etc for Express apps via Socket.io. I've been using it as sort of a workaround for Node monitoring.

https://www.npmjs.com/package/express-status-monitor

Since it is socket.io based, perhaps this could be extended to report into netdata or the package could be re-tooled entirely to support all Node apps (not just Express) for a netdata type plugin like mentioned.

[gc]

I made a quick script which might be helpful, it just shows how you can access data through PM2 using one of the 3 commands I mentioned above (jlist): https://github.com/gc/netdata-pm2-test

It seems the options are: hooking into PM2, hooking directly into nodejs, or both.

As a sidenote, if the easiest way is for the nodejs apps to be running a specially created netdata module, that would be acceptable I think, a special could be shown on the dashboard "It seems like you're running a nodejs application, you can install this package to monitor it, or dismiss this message".

Thanks!

[ilyam8]

When I say deeper im talking about creating a npm module that monitors all the stats for us then exposes some kind of API to gather the information for netdata to use.

New prometheus collector :smile:

As a sidenote, if the easiest way is for the nodejs apps to be running a specially created netdata module,

i am not familiar with js, so if someone is willing to help/work on it it would be great.

What can i do is to write a simple module that parses output from pm2 list/pm2 jlist. Is this ok?

[gc]

@ilyam8 I'm assuming by 'module' you mean a Netdata Module which plugs the PM2 data into Netdata(I was referring to a nodejs module)? If so then yea that's great. I'm pretty good with js and would be willing to help out as much as I can. Just let me know how I can.

The only thing I think that would be missing would be logs, I'm not familiar with exactly how it works, but I think pm2 logs might be able to be accessed as a "stream", i.e whenever a new log is emitted onto the stream, Netdata could add that log to a view in the dashboard. If you periodically run pm2 logs, its quite possible you might miss some output as I believe it only returns the past 10-15 lines of logs, even if logging more than that per second is unlikely. It also seems inefficient to pull lots of logs every second.

Alternatively, the logs from PM2 are saved to disk in ~/.pm2/logs/*.log and could be received from those files in some way by watching for changes.

EDIT: Perhaps logs aren't in the scope of Netdata, but it would be nice if possible.

[cakrit]

Why not incorporate this?
Also, is anyone using PM2 enterprise? We can also use this with the new prometheus write API that @vlkobal is building, to push the metrics to it.

[ElleshaHackett]

I would really love to see this supported in Netdata as we use a lot of node processes and pm2 to manage them. I think using pm2 jlist for easy JSON parsing as @gc suggested is a good way to go. That seems more logic then to use pm2-prometheus-exporter for the simple reason that the exporter would introduce more overhead.

[ilyam8]

@ElleshaHackett could you share time pm2 jlist output?

[cakrit]

In a couple of weeks, we'll have the two year anniversary of this issue. There's enough information here already, at minimum for a first version. So let's plan it, install it ourselves, and move on with the implementation. No problem going with JSON, we can look at the performance vs the prometheus exporter.

[ilyam8]

Alternatively, the logs from PM2 are saved to disk in ~/.pm2/logs/*.log and could be received from those files in some way by watching for changes.

Could you provide an example of the logs?

---

I think using pm2 jlist for easy JSON parsing as @gc suggested is a good way to go.

Well, ok. Could someone provide the output of pm2 jlist?

[EnzoAkira]

Hi!
The company that I'm working use PM2 (v.2 to v.4) on production.
I'm redacting some sensitive info and I will post a example very soon.

[EnzoAkira]

Hi!
The company that I'm working use PM2 (v.2 to v.4) on production.
I'm redacting some sensitive info and I will post a example very soon.

Here it is.
Only a few IP addresses, git links and server name have been changed. All fields are intact!
pm2_jlist.zip

[ilyam8]

I'm working use PM2 (v.2 to v.4) on production

Is it v2 or v4? i guess v4

[EnzoAkira]

The doc I sent is V3.0.3.
In production we have several versions from V2 to V4 (one version per host, just to clarify)

[ilyam8]

Did you compare the output? is it ~ the same?

[gc]

I am using pm2 4.2.3, i compared the difference between my jlist and the one uploaded above and there are these differences: https://www.diffchecker.com/dfzIpeoo (mine is on the LEFT)

Note: I am only comparing the "Types" of them not the actual values, i used https://transform.tools/json-to-typescript to convert the pm2 jlist JSON to typescript types.

Hopefully is helpful.

EDIT: obviously note some of these differences will always be present between any version because it puts the name of your application as a property in the output.

[EnzoAkira]

Did you compare the output? is it ~ the same?

No, I didn't.
I personally never had to look at or compare the pm2 jlist.
I know that development colleagues use pm2 logs

[ilyam8]

do we need sudo to execute pm2 jlist? or it is enough to add Netdata to some group?

[gc]

As far as I know - you only need sudo if pm2 was started with sudo. For example, if pm2 is currently not running and you do sudo pm2 start, all future pm2 commands will require sudo until you pm2 kill to shut it down.

[ilyam8]

So pm2 jlist can be executed by any user? Any user gets data from pm2 jlist? @gc could you check if netdata user is able to run it?

[gc]

Would be happy to test this. I tried doing su - netdata to switch to netdata user, and its asking me for the password of that account. Does netdata automatically give it a password or something? or should I use sudo to force change the password? Apologies I'm not very experienced with linux. (I am using Ubuntu)

[ilyam8]

try sudo su -s /bin/bash netdata (require root password)

[gc]

Ok - on the netdata account, when using any PM2 command, its spinning up a new and separate instance of PM2, so when I go to netdata account, its like i havent even started it yet, but switching back to my account, my normal pm2 apps are still there.

[ilyam8]

on the netdata account, when using any PM2 command, its spinning up a new and separate instance of PM2, so when I go to netdata account, its like i havent even started it yet

I dont know how to overcome this issue. cc @Ferroin

[rcrvano]

any update on this?

[ilyam8]

@rcrvano there is no ongoing work on this.

See https://github.com/netdata/netdata/issues/1960#issuecomment-595191830