Pm2: PM2 crash - segfault error

[ChenL88]

While running on Production , PM2 main process crash with kernel error:
kernel: [603170.036435] PM2 v2.10.4: Go[3113]: segfault at 4052458c8e0 ip 000004052458c8e0 sp 00007ffdd456f4b8 error 15

Amazon Linux AMI release 2018.03
PM2 version 2.2.3
Main Node version : 8.11.3
** We using it to run node applications that use node interpreter 0.10 and up.

Help needed.

[Unitech]

Seems like a system / node issue not related to pm2 itself.
Could you try with the latest pm2 (3.0): npm install pm2 -g ; pm2 update

[ion-willo]

We just got
Jul 25 20:06:49 ip-10-0-32-27 kernel: [621191.242432] PM2 v3.0.0: God[1683]: segfault at f5cb80 ip 0000000000f5cb80 sp 00007ffc291bf6a8 error 15 in node[400000+1dc1000]
Also Amazon Linux AMI

[vmarchaud]

@ion-willo Is there a core dump around ? Maybe in ~/.pm2 ?

[vmarchaud]

Note that the latest pm2 version require at least node 4 so if we find a fix and publish a fix, would you be able to upgrade ?

[ion-willo]

Sorry, no core dump files :(

We're currently running on node v10.7.0, so we should be good to go

[soyuka]

With no core dump we won't be able to help you debug this issue, please make sure to set up your system so that dumps are generated and come back to us! Thanks!

[ion-willo]

Had another crash last night. Unfortunately the ulimit -c command did not stick. Hopefully we'll have another crash, this time with core dump, today or tomorrow.

[ChenL88]

@Unitech i had this crash again , i got the dump file how can i send it to you? (107M)

[ChenL88]

after zip its 7M

[vmarchaud]

@ChenL88 You can send it at tech at keymetrics dot io

[ChenL88]

sent .

[melinite]

we are experiencing random segfaults since we upgraded from 2.10.4 to v3.0.3. Things were extremely stable in 2.10.

PM2 v3.0.3: God[17519]: segfault at ea4cf0 ip 0000000000ea4cf0 sp 00007ffc7cbdf838 error 15 in node[400000+181d000]

we are on Node v6.14.3

[ion-willo]

We got some core dumps recently again. You want me to send them to tech at keymetrics dot io?

[vmarchaud]

I didn't received the other one, would it be possible to send directly to valentin at keymetrics dot io and vincent at keymetrics dot io ?

[ion-willo]

Done. Please let me know if you don't receive it.

[vmarchaud]

@ion-willo got them, i will inspect them thanks

[vmarchaud]

@ion-willo I also need your exact nodejs version to debug it

[ion-willo]

@vmarchaud

node -v
v10.7.0

[ion-willo]

@vmarchaud any news on this issue? I have several more core dump files available if it will help.

[danamajid]

@ion-willo do you happen to be on the C5 EC2 instance type?

I am seeing this happen too in my set up, but only for services that are running on c5.

[ChenL88]

I'm also using C5 EC2 instance.
is there a fix for this?

[Unitech]

Could you try with the latest version of PM2 (3.2.0):

$ npm install pm2 -g
$ pm2 update

Blog post: https://blog.pm2.io/pm2-runtime-3-2-autumn-features/
Changelog: https://github.com/Unitech/pm2/releases/tag/3.2.0

[danamajid]

@Unitech fyi, after upgrading from 2.8.0 to 3.2.0 about 28 days ago we stopped seeing the occasional segfaults! Will report back if this changes.

[cchadwickk]

Hi
We're also experiencing pm2 randomly getting killed on m5 ec2 instance type.
Other than the log provided above we got a slightly longer log.

kernel: [1084396.610433] PM2 v1.1.3: God[7020]: segfault at 7efc96cc7d80 ip 00007efc96cc7d80 sp 00007ffd61656208 error 15 in libstdc++.so.6.0.19[7efc96c69000+e6000]

Does this help ?
Regards

[ion-willo]

Upgrading to v3.2.0 has been to good us so far too

[cchadwickk]

I have another doubt, and sorry if this is the wrong place to post this but pm2 version > 3.0 drops "gracefulReload", which is critical to our production environment. Do we need to alter the code somehow to use the "reload" option instead or is that a direct replacement for gracefulReload? @ion-willo

[ion-willo]

@cchadwickk I'm sure. We're not using that feature. We're using clustering where a restart is effectively a graceful restart AFAIK