Lisk-sdk: error: Forever detected script was killed by signal: SIGKILL

[karek314]

Received error
error: Forever detected script was killed by signal: SIGKILL

While synchronizing with network. Occurs randomly with first release of 0.2 testnet.

[karmacoma]

@karek314 Do you have any further logs to give context. Would be much appreciated, thanks.

[karek314]

log 2016-04-30 13:59:02 POST /peer/blocks from 194.116.72.47
log 2016-04-30 13:59:02 GET /peer/height from 13.82.31.30
info 2016-04-30 13:59:03 Block 2851127644657975087 loaded from 45.32.157.218:7000 at 708
info 2016-04-30 13:59:05 Block 12078856168248327627 loaded from 45.32.157.218:7000 at 709
info 2016-04-30 13:59:08 Block 13873829856052334373 loaded from 45.32.157.218:7000 at 710
log 2016-04-30 13:59:11 POST /peer/blocks from 13.82.31.30
log 2016-04-30 13:59:11 POST /peer/blocks from 108.61.210.213
log 2016-04-30 13:59:11 POST /peer/blocks from 45.32.186.115
log 2016-04-30 13:59:14 POST /peer/blocks from 149.56.15.35
error: Forever detected script was killed by signal: SIGKILL

[karmacoma]

As far as I can see from logs, this error indicates the process received an explicit SIGKILL. So the process didn't terminate abruptly due an error in the application code as such. The other messages are all normal log/info messages.

Are you still experiencing this issue randomly?

[karek314]

I still experience issue with one node. I haven't send any SIGKILL to that process. After restarting LISK, it took around 3minutes to crash again. Here is another log

info 2016-05-01 04:26:04 Block 17266185621837264719 loaded from 139.59.161.220:7000 at 264
info 2016-05-01 04:26:04 Block 987812539636481016 loaded from 139.59.161.220:7000 at 265
info 2016-05-01 04:26:04 Block 11453626418785353969 loaded from 139.59.161.220:7000 at 266
info 2016-05-01 04:26:04 Block 9576553020337839637 loaded from 139.59.161.220:7000 at 267
info 2016-05-01 04:26:04 Block 4954061914605136985 loaded from 139.59.161.220:7000 at 268
info 2016-05-01 04:26:04 Block 16613027282538875542 loaded from 139.59.161.220:7000 at 269
info 2016-05-01 04:26:05 Block 1810242481542628686 loaded from 139.59.161.220:7000 at 270
info 2016-05-01 04:26:05 Block 11901615463686195987 loaded from 139.59.161.220:7000 at 271
info 2016-05-01 04:26:05 Block 11222399559809391928 loaded from 139.59.161.220:7000 at 272
info 2016-05-01 04:26:05 Block 17730636047923467298 loaded from 139.59.161.220:7000 at 273
info 2016-05-01 04:26:05 Block 8914043797201474046 loaded from 139.59.161.220:7000 at 274
info 2016-05-01 04:26:05 Block 5282108974035178002 loaded from 139.59.161.220:7000 at 275
info 2016-05-01 04:26:05 Block 12088094202000844423 loaded from 139.59.161.220:7000 at 276
info 2016-05-01 04:26:05 Block 16867210900830281441 loaded from 139.59.161.220:7000 at 277
info 2016-05-01 04:26:06 Block 168749140277363580 loaded from 139.59.161.220:7000 at 278
log 2016-05-01 04:26:06 POST /peer/transactions from 77.81.229.206 
log 2016-05-01 04:26:06 POST /peer/transactions from 188.166.166.164 
info 2016-05-01 04:26:06 Block 6785728156862987244 loaded from 139.59.161.220:7000 at 279
log 2016-05-01 04:26:06 POST /peer/transactions from 108.61.213.120 
info 2016-05-01 04:26:06 Block 15854978945448908915 loaded from 139.59.161.220:7000 at 280
info 2016-05-01 04:26:06 Block 6394893356752634808 loaded from 139.59.161.220:7000 at 281
info 2016-05-01 04:26:06 Block 3436205745750444713 loaded from 139.59.161.220:7000 at 282
info 2016-05-01 04:26:06 Block 2244888639003745467 loaded from 139.59.161.220:7000 at 283
info 2016-05-01 04:26:07 Block 7461035510578670240 loaded from 139.59.161.220:7000 at 284
info 2016-05-01 04:26:07 Block 6283422584142856057 loaded from 139.59.161.220:7000 at 285
log 2016-05-01 04:26:07 POST /peer/transactions from 77.81.229.206 
info 2016-05-01 04:26:07 Block 17094055486189331624 loaded from 139.59.161.220:7000 at 286
info 2016-05-01 04:26:08 Block 5200043838031952340 loaded from 139.59.161.220:7000 at 287
log 2016-05-01 04:26:08 POST /peer/transactions from 77.81.229.206 
log 2016-05-01 04:26:08 POST /peer/transactions from 188.166.166.164 
info 2016-05-01 04:26:08 Block 7025989831050843942 loaded from 139.59.161.220:7000 at 288
info 2016-05-01 04:26:08 Block 9479098225376444778 loaded from 139.59.161.220:7000 at 289
info 2016-05-01 04:26:08 Block 880863854931255532 loaded from 139.59.161.220:7000 at 290
log 2016-05-01 04:26:08 POST /peer/transactions from 108.61.213.120 
info 2016-05-01 04:26:09 Block 4739656008429273386 loaded from 139.59.161.220:7000 at 291
info 2016-05-01 04:26:09 Block 4996520393151217157 loaded from 139.59.161.220:7000 at 292
info 2016-05-01 04:26:09 Block 2759725530049970572 loaded from 139.59.161.220:7000 at 293
info 2016-05-01 04:26:09 Block 12385970128939018809 loaded from 139.59.161.220:7000 at 294
log 2016-05-01 04:26:10 POST /peer/blocks from 192.99.232.52 
log 2016-05-01 04:26:10 POST /peer/blocks from 159.203.166.60 
log 2016-05-01 04:26:11 POST /peer/blocks from 40.68.34.176 
log 2016-05-01 04:26:11 POST /peer/transactions from 188.166.166.164 
log 2016-05-01 04:26:11 POST /peer/transactions from 77.81.229.206 
info 2016-05-01 04:26:11 Block 12002758065131595408 loaded from 139.59.161.220:7000 at 295
log 2016-05-01 04:26:14 POST /peer/transactions from 188.166.166.164 
log 2016-05-01 04:26:14 POST /peer/transactions from 77.81.229.206 
info 2016-05-01 04:26:15 Block 16722622257605594770 loaded from 139.59.161.220:7000 at 296
log 2016-05-01 04:26:15 GET /peer/signatures from 157.7.221.204 
log 2016-05-01 04:26:20 POST /peer/blocks from 142.54.180.130 
log 2016-05-01 04:26:20 POST /peer/blocks from 159.203.166.60 
log 2016-05-01 04:26:20 POST /peer/blocks from 192.99.232.52 
log 2016-05-01 04:26:20 POST /peer/blocks from 107.170.64.85 
log 2016-05-01 04:26:21 POST /peer/blocks from 40.68.34.176 
info 2016-05-01 04:26:22 Block 17622227552341809296 loaded from 139.59.161.220:7000 at 297
log 2016-05-01 04:26:23 GET /peer/signatures from 188.166.166.164 
log 2016-05-01 04:26:27 POST /peer/transactions from 77.81.229.206 
log 2016-05-01 04:26:27 POST /peer/transactions from 77.81.229.206 
log 2016-05-01 04:26:27 POST /peer/transactions from 188.166.166.164 
log 2016-05-01 04:26:28 GET /api/accounts?address=17957303129556813956L from 127.0.0.1 
log 2016-05-01 04:26:29 GET /api/delegates/voters?publicKey= from 127.0.0.1 
log 2016-05-01 04:26:30 POST /peer/blocks from 142.54.180.130 
log 2016-05-01 04:26:31 POST /peer/blocks from 107.170.43.240 
log 2016-05-01 04:26:31 POST /peer/blocks from 192.99.232.52 
log 2016-05-01 04:26:31 POST /peer/blocks from 159.203.166.60 
log 2016-05-01 04:26:31 POST /peer/blocks from 40.68.34.176 
log 2016-05-01 04:26:32 POST /peer/blocks from 107.170.64.85 
log 2016-05-01 04:26:32 GET /api/accounts?address=17957303129556813956L from 127.0.0.1 
log 2016-05-01 04:26:33 GET /api/blocks/?generatorPublicKey=&limit=1&offset=0&orderBy=height:desc from 127.0.0.1 
info 2016-05-01 04:26:37 Block 1498716325893197268 loaded from 139.59.161.220:7000 at 298
info 2016-05-01 04:26:37 Block 13314702172410576757 loaded from 139.59.161.220:7000 at 299
info 2016-05-01 04:26:38 Block 6785468108148877919 loaded from 139.59.161.220:7000 at 300
info 2016-05-01 04:26:39 Block 17930160464421018560 loaded from 139.59.161.220:7000 at 301
info 2016-05-01 04:26:39 Block 166526822674084389 loaded from 139.59.161.220:7000 at 302
log 2016-05-01 04:26:40 POST /peer/blocks from 107.170.64.85 
log 2016-05-01 04:26:40 POST /peer/blocks from 142.54.180.130 
log 2016-05-01 04:26:40 POST /peer/blocks from 50.7.92.74 
log 2016-05-01 04:26:40 POST /peer/blocks from 40.68.34.176 
log 2016-05-01 04:26:40 POST /peer/blocks from 159.203.166.60 
info 2016-05-01 04:26:42 Block 8444502939185522718 loaded from 139.59.161.220:7000 at 303
info 2016-05-01 04:26:42 Block 2348848402767349996 loaded from 139.59.161.220:7000 at 304
info 2016-05-01 04:26:43 Block 16972595880237508882 loaded from 139.59.161.220:7000 at 305
info 2016-05-01 04:26:43 Block 9135425830904631191 loaded from 139.59.161.220:7000 at 306
info 2016-05-01 04:26:44 Block 12587969601615257868 loaded from 139.59.161.220:7000 at 307
info 2016-05-01 04:26:44 Block 8522782131311762577 loaded from 139.59.161.220:7000 at 308
info 2016-05-01 04:26:45 Block 9090563611519259527 loaded from 139.59.161.220:7000 at 309
info 2016-05-01 04:26:45 Block 12916078866577302697 loaded from 139.59.161.220:7000 at 310
info 2016-05-01 04:26:46 Block 13933076832465260570 loaded from 139.59.161.220:7000 at 311
info 2016-05-01 04:26:46 Block 12019205642538556389 loaded from 139.59.161.220:7000 at 312
info 2016-05-01 04:26:46 Block 4808781496843528708 loaded from 139.59.161.220:7000 at 313
info 2016-05-01 04:26:46 Block 11994744723465282525 loaded from 139.59.161.220:7000 at 314
info 2016-05-01 04:26:47 Block 11948808014184102414 loaded from 139.59.161.220:7000 at 315
info 2016-05-01 04:26:49 Block 2723383221840495844 loaded from 139.59.161.220:7000 at 316
info 2016-05-01 04:26:49 Block 7177807157052046847 loaded from 139.59.161.220:7000 at 317
info 2016-05-01 04:26:49 Block 5008706619970874531 loaded from 139.59.161.220:7000 at 318
info 2016-05-01 04:26:49 Block 7137645143205463743 loaded from 139.59.161.220:7000 at 319
info 2016-05-01 04:26:49 Block 3838978981938795427 loaded from 139.59.161.220:7000 at 320
log 2016-05-01 04:26:50 GET /peer/list from 84.200.32.172 
info 2016-05-01 04:26:50 Block 18179508245803596022 loaded from 139.59.161.220:7000 at 321
info 2016-05-01 04:26:50 Block 7555647167191910931 loaded from 139.59.161.220:7000 at 322
log 2016-05-01 04:26:50 POST /peer/blocks from 142.54.180.130 
info 2016-05-01 04:26:50 Block 3001960916311716074 loaded from 139.59.161.220:7000 at 323
log 2016-05-01 04:26:51 POST /peer/blocks from 192.99.232.52 
log 2016-05-01 04:26:51 POST /peer/blocks from 159.203.166.60 
log 2016-05-01 04:26:51 POST /peer/blocks from 107.170.43.240 
log 2016-05-01 04:26:51 POST /peer/blocks from 50.7.92.74 
log 2016-05-01 04:26:51 POST /peer/blocks from 94.237.25.87 
log 2016-05-01 04:26:51 POST /peer/blocks from 86.105.187.102 
log 2016-05-01 04:26:51 POST /peer/blocks from 92.222.69.182 
info 2016-05-01 04:26:52 Block 5798936154535412404 loaded from 139.59.161.220:7000 at 324
info 2016-05-01 04:26:52 Block 9497356467142177836 loaded from 139.59.161.220:7000 at 325
info 2016-05-01 04:26:52 Block 13160288507462141121 loaded from 139.59.161.220:7000 at 326
info 2016-05-01 04:26:52 Block 13831415514846212807 loaded from 139.59.161.220:7000 at 327
info 2016-05-01 04:26:53 Block 841148600613103278 loaded from 139.59.161.220:7000 at 328
info 2016-05-01 04:26:53 Block 15209757867744973803 loaded from 139.59.161.220:7000 at 329
info 2016-05-01 04:26:53 Block 9525408839940279639 loaded from 139.59.161.220:7000 at 330
info 2016-05-01 04:26:54 Block 1696716498383402611 loaded from 139.59.161.220:7000 at 331
info 2016-05-01 04:26:54 Block 7828762448256006844 loaded from 139.59.161.220:7000 at 332
info 2016-05-01 04:26:54 Block 8613658221348274448 loaded from 139.59.161.220:7000 at 333
info 2016-05-01 04:26:54 Block 12220133167610632079 loaded from 139.59.161.220:7000 at 334
info 2016-05-01 04:26:54 Block 1001908156158007028 loaded from 139.59.161.220:7000 at 335
log 2016-05-01 04:26:54 GET /peer/height from 92.222.69.182 
info 2016-05-01 04:26:55 Block 9329534948368865627 loaded from 139.59.161.220:7000 at 336
info 2016-05-01 04:26:55 Block 3335702054249275324 loaded from 139.59.161.220:7000 at 337
info 2016-05-01 04:26:55 Block 17480477436431873700 loaded from 139.59.161.220:7000 at 338
info 2016-05-01 04:26:56 Block 7123896381393765254 loaded from 139.59.161.220:7000 at 339
info 2016-05-01 04:26:56 Block 15376689453140619338 loaded from 139.59.161.220:7000 at 340
info 2016-05-01 04:26:56 Block 15979314268236912094 loaded from 139.59.161.220:7000 at 341
info 2016-05-01 04:26:56 Block 17639258755586988610 loaded from 139.59.161.220:7000 at 342
info 2016-05-01 04:26:56 Block 7791858931881949171 loaded from 139.59.161.220:7000 at 343
info 2016-05-01 04:26:56 Block 569687152777931279 loaded from 139.59.161.220:7000 at 344
info 2016-05-01 04:26:57 Block 2251077918957817174 loaded from 139.59.161.220:7000 at 345
info 2016-05-01 04:26:57 Block 17567514084232000072 loaded from 139.59.161.220:7000 at 346
info 2016-05-01 04:26:57 Block 15160780758259458581 loaded from 139.59.161.220:7000 at 347
info 2016-05-01 04:26:57 Block 695680185930961622 loaded from 139.59.161.220:7000 at 348
info 2016-05-01 04:26:57 Block 10312675412459187774 loaded from 139.59.161.220:7000 at 349
info 2016-05-01 04:26:57 Block 14205491488511457726 loaded from 139.59.161.220:7000 at 350
info 2016-05-01 04:26:58 Block 7651415199990035538 loaded from 139.59.161.220:7000 at 351
info 2016-05-01 04:26:58 Block 1402332506166777841 loaded from 139.59.161.220:7000 at 352
info 2016-05-01 04:26:58 Block 5736460198323980690 loaded from 139.59.161.220:7000 at 353
info 2016-05-01 04:26:58 Block 7795367541957320064 loaded from 139.59.161.220:7000 at 354
info 2016-05-01 04:26:58 Block 17864434097682462487 loaded from 139.59.161.220:7000 at 355
info 2016-05-01 04:26:58 Block 17925116635675035494 loaded from 139.59.161.220:7000 at 356
info 2016-05-01 04:26:58 Block 11473351570957573433 loaded from 139.59.161.220:7000 at 357
info 2016-05-01 04:26:59 Block 2886006286204806480 loaded from 139.59.161.220:7000 at 358
info 2016-05-01 04:26:59 Block 5146093634537157479 loaded from 139.59.161.220:7000 at 359
info 2016-05-01 04:26:59 Block 5660613592185213603 loaded from 139.59.161.220:7000 at 360
info 2016-05-01 04:26:59 Block 7806036836005877662 loaded from 139.59.161.220:7000 at 361
info 2016-05-01 04:27:00 Block 3580807664667500904 loaded from 139.59.161.220:7000 at 362
info 2016-05-01 04:27:00 Block 9287135780202729978 loaded from 139.59.161.220:7000 at 363
info 2016-05-01 04:27:00 Block 825608856471728725 loaded from 139.59.161.220:7000 at 364
info 2016-05-01 04:27:00 Block 13479885315157080267 loaded from 139.59.161.220:7000 at 365
log 2016-05-01 04:27:00 POST /peer/blocks from 192.99.232.52 
log 2016-05-01 04:27:00 POST /peer/blocks from 104.251.218.240 
log 2016-05-01 04:27:00 POST /peer/blocks from 142.54.180.130 
log 2016-05-01 04:27:00 POST /peer/blocks from 40.68.34.176 
log 2016-05-01 04:27:00 POST /peer/blocks from 63.142.254.178 
log 2016-05-01 04:27:00 POST /peer/blocks from 107.170.43.240 
log 2016-05-01 04:27:01 POST /peer/blocks from 50.7.92.74 
log 2016-05-01 04:27:01 POST /peer/blocks from 91.92.109.141 
log 2016-05-01 04:27:01 POST /peer/blocks from 92.222.69.182 
log 2016-05-01 04:27:01 POST /peer/blocks from 86.105.187.102 
log 2016-05-01 04:27:02 POST /peer/blocks from 188.165.205.32 
log 2016-05-01 04:27:04 GET /api/accounts?address=17957303129556813956L from 127.0.0.1 
log 2016-05-01 04:27:05 GET /api/blocks/?generatorPublicKey=&limit=1&offset=0&orderBy=height:desc from 127.0.0.1 
log 2016-05-01 04:27:18 POST /peer/blocks from 107.170.43.240 
log 2016-05-01 04:27:18 POST /peer/blocks from 142.54.180.130 
log 2016-05-01 04:27:19 POST /peer/blocks from 159.203.166.60 
log 2016-05-01 04:27:19 POST /peer/blocks from 104.251.218.240 
log 2016-05-01 04:27:19 POST /peer/blocks from 192.99.232.52 
log 2016-05-01 04:27:20 POST /peer/blocks from 107.170.64.85 
log 2016-05-01 04:27:20 POST /peer/blocks from 40.68.34.176 
log 2016-05-01 04:27:21 POST /peer/blocks from 86.105.187.102 
log 2016-05-01 04:27:21 POST /peer/blocks from 91.92.109.141 
log 2016-05-01 04:27:21 POST /peer/blocks from 50.7.92.74 
log 2016-05-01 04:27:22 POST /peer/blocks from 188.165.205.32 
error: Forever detected script was killed by signal: SIGKILL

Everything looks healthy until SIGKILL.

[TraderBill]

This is probably my most common error. My nodes will run for maybe an hour maximum before they crash with:

error: Forever detected script was killed by signal: SIGKILL

Here's the logs from most recent crash with this error after running for 15 minutes:

http://pastebin.com/ZEkCnhE7

[fix]

You need to create a swap on your VPS.

[karek314]

In my case, swap is big enough, it's dedicated.

[fix]

Can you output the result of free -m?

[karek314]

[fix]

do you mean that almost all your 8G of RAM is used?

[karek314]

Yes but this machine is not only used for lisk. Swap if free enough. I have noticed this issue also on another machine which was used only for lisk.

[Isabello]

Both PostgreSQL and Nodejs are not designed to make heavy use of swap file, as its extremely CPU and disk intensive. The extra latency from using a significant amount of swap will cause the Nodejs process to become unstable.

Can you post usage statistics for the machine that only has Lisk on it?

[karek314]

Machine with almost entire ram exhausted and almost completely free swap was working much more stable than actually machine dedicated only for Lisk purposes which was crashing few times per hour. As far i remember on the other machine ram was used less than 50%, swap almost empty, i think that's not the source of the problem. Im not able currently to post usage of the other node, maybe @TraderBill will be able to do that. I think it needs deeper and more detailed logging to locate issue, i haven't checked source yet.
Anyway machine i have pasted log up here with entire ram exhausted crashed maybe one or two times like this from the end of ICO. Other machine dedicated only for Lisk was working entirely fine without crashing up to 1.3-1.4 version of testnet 1 or maybe testnet2 with postgreqsl, not exactly sure if it was one release before migration or just after.

[TraderBill]

0.2.2 seems to be running fine after I created a swap following this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-memory-swap-file-on-a-vps

node with 1 GB RAM:

node with 4 GB RAM:

[fix]

yes monitoring this but with the work of @Isabello i think the issue will be fixed. I leave it open for monitoring

[Isabello]

Is this still an issue for anyone?

[fix]

Closing with no activity, i think it is fixed with swap and last version.

[karek314]

Yes, version 0.3.0 fixed issue on the other node.

[chhorn]

Hello
until now i still meet this error with nodejs:
error: Forever detected script was killed by signal: SIGKILL
How to fix it?

[charsi]

Just encountered this issue. Dropping the -w flag seems to have fixed it. Hope this gives clues to what could be causing it.

[palewar]

Just encountered this issue. Dropping the -w flag seems to have fixed it. Hope this gives clues to what could be causing it.

We were getting this error too and could fix it by dropping -w, but we later realised that we actually did a silly mistake of keeping our Forever logs in the same folder as our project files and obviously Forever started and wrote into log files and that means log file changed and Forever was watching it treating it as a project file so if it changes, it will restart the node. It's probably a silly mistake but I thought I will mention it here anyways so that others don't have forever logs or other logs/files which will change when project is running, but it's not a project file so shouldn't lead to restarting the node and project itself.