Peertube: Server crash with « Error: Missing lock for job xxx failed »

[gegeweb]

What happened?
I don't know if this happens when trying to upload large files (long videos) or when encoding.
But I have (since the transition to 1.0.0) this error regularly that crashes the servers.
Usually at the end of an encoding task visibly.
But I also have the upload of the long videos that fails, often with the server crash at the end.

Additional information

• PeerTube version or URL: 1.1.0 (but having this bug with 1.0.0 and 1.0.1)
• NodeJS version: 11.2.0

• Link to server log if useful (journalctl or /var/www/peertube/storage/logs/):

[peertube.gegeweb.eu:443] 2018-12-09 19:54:21.990 error: Error in job queue activitypub-refresher. {
  "meta": {
    "err": {
      "stack": "Error: Missing lock for job 1419 failed\n    at Object.finishedErrors (/var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/scripts.js:187:16)\n    at /var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/job.js:271:31\nFrom previous event:\n    at Queue.processJob (/var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/queue.js:1087:13)\n    at processImmediate (timers.js:632:19)\nFrom previous event:\n    at /var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/queue.js:979:14\n    at process.internalTickCallback (internal/process/next_tick.js:77:7)",
      "message": "Missing lock for job 1419 failed",
      "__stackCleaned__": true
    }
  }
}
npm ERR! code ELIFECYCLE
npm ERR! errno 255
npm ERR! [email protected] start: `node dist/server`
npm ERR! Exit status 255
npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/www/peertube/.npm/_logs/2018-12-09T18_54_26_522Z-debug.log

2018-12-09T18_54_26_522Z-debug.log :

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'start' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle [email protected]~prestart: [email protected]
6 info lifecycle [email protected]~start: [email protected]
7 verbose lifecycle [email protected]~start: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~start: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/var/www/peertube/versions/peertube-v1.1.0/node_modules/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
9 verbose lifecycle [email protected]~start: CWD: /var/www/peertube/versions/peertube-v1.1.0
10 silly lifecycle [email protected]~start: Args: [ '-c', 'node dist/server' ]
11 silly lifecycle [email protected]~start: Returned: code: 255  signal: null
12 info lifecycle [email protected]~start: Failed to exec start script
13 verbose stack Error: [email protected] start: `node dist/server`
13 verbose stack Exit status 255
13 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack     at EventEmitter.emit (events.js:182:13)
13 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:182:13)
13 verbose stack     at maybeClose (internal/child_process.js:978:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:265:5)
14 verbose pkgid [email protected]
15 verbose cwd /var/www/peertube/versions/peertube-v1.1.0
16 verbose FreeBSD 11.2-RELEASE-p5
17 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
18 verbose node v11.2.0
19 verbose npm  v6.4.1
20 error code ELIFECYCLE
21 error errno 255
22 error [email protected] start: `node dist/server`
22 error Exit status 255
23 error Failed at the [email protected] start script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 255, true ]

[gegeweb]

Same this night :

[peertube.gegeweb.eu:443] 2018-12-10 06:15:56.878 info: Creating torrent /var/www/peertube/storage/torrents/c859dbe9-909d-4a3a-a860-f1c864cf5a35-480.torrent.                                                        
[peertube.gegeweb.eu:443] 2018-12-10 06:15:58.624 error: Error in job queue video-file. {
  "meta": {
    "err": {
      "stack": "Error: Missing lock for job 150 failed\n    at Object.finishedErrors (/var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/scripts.js:187:16)\n    at /var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/job.js:271:31\nFrom previous event:\n    at Queue.processJob (/var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/queue.js:1087:13)\n    at processImmediate (timers.js:632:19)\nFrom previous event:\n    at /var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/queue.js:979:14\n    at process.internalTickCallback (internal/process/next_tick.js:77:7)",                       
      "message": "Missing lock for job 150 failed",
      "__stackCleaned__": true
    }
  }
}
npm ERR! code ELIFECYCLE
npm ERR! errno 255
npm ERR! [email protected] start: `node dist/server`
npm ERR! Exit status 255
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/www/peertube/.npm/_logs/2018-12-10T05_16_44_330Z-debug.log

Version of ffmpeg, but same error with the previous version (3.x) :
ffmpeg version 4.1 Copyright (c) 2000-2018 the FFmpeg developers
built with FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)

[gegeweb]

Is there a way to cancel queue job ?
Server crashed again last night on another video ( job 149 ).

At installation on FreeBSD, there is this warning, don't know if it's related or not :

warning Resolution field "@types/[email protected]" is incompatible with requested version "@types/[email protected]"

[gegeweb]

It was not one of the encoding tasks that caused this crash :

[peertube.gegeweb.eu:443] 2018-12-10 23:45:27.166 error: Error in job queue activitypub-refresher. {
  "meta": {
    "err": {
      "stack": "Error: Missing lock for job 8547 failed\n    at Object.finishedErrors (/var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/scripts.js:187:16)\n    at /var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/job.js:271:31\nFrom previous event:\n    at Queue.processJob (/var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/queue.js:1087:13)\n    at processImmediate (timers.js:632:19)\nFrom previous event:\n    at /var/www/peertube/versions/peertube-v1.1.0/node_modules/bull/lib/queue.js:979:14\n    at process.internalTickCallback (internal/process/next_tick.js:77:7)",
      "message": "Missing lock for job 8547 failed",
      "__stackCleaned__": true
    }
  }
}

[Chocobozzz]

Should be fixed by https://github.com/Chocobozzz/PeerTube/commit/9f8ca79284f93693c734dd4b9a27b471017fc441

[gegeweb]

Thanks @Chocobozzz!

How can I apply the patch while waiting for the next release?

[rigelk]

@gegeweb with the source of your release, apply the commit as patch, rebuild the server and restart the server.

[gegeweb]

@rigelk Okay, thank you.
I thought I would avoid having to rebuild.
npm run build:server is enough?

[rigelk]

Yes, that's enough since just the server code is changed.

[gegeweb]

I think i could edit dist/server/lib/job-queue/job-queue.jsand restart the server. Right?
There is no ts sources in the release (archives). So js code already building.

[rigelk]

Exactly. That's why I referred to the source of your release so that you could build the source.

But of course you can edit the compiled file, it's just less reproducible.

[gegeweb]

I've edited the compiled file and restarted the server. So… wait and see!