Patchwork: maximum call stack exceeded, then sbot is missing

[justinabrahms]

I'm trying to start up patchwork from a local checkout. I see a stream of Error {} followed by:

Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
RangeError: Maximum call stack size exceeded
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/cursor.js:26:43
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/inject.js:42:22
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:14)
    at next (/home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:47:16)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:43:11
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:41:16
unix socket client
/home/justin/.local/lib/node_modules/git-ssb/node_modules/git-ssb-web/server.js:19
  if (err) throw err
           ^

Error: could not connect to sbot
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/ssb-client/index.js:100:23
    at next (/home/justin/.local/lib/node_modules/git-ssb/node_modules/multiserver/compose.js:24:14)
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/multiserver/compose.js:34:9
    at _cb (/home/justin/.local/lib/node_modules/git-ssb/node_modules/multiserver/plugins/shs.js:28:20)
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/secret-handshake/protocol.js:115:22
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-handshake/index.js:10:5
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-handshake/index.js:23:13
    at callback (/home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-pushable/index.js:81:7)
    at drain (/home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-pushable/index.js:68:39)
  Error: stream ended with:64 but wanted:80
    at drain (/home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-reader/index.js:43:26)
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-reader/index.js:63:18
    at /home/justin/.local/lib/node_modules/git-ssb/node_modules/pull-reader/index.js:20:7
    at drain (/home/justin/.local/lib/node_modules/git-ssb/node_modules/stream-to-pull-stream/index.js:141:18)
    at Socket.<anonymous> (/home/justin/.local/lib/node_modules/git-ssb/node_modules/stream-to-pull-stream/index.js:162:5)
    at emitOne (events.js:121:20)
    at Socket.emit (events.js:211:7)
    at TCP._handle.close [as _onclose] (net.js:567:12)

I've done:

rm -rf node_modules/
git reset --hard origin/master
npm install
npm start

The UI seems to hang on indexing w/o an apparent error message.

[justinabrahms]

f1080ff694f7de88aa41994d7fc5e98c0047b0ba is my latest sha, for reference.

[christianbundy]

Thanks for the bug report! I've seen the Error {} garbage before but I've never had a real stack trace. I think this should be helpful:

RangeError: Maximum call stack size exceeded
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/cursor.js:26:43
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/inject.js:42:22
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:14)
    at next (/home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:47:16)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:43:11
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:41:16

Could you try running $(npm bin)/ssb-server start to see whether this is specific to Patchwork?

[justinabrahms]

The ssb-server start command had the output below. The Error (not the RangeError) was printed many, many times.

Error: view stream error, from:last
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/ssb-db/node_modules/flumedb/index.js:139:35
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/sinks/drain.js:20:24
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/throughs/async-map.js:28:19
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/skip.js:12:26
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/cursor.js:28:28
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/inject.js:42:22
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:14)
  Error: past end:672420546-2374485795 < 673286541
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:17)
    at next (/home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:47:16)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:43:11
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:41:16
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
console.js:116
      throw e;
      ^

RangeError: Maximum call stack size exceeded
    at process.nextTick (internal/process/next_tick.js:259:16)
    at onwrite (_stream_writable.js:453:15)
    at WriteStream.Socket._writeGeneric (net.js:789:5)
    at WriteStream.Socket._write (net.js:799:8)
    at doWrite (_stream_writable.js:397:12)
    at writeOrBuffer (_stream_writable.js:383:5)
    at WriteStream.Writable.write (_stream_writable.js:290:11)
    at WriteStream.Socket.write (net.js:717:40)
    at write (console.js:111:12)
    at Console.warn (console.js:143:3)

[joshfleming]

I'm seeing something similar with the packaged Appimage.

The app appears to be stuck indexing for hours with no progress at the same time. Not sure if it's related.

installed: X-AppImage-BuildId=da790b00-489a-11a9-1c93-4571d2aabea5 image: X-AppImage-BuildId=da790b00-489a-11a9-1c93-4571d2aabea5
{ path: '/home/user/.ssb',
  party: true,
  timeout: 0,
  pub: true,
  local: true,
  friends: { dunbar: 150, hops: 2 },
  gossip: { connections: 3 },
  connections: 
   { outgoing: { net: [Array] },
     incoming: { net: [Array], ws: [Array] } },
  timers: { connection: 0, reconnect: 5000, ping: 300000, handshake: 5000 },
  caps: 
   { shs: ...
     sign: null },
  master: [],
  logging: { level: 'notice' },
  port: 8008,
  blobsPort: 8989,
  server: true,
  _: [],
  host: '::',
  ws: 
   { host: '::',
     port: 8989,
     scope: [ 'device', 'local', 'public' ],
     transform: 'shs' },
  keys: 
   { curve: 'ed25519',
     public: ...
     private: null,
     id: ... } }
ssb-friends: createFriendStream legacy api used
git-ssb is not installed, or not available in path
Error: listen EADDRINUSE :::8989
    at Object._errnoException (util.js:1024:11)
    at _exceptionWithHostPort (util.js:1046:20)
    at Server.setupListenHandle [as _listen2] (net.js:1351:14)
    at listenInCluster (net.js:1392:12)
    at Server.listen (net.js:1476:7)
    at EventEmitter.emitter.listen (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-ws/server.js:50:14)
    at Object.server (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/scuttlebot/node_modules/multiserver/plugins/ws.js:20:31)
    at Object.server (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/multiserver/compose.js:96:29)
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/multiserver/index.js:45:21
    at Array.map (<anonymous>)
Error {}
Error {}
Error {}
Error {}
Error {}
...
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
RangeError: Maximum call stack size exceeded
    at Array.join (native)
    at module.exports (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/explain-error/index.js:23:68)
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/flumedb/index.js:139:35
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-stream/sinks/drain.js:20:24
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-stream/throughs/async-map.js:28:19
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-cursor/skip.js:12:26
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-cursor/cursor.js:28:28
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/flumelog-offset/inject.js:42:22
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/aligned-block-file/blocks.js:49:14)
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
...
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
Error {}
RangeError: Maximum call stack size exceeded
    at module.exports (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/explain-error/index.js:25:18)
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/flumedb/index.js:139:35
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-write/index.js:40:31
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-write/index.js:41:13
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-stream/throughs/async-map.js:28:19
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-cursor/skip.js:12:26
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/pull-cursor/cursor.js:28:28
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/flumelog-offset/inject.js:42:22
    at /tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/tmp/.mount_patchwr3IE5P/app/resources/app.asar/node_modules/aligned-block-file/blocks.js:49:14)
ssb-friends: stream legacy api used
ssb-friends: get legacy api used

[christianbundy]

Hmm, I'm not sure why that would be happening. Maybe some more data would be helpful:

• Which operating systems are you using?
• Do you have git-ssb installed?
• @justinabrahms could you try installing with npm ci just in case it's a dependency issue?

[black-puppydog]

FYI I seem to have the same with the current AppImage. Although I never let it sit for long enough to go past Error: listen EADDRINUSE :::8989

Edit: just finished a git clean -fxd && npm i && npm start and it seems to work fine.
This is in a test environment though, so not quite the same as the AppImage.
FWIW, I use a conda env with a recent npm for this, since my work machine is ubuntu 16.04... :|

[justinabrahms]

I'm using Ubunu 18.04.2 LTS.

I do have git-ssb installed at ~/.local/bin/git-ssb, which I believe happened due to installing the patchfoo client when cel helped me at scuttlecamp.

I have an identical problem when I do this:

rm -rf node_modules
npm ci
npm start

@black-puppydog that addrinuse thing means you have another process running on 8989, likely another sbot that's running for some reason.

[christianbundy]

This looks to me like an issue with ssb-server. Do you have anything in ~/.ssb/config? If so, I might try moving that to ~/.ssb/config.bak or something and then re-running $(npm bin)/ssb-server start to see if those Error {} outputs are still being spammed.

[justinabrahms]

Previous content:

{ 
  "blobs": {
    "max": 10000000
  }
}

Moved to the .bak location, but ssb-server still fails with the same errors.

Error: view stream error, from:last
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/ssb-db/node_modules/flumedb/index.js:139:35
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/sinks/drain.js:20:24
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/throughs/async-map.js:28:19
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/skip.js:12:26
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/cursor.js:28:28
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/inject.js:42:22
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:14)
  Error: past end:672420546-2374485795 < 673286541
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:17)
    at next (/home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:47:16)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:43:11
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:41:16
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
console.js:116
      throw e;
      ^

RangeError: Maximum call stack size exceeded
    at process.nextTick (internal/process/next_tick.js:259:16)
    at onwrite (_stream_writable.js:453:15)
    at WriteStream.Socket._writeGeneric (net.js:789:5)
    at WriteStream.Socket._write (net.js:799:8)
    at doWrite (_stream_writable.js:397:12)
    at writeOrBuffer (_stream_writable.js:383:5)
    at WriteStream.Writable.write (_stream_writable.js:290:11)
    at WriteStream.Socket.write (net.js:717:40)
    at write (console.js:111:12)
    at Console.warn (console.js:143:3)

[christianbundy]

It looks like this error is being thrown by the last index, maybe it's corrupted? If you'd like you can create a copy of ~/.ssb/flume/last.json and then delete the original, which should make ssb-server regenerate it cleanly, but I'm not sure how/why that file would be corrupted.

[justinabrahms]

mv ~/.ssb/flume/last.json{,.bak}
$(npm bin)/ssb-server start

This yields this similar (but slightly different) callstack, which points to drain.js

Error: view stream error, from:keys
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/ssb-db/node_modules/flumedb/index.js:139:35
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumeview-hashtable/index.js:91:13
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/sinks/drain.js:20:24
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/throughs/async-map.js:28:19
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/skip.js:12:26
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/cursor.js:28:28
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/inject.js:42:22
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:14)
  Error: past end:672420546-2374485795 < 673286541
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:17)
    at next (/home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:47:16)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:43:11
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:41:16
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11
    at get (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:29:7)
    at Object.readUInt32BE (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:92:9)
/home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/sinks/drain.js:16
          read(null, function (end, data) {
                              ^

RangeError: Maximum call stack size exceeded
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/sinks/drain.js:16:31
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-stream/throughs/async-map.js:28:19
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/skip.js:12:26
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/pull-cursor/cursor.js:28:28
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/inject.js:42:22
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:48:11
    at Object.read (/home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:49:14)
    at next (/home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:47:16)
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/flumelog-offset/frame/recoverable.js:43:11
    at /home/justin/src/github.com/ssbc/patchwork/node_modules/aligned-block-file/blocks.js:95:11

[christianbundy]

Are you able to use any version of ssb-server or Patchwork? This looks to me like your flumedb views are corrupted (see https://github.com/flumedb/flumedb/issues/32), but I've only ever personally seen this when running my flumedb deletion scripts.

You can regenerate all of your views by deleting everything in ~/.ssb/flume/ except for log.offset, but I might make a backup so that we can try to debug them. Sorry, that's a total pain for Scuttlebutt to just randomly explode without warning, hopefully just moving/removing your view files will help?

---

I'm sure you already know, but I'd recommend that you make a backup of ~/.ssb/secret and ~/.ssb/gossip.json before running rm anywhere near your SSB folder. It wouldn't hurt to make a backup of ~/.ssb/flume/log.offset either, but it's not quite as important as the first two.

[black-puppydog]

@christianbundy I checked; no sbot running. Also, this doesn't show up when stating with npm start, I also re-checked that.

[christianbundy]

Hmm, any chance you two are still having this issue? Sorry I dropped the ball on debugging this.

[justinabrahms]

I believe I moved my log.offset which "fixed" it, though that's not a very satisfactory fix.

[black-puppydog]

I haven't been running AppImages for a while now, that seems to have fixed it too :D

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?