Beaker: Failed to connect to daemon, retrying

[todrobbins]

Describe the bug
The app will not launch stuck at Failed to connect to daemon, retrying:

Failed to connect to daemon, retrying [info]
{ "category": "hyper", "subcategory": "daemon", "message": "Failed to connect to daemon, retrying", "level": "info", "timestamp": "2020-05-22T19:43:58.227Z", "type": "data", "target": {} }

To Reproduce
Steps to reproduce the behavior:

1. Open Beaker Browser
2. Daemon error persists

Expected behavior
Beaker loads

Screenshots

Desktop (please complete the following information):

• OS: macOS 10.15.4
• Browser Beaker
• Version 1.0.2

Additional context
I checked hyperdrive status as well before opening. I think the daemon is not staying active (these commands are being run right after each other):

➜  ~ hyperdrive start
✔ Hyperdrive daemon listening on localhost:3101
➜  ~ hyperdrive status
The Hyperdrive daemon is not running.

[pfrazee]

Anything show up in ~/.hyperdrive/log.json ?

[todrobbins]

Here you go:

{"level":30,"time":1590358669119,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","msg":"memory only? false no announce? false","v":1}
{"level":30,"time":1590358669341,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","swarmId":"dc7374159b7d6e14a20e8b9518146dd21993423dbf018edc9431185ebb6eb11c","seed":"34bf82f23c01b4332a43124fbbeb3850285040fb589b7f01460fad92262e31d5","msg":"creating replication keypair and swarm ID","v":1}
{"level":50,"time":1590358702757,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","error":{},"stack":"Error: Mountpoint in use\n    at /Applications/Beaker Browser.app/Contents/Resources/app/node_modules/fuse-native/index.js:317:60\n    at FSReqCallback.oncomplete (fs.js:159:5)","msg":"error in daemon ready function -- cleaning up","v":1}
{"level":50,"time":1590358702758,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","error":true,"err":{"type":"Error","message":"Mountpoint in use","stack":"Error: Mountpoint in use\n    at /Applications/Beaker Browser.app/Contents/Resources/app/node_modules/fuse-native/index.js:317:60\n    at FSReqCallback.oncomplete (fs.js:159:5)"},"message":"Mountpoint in use","stack":"Error: Mountpoint in use\n    at /Applications/Beaker Browser.app/Contents/Resources/app/node_modules/fuse-native/index.js:317:60\n    at FSReqCallback.oncomplete (fs.js:159:5)","msg":"stopping daemon due to error","v":1}
{"level":30,"time":1590358702759,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","msg":"waiting for fuse to unmount","v":1}
{"level":30,"time":1590358702759,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","msg":"waiting for networking to close","v":1}
{"level":30,"time":1590358786947,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","msg":"waiting for corestore to close","v":1}
{"level":30,"time":1590358786954,"pid":77670,"hostname":"Todotron.local","name":"hyperdrive","component":"server","msg":"waiting for db to close","v":1}

[todrobbins]

@pfrazee 👆🏻 just in case you missed it.

[todrobbins]

And this what I see in the console when building/running from source (latest pull):

info:     Starting hyperdrive {"category":"browser","subcategory":"init"}
info:     Failed to connect to an external daemon. Launching the daemon... {"category":"hyper","subcategory":"daemon"}
info:     Starting daemon process, assuming process control {"category":"hyper","subcategory":"daemon"}
(node:50264) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(node:50264) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
info:     Failed to connect to daemon, retrying {"category":"hyper","subcategory":"daemon"}
info:     Failed to connect to daemon, retrying {"category":"hyper","subcategory":"daemon"}
info:     Failed to connect to daemon, retrying {"category":"hyper","subcategory":"daemon"}
info:     Failed to connect to daemon, retrying {"category":"hyper","subcategory":"daemon"}
info:     Failed to connect to daemon, retrying {"category":"hyper","subcategory":"daemon"}
info:     Hyperdrive daemon has been restored {"category":"hyper","subcategory":"daemon"}

[pfrazee]

@todrobbins aha yep I know what's up. You have a zombie FUSE mount. Try installing the hyperdrive-daemon and running its force-unmount command.

[todrobbins]

Trying that now. Interestingly, I just did a port scan and nothing came up under :3101

[todrobbins]

Uh...

➜  scripts git:(master) hyperdrive force-unmount
Password:
Unmount failed for /Users/trobbins/Hyperdrive
Error: Command failed: sudo diskutil unmount /Users/trobbins/Hyperdrive
Unmount failed for /Users/trobbins/Hyperdrive
    at ForceUnmountCommand.run (~/.nvm/versions/node/v12.13.1/lib/node_modules/hyperdrive-daemon/node_modules/hyperdrive-daemon-client/bin/commands/force-unmount.js:14:7)
    at ForceUnmountCommand._run (~/.nvm/versions/node/v12.13.1/lib/node_modules/hyperdrive-daemon/node_modules/@oclif/command/lib/command.js:43:31)

hyperdrive-daemon/1.13.12 darwin-x64 node-v12.13.1

[pfrazee]

hmm, @andrewosh any thoughts?

[andrewosh]

Yeah @todrobbins can you try manually running diskutil unmount force /Users/trobbins/Hyperdrive?

Unclear why the mountpoint would have been left in a bad state. Do you know if the daemon was force-killed or crashed before you started seeing this?

[todrobbins]

@andrewosh I didn't even know about force-unmount until @pfrazee suggested it 😂

And...

➜ diskutil unmount force /Users/trobbins/Hyperdrive
Unmount failed for /Users/trobbins/Hyperdrive

[andrewosh]

Yeah it's one of those unfortunately-necessary things that are occasionally needed (FUSE cleanup can be a real pain). If you ls ~/Hyperdrive are you seeing an error like Transport endpoint is not connected, or is it an empty folder?

Want to make sure that the force-unmount is still necessary.

[todrobbins]

ls: Hyperdrive: Device not configured

[todrobbins]

But I do still see it in Finder:

And this is what happens when I double-click on it:

[andrewosh]

Ack. Alright do you have any Finder windows open that are pointed anywhere in the ~/Hyperdrive tree?

Yeah the mountpoint will still be visible even if it's misbehaving.

Sometimes when the mountpoint is "in use" (Finder windows) it'll block unmounting it, even "forcefully"...real piece of work.

[todrobbins]

I don't have any visibility into ~/Hyperdrive in Finder. Sorry.

[andrewosh]

No prob, was just wondering if you had a Finder window currently open pointed to any directory inside ~/Hyperdrive -- that's been problematic in the past (blocking proper cleanup), though we do have a test that opens Finder windows and makes sure they auto-close when the mountpoint disconnects.

Let me dig around a bit.

[andrewosh]

Hey @todrobbins apologies for being slow to reply. It's pretty crude, but have you tried closing all active terminals or Finder windows?

I'm scratching my head as to why the diskutil unmount force wouldn't cut it, and unfortunately there aren't many options beyond that. Empirically, when I've seen this scenario in the past, it's because some program is still operating on the FUSE mount, and when that program is closed it can be unmounted.

If there aren't any Finder windows or terminals open (with their current working directories being somewhere inside ~/Hyperdrive), then alas a reboot might be in order. Apologies for that -- this is a nasty state to wind up in.

[todrobbins]

Ok... I didn't have any Finder windows or terminal sessions open, but what fixed it was when I deleted the shortcut to Hyperdrive in Finder, emptied my Trash, and then ran hyperdrive start again and it stayed open. I booted up Beaker and it's working like normal.

👻 SPOOPY

[todrobbins]

So the mount's alias was haunted 😨 or something LOL

[mafintosh]

@todrobbins so you got it working now? I’ve noticed that umount sometimes work also when diskutil doesn’t

[todrobbins]

Yes, working now but all it took was removing the Hyperdrive alias in Finder. 🤷‍♂️

[mafintosh]

@todrobbins gotta pay that early adopter tax

[aadilayub]

Facing this same issue, though I'm not sure if it's for the same reason (I can't see any hyperdrives mounted in Finder)

Here's the log.json:

{"level":30,"time":1592750561255,"pid":2455,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"memory only? false no announce? false","v":1}
{"level":30,"time":1592750561501,"pid":2455,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","port":3101,"msg":"server listening","v":1}
{"level":50,"time":1592750574850,"pid":2455,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","error":true,"err":"SIGINT","msg":"stopping daemon due to error","v":1}
{"level":30,"time":1592750574851,"pid":2455,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"waiting for fuse to unmount","v":1}
{"level":30,"time":1592750574851,"pid":2455,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"waiting for networking to close","v":1}

[aadilayub]

Don't see any errors when trying to start hyperdrive from the terminal:

[pfrazee]

@aadilayub are you in macos? (looks like yes)

It looks like init is failing because hyperdrive status says it's not running after start.

Could you check ~/.hyperdrive/log.json for any errors? (If you share make sure not to include any drive URLs you dont want to leak)

[aadilayub]

@pfrazee yes, I'm in macos!

hmm you're right about the daemon not running

can't find any errors in log.json

{"level":30,"time":1592764785357,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"memory only? false no announce? false","v":1}
{"level":30,"time":1592764785572,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","port":3101,"msg":"server listening","v":1}
{"level":30,"time":1592764845572,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.7671,"memory":{"percent":0.0164,"rss":141062144,"total":8589934592,"heapPercent":0.9543,"heapUsed":68618008,"heapTotal":71905280,"external":23065592},"delay":0,"time":2,"msg":"process stats","v":1}
{"level":30,"time":1592764905571,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.903,"memory":{"percent":0.0169,"rss":145346560,"total":8589934592,"heapPercent":0.9401,"heapUsed":68833140,"heapTotal":73216000,"external":25218650},"delay":null,"time":1,"msg":"process stats","v":1}
{"level":30,"time":1592764965571,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.8573,"memory":{"percent":0.0169,"rss":145416192,"total":8589934592,"heapPercent":0.941,"heapUsed":68897024,"heapTotal":73216000,"external":27221437},"delay":null,"time":1,"msg":"process stats","v":1}
{"level":30,"time":1592765025573,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.8869,"memory":{"percent":0.0169,"rss":145235968,"total":8589934592,"heapPercent":0.94,"heapUsed":68819904,"heapTotal":73216000,"external":27256943},"delay":null,"time":1,"msg":"process stats","v":1}
{"level":30,"time":1592765085574,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.871,"memory":{"percent":0.017,"rss":146001920,"total":8589934592,"heapPercent":0.9476,"heapUsed":69382384,"heapTotal":73216000,"external":28638469},"delay":null,"time":1,"msg":"process stats","v":1}
{"level":30,"time":1592765145573,"pid":411,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.9136,"memory":{"percent":0.017,"rss":145711104,"total":8589934592,"heapPercent":0.944,"heapUsed":69118852,"heapTotal":73216000,"external":27755648},"delay":null,"time":0,"msg":"process stats","v":1}

[aadilayub]

@pfrazee any pointers on how to fix this? Was considering doing a reinstall but I don't want to lose my profile

[pfrazee]

@aadilayub Have you restarted your computer since this started? See if there are any processes in your activity manager named Beaker, Hyperdrive, or Electron, and close them.

[aadilayub]

@pfrazee Couldn't find any Beaker/hyperdrive/electron/node related processes in Activity Monitor. I restart pretty much every day and have beaker set to launch on boot.

[pfrazee]

@aadilayub is both your beaker and hyperdrive daemon at latest releases?

If you run the latest hyperdrive daemon by itself, and then status shows its not running, you don't see any error in ~/.hyperdrive/log.json?

[aadilayub]

@pfrazee yep, both hyperdrive and beaker are latest version. To the second question, also yes.

I think this might have been caused by me force-closing Beaker before shutdown last week. It hasn't worked since.

[aadilayub]

Did a reinstall today (deleted Beaker Browser.app, downloaded and copied new executable from .dmg), doesn't seem to have had any effect.

[aadilayub]

Oh, I found some errors in ~/.hyperdrive/log.old.json!

{"level":30,"time":1593332245934,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"memory only? false no announce? false","v":1}
{"level":30,"time":1593332246383,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","port":3101,"msg":"server listening","v":1}
{"level":30,"time":1593332306389,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.8045,"memory":{"percent":0.0577,"rss":495333376,"total":8589934592,"heapPercent":0.9594,"heapUsed":353784840,"heapTotal":368754688,"external":42404951},"delay":0,"time":1,"msg":"process stats","v":1}
{"level":30,"time":1593332366407,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.8751,"memory":{"percent":0.057,"rss":489570304,"total":8589934592,"heapPercent":0.9806,"heapUsed":354905392,"heapTotal":361938944,"external":45457407},"delay":null,"time":0,"msg":"process stats","v":1}
{"level":30,"time":1593332426412,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","cpu":0.8831,"memory":{"percent":0.057,"rss":490008576,"total":8589934592,"heapPercent":0.9804,"heapUsed":355354384,"heapTotal":362463232,"external":45826068},"delay":null,"time":1,"msg":"process stats","v":1}
{"level":50,"time":1593332450720,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","error":true,"err":"SIGINT","msg":"stopping daemon due to error","v":1}
{"level":30,"time":1593332450720,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"waiting for fuse to unmount","v":1}
{"level":30,"time":1593332450720,"pid":1262,"hostname":"Aadils-Thinkpad.local","name":"hyperdrive","component":"server","msg":"waiting for networking to close","v":1}

[aadilayub]

Tried hyperdrive force-unmount, and got the same error as @todrobbins, presumably because there's nothing to unmount (I don't see a Hyperdrive alias in Finder, and never did previously even when Beaker was running correctly)

Sorry, try again.
Unmount failed for /Users/aadil/Hyperdrive
Error: Command failed: sudo diskutil unmount /Users/aadil/Hyperdrive
Sorry, try again.
Unmount failed for /Users/aadil/Hyperdrive
    at ForceUnmountCommand.run (~/.nvm/versions/node/v12.16.3/lib/node_modules/hyperdrive-daemon/node_modules/hyperdrive-daemon-client/bin/commands/force-unmount.js:14:7)
    at ForceUnmountCommand._run (~/.nvm/versions/node/v12.16.3/lib/node_modules/hyperdrive-daemon/node_modules/@oclif/command/lib/command.js:43:31)

[aadilayub]

oof, turns out it wasn't working because there was a typo in the first line of my hosts file. Problem resolved thanks to troubleshooting guidance from @andrewosh

[jessehattabaugh]

I'm experiencing this issue on Windows. I just installed Beaker so I've never mounted any FUSE filesystems that I'm aware of, and I don't have the stand alone hyperdrive daemon installed. I've tried deleting the ~/.hyperdrive directory, but it just gets recreated.

Should I try running the daemon manually?