Beaker: Beaker won't respond after attempting to add a large directory to library

[AlbertoElias]

Operation System: Windows 10
Beaker Version: Latest (can't check exactly as it won't load properly)

I attempted to add a large directory to my library. First I tried by creating a new site, and changing its directory to my site's one, but that didn't work, maybe due to the .dat directory. So I then tried to use the Dat CLI and created a Dat directory manually, and then ran dat share. That allowed to visit my site from Beaker, but I then clicked __Add to my library__ and that froze Beaker. And now when I attempt to open it, it won't respond. (The directory is large due to the node_modules directory that I haven't found a way to ignore it)

[pfrazee]

Yeah we're discovering that the current Dat algorithm has some bad performance characteristics when dealing with large numbers of files. Here are the plans to mitigate that:

• In Beaker 0.8, we're improving the flows so that you don't end up forced to commit directories like node_modules (sorry about that!) or even processing them unless you're ready to.
• In future versions of Beaker, we're going to move the Dat networking stack into a separate thread/process so that large workloads will not block the thread. An ideal future will use multiple threads for the Dat stack in order to handle jobs like this in isolation.
• The Dat protocol team is going to keep refining the algorithms and implementations.

With that said, let me address your specific issue -- is Beaker unresponsive even when you restart it, or only when you try to visit the site?

[AlbertoElias]

That's great to hear. @taravancil told me on Twitter about .datignore, and I created one with node_modules/**, though I haven't figured out how to publish an updated archive without node_modules.

Back to the issue, Beaker is unresponsive when restarting, it's unable to load the new tab page. It stays blank, unresponsive, and Windows just ends up killing it by itself. I'm guessing reinstalling might help, but wanted to keep it in case there's any info I can send you that might help to debug

[pfrazee]

Before a reinstall, try renaming the directory on your filesystem that you added to your library. It's possible that what's happening is that Beaker is running a costly diff on startup between what it has in the library and what you have in the folder. EDIT: to elaborate on why- renaming will cause Beaker to lose track of the source folder and abort the diff.

[pfrazee]

@AlbertoElias oh wait, did you create this dat with the cli?

[AlbertoElias]

Renaming didn't seem to do the trick.

Yes, I created it with the CLI, visited the URI from Beaker, and then clicked Add to Library. I tried first to do it from the Library UI but was unsuccessful

[pfrazee]

Hm, ok that's going to make this trickier.

Open the AppData folder for your beaker. On windows 10 it should be something like C:\Users\myusername\AppData\Beaker Browser. Inside there, go to Dat\Archives\Meta and you should see some folders with two-letter names.

The folder that contains the dat's data is in there. So if your dat's url is 0123456789abcdef then the folder would be Dat\Archives\Meta\01\23456789abcdef.

Delete that folder and start beaker again. (Dont leave your dat cli running with the dat that's causing the problems.) Go to your library, see if you can remove the offending dat.

Good luck and sorry for the headache.

[AlbertoElias]

Yup! That worked! In case anyone else runs into this issue, in my case, it was in C:\Users\myusername\AppData\Roaming\Beaker Browser

Now, when going to my library, if I click on that archive, I get a TimeoutError: Timed out error and the delete menu doesn't appear.

[pfrazee]

Now, when going to my library, if I click on that archive, I get a TimeoutError: Timed out error and the delete menu doesn't appear.

Blah of course. Well if you want to fix that, open the devtools within the library and run

beaker.archives.remove('<dat url>')

[AlbertoElias]

Great! Everything is back to normal. I'll try readding the site once I get the files ignored. Thank you very much for your help, really appreciate it!

Should I close this issue or is it something that needs to be looked into to prevent in the future?

[pfrazee]

Glad we got it. Thanks for being patient with us.

I'll keep this open till 0.8 lands, at least.

[pfrazee]

New algorithm has landed in the beta. We're going to close this. Reopen (or open a new issue) if there's still an issue.