Beaker: Beaker keeps crashing after trying to add and publish a large website

[mroystonward]

Operation System: macOS 10.13.6 (17G65)
Beaker version: v0.8.0-prerelease.9

Likely the same issue as listed here - https://github.com/beakerbrowser/beaker/issues/827 - but new algorithm hasn't solved this (in my case at least).

I am attempting to mirror an existing site as a proof-of-concept. I have scraped my friends hosted wordpress site with wget and I get an offline copy of the site that is ~550MB with ~15,500 files (in various directories).

This is highly inefficient as content is duplicated, multiple times, in various folders relating to tags and searches etc.

As a result of this inefficiency I'm not sure if expect Beaker to handle this okay or not but I thought I'd raise it as it relates to other issues regarding hanging/crashing when working with large files/directories.

There's nothing in this filesystem than should be ignored etc.

If I'm being absurd, please feel free to close this issue.

I create a new archive from the folder containing the website mirror and as soon as I try and publish it I get CPU usage of ~150% and Beaker freezes up.

I have deleted and retried several times. If I try and publish one folder at a time (from the review UX) it manages the first 2 or 3 fine, then the system starts slowing and then it eventually grinds to a halt.

I have also tried creating an empty project and then dragging the website over, in managable chunks, but again, after a few folders of files, things slow until freezing.

I'm getting a hold of the wordpress xml export and will import into a jekyll site and see if that works better but for now, beaker/dat cannot handle my heavy handed mirror.

[pfrazee]

This is definitely a performance issue with the core Dat stack. We're about to roll out an update to its data structures that is designed to perform better with large datasets like this. I'll keep this issue updated when we deploy it.

[roscoevanderboom]

I had a similar issues when trying to add p2p chat to my dat archive. It's not a big project (about 200mb) but more than 6000 files. Beaker just hangs and doesn't finish publishing. (left it on over-night) This is a great little chat app that would be perfect if served through Beaker.

[oleometal]

me pasa lo mismo en mi caso solo trato de publicar 6000 archivos todos html pero se queda congelada en ubuntu 18 beaker version 0.8.4

[pfrazee]

Google translate:

the same thing happens to me in my case I just try to publish 6000 files all html but it stays frozen in ubuntu 18 beaker version 0.8.4

[mroystonward]

Just curious, is this closed because the upcoming update fixes things or because it's stale?

[pfrazee]

Sorry I didn't give a message, I had to close about 200 issues.

New update replaces that part of the stack with a new implementation which was designed to scale much better. If this still happens we'll reopen but it definitely shouldn't.