Archivebox: Switch all dependencies to pure python and release ArchiveBox pip package

[pirate]

I originally thought moving to Python-only dependencies would be intractable, but after some more research I now realize this is quite straightforward.

• [ ] apt install curl -> pip install requests archivenow (requests docs, archivenow docs)
• [ ] apt install wget -> pip install wpull pywb (wpull docs, pywb docs)
• [ ] apt install git -> pip install GitPython (docs)
• [ ] apt install youtube-dl -> pip install youtube-dl (docs)
• [ ] apt install chromium-browser -> pip install pyppeteer (docs)

Then we wont need users to install any system dependencies anymore, and we can move to using only requirements.txt and setup.py to install ArchiveBox via pip.

[anarcat]

awesome, can't wait to see that one fly! :) let me know if you need help testing the stuff or get stuck.

[007]

Anything you're fetching with curl should be replaced with wget or vice versa, and that'll cut down on some dependencies in the pip translation.

[makeworld-the-better-one]

wpull only officially supports Python 3.4 and 3.5, even now it seems. The most recent commit was in Oct. 2019, and the version on PyPI is still outdated. It's a cool tool, but I would not recommend using it, and it doesn't seem to be well maintained.

If you still want to use it anyway, you can install it from Git, and then use a Python dependency manager to only use Python 3.5 for it, but I would not recommend that.

Git install:

pip3 install git+https://github.com/ArchiveTeam/[email protected]#egg=wpull

[pirate]

Yeah I looked at wpull recently and came to the same conclusion. Wget2 looks more promising than wpull.

I think I'm going to close this issue for now, as we start to expand the suite of archiving methods it's looking more and more like many of them will be node-based. Considering we already support pip install archivebox now to get the bulk of archivebox's functionality, and we offer all the methods out-of-the-box via docker, making everything python-only is no longer a priority.

[makeworld-the-better-one]

The other issue I see with this is managing conflicting versions of Python dependencies for these tools. I would personally recommend Poetry for that, as it's popular and I've had great experiences with it, but whatever you choose, I still think is an important step. Apologies if you were already going to do this.

I also don't see the value in replacing git with a Python version.