Archivebox: Architecture: Use multiple cores to run link archiving in parallel

[pirate]

Add a --parallel=8 cli option to enable using multiprocessing to download a large number of links in parallel. Default to number of cores on machine, allow --parallel=1 to override it to 1 core.

[pirate]

Inspired by https://github.com/aurelg/linkbak I've had this on my mind for a while since it's super easy to implement, but @aurelg inspired me to actually make an issue for it.

[aurelg]

The relevant code is in this file:

    nb_workers = args.j if args.j else os.cpu_count()
    get_logger().warning("Using %s workers", nb_workers)

    if nb_workers > 1:
        with contextlib.closing(multiprocessing.Pool(nb_workers)) as pool:
            pool.starmap(start_link_handler,
                         [(l, args) for l in get_links(args.file[0])])
    else:
        for link in get_links(args.file[0]):
            start_link_handler(link, args)

This is actually pretty easy. In your case, the only difficulty might be to handle the screen output/progression bar properly: if different workers are updating the screen at the same time, it may quickly become a bit messy.

[pirate]

I think I can fix the parallel process stdout multiplexing problem with one of two solutions:

• a program like curtsies two show each stream in a grid, tail -f style
• having each subprocess >> /logfile, but obtain a lock for the duration of each link archive, so the output remains in contiguous chunks instead of being mixed together line by line:

some simplified pseudocode:

import fcntl

def archive_links(link):
    link, stdout = run_archive_methods(link)

    with open(f'data/logs/archive.log', 'a') as f:
        fcntl.flock(f, fcntl.LOCK_EX)
        f.write(stdout)
        fcntl.flock(f, fcntl.LOCK_UN)

if parallel > 1:
    pool = subprocess.Pool(count=parallel)
    pool_status = pool.map_async(bar, links)
    last_pos = 0
    while not pool_status.ready():
        with open(f'data/logs/archive.log', 'r') as f:
            f.seek(last_pos)
            print(f.read())
            f.seek(0, os.SEEK_END)
            last_pos = f.tell()
        sys.stdout.flush()
        mr.wait(0.2)
else:
    for link in links:
        link, stdout = run_archive_methods(link)
        print(stdout)

[anarcat]

parallel downloading is tricky - you need to be nice to remote hosts and not hit them too much. there's a per-host limit specified in RFCs which most web browsers override. most clients do 7 or 10 requests per domain, IIRC.

but since you're crawling different URLs, you have a more complicated problem to deal with and can fetch more than 7-10 simultaneous queries globally... you need to be careful because it requires coordination among the workers as well, or have a director that does the right thing. in my feed reader, I believe I just throw about that number of threads (with multiprocessing.Pool and pool.apply_async, see the short design discussion) at all sites and hope for the best, but that's clearly inefficient.

I came here because I was looking at a bug report about running multiple archivebox add in parallel: I think this is broken right now (see #234 for an example failure) because the index.json gets corrupted or stolen between processes. it would be nice to have at least a lock file to prevent such problems from happening.

[LaserWires]

parallel downloading is tricky - you need to be nice to remote hosts and not hit them too much. there's a per-host limit specified in RFCs which most web browsers override. most clients do 7 or 10 requests per domain, IIRC.

but since you're crawling different URLs, you have a more complicated problem to deal with and can fetch _more_ than 7-10 simultaneous queries globally... you need to be careful because it requires coordination among the workers as well, or have a director that does the right thing. in my feed reader, I believe I just throw about that number of threads (with multiprocessing.Pool and pool.apply_async, see the short design discussion) at all sites and hope for the best, but that's clearly inefficient.

I came here because I was looking at a bug report about running multiple archivebox add in parallel: I think this is broken right now (see #234 for an example failure) because the index.json gets corrupted or stolen between processes. it would be nice to have at least a lock file to prevent such problems from happening.

Most site mirroring apps incorporate download by proxy it is a very common feature which might be implemented into archivebox hence a large --parallel value is of no issue with such a feature. A list list of proxies can enable an archive operation to use a very large --parallel value. Most webservers currently are very adequately suited with high bandwidth telecoms along with very capable hardware and users with archivebox overloading them isn't likely an issue.

[pirate]

It's not an issue of overloading archivebox, it's an issue of overloading / hitting rate-limits on the content servers, which piping through a proxy wont solve.

[karlicoss]

Even more cores than 8 might make sense, because often things are blocked on IO with no throughput, e.g. pages that would timeout. Might need some careful scheduling, but would be very cool to have!
Another IMO useful thing is having some sort of "pipeline" concurrency, e.g. one executor only archives DOM and always runs in front. The other executors run behind and handle singlepage/screenshots/media/etc, i.e. slower, but not as essential bits. This might also make it easier to schedule the load depending on which archivers are IO/CPU bound.

[pirate]

A quick update for everyone watching this, v0.5.0 is going to be released soon with improvements to how ArchiveResults are stored (we moved them into the SqliteDB). This was a necessary blocker to fix before we can get around to parallel archiving in the next version.

v0.5.0 will be faster, but it wont have built-in concurrent archiving support yet, that will be the primary focus for v0.6.0. The plan is to add a background task queue handler like dramatiq or more likely huey (because it has sqlite3 support so we don't need to run redis).

Once we have the background task worker system in place, we can implement a worker pool for Chrome/playwright and each of the other extractor methods. Then archiving can run in parallel by default, archiving like 5-10 sites at a time depending on the system resources available and how well the worker pool system performs for each extractor type. Huey and dramatic both have built-in rate limiting systems that will allow us to cap the number of concurrent requests going to each site or being handled by each extractor. It's still quite a bit of work left, but we're getting closer!

Having a background task system will also enable us to do many other cool things, like building the scheduled import system into the UI #578, using a single shared chrome process instead of relaunching chrome for each link, and many other small improvements to performance.