Archivebox: Config: Add a way to blacklist URLs from being archived or indexed

[nodiscc]

There are some items in my bookmarks I wish were not archived (to save bandwidth and disk space), for example items I already have archived via other means (youtube-dl), or items that are just bookmarks, with no value in offline use, etc.

Would it be possible to add an option where the user can provide a list of URLs, which will never be archived?

They would still appear in the index archive, maybe as grayed out items? With the files/pdf/screenshot/a.org links removed?

[pirate]

How about env EXCLUDE_URLS=.*youtube.com.*,.*facebook.com/.*,.*.exe and env INDEX_EXCLUDED=True/False to choose whether they appear in the index (without files/screenshot) or are entirely removed.

(defined in config.py of course)

[nodiscc]

env EXCLUDE_URLS=.youtube.com.,.facebook.com/.,.*.exe

sound really good. Having this in the config file would be better than having to copy-paste it to the command line (especially if the exclude list gets long).

[issmirnov]

Great, started work on this.

@pirate do you prefer to keep EXCLUDE_URLS as a comma separated list of regex rules, or just have users provide strings and assume an implied .* suffix and prefix?

Personally I prefer the brevity of youtube.com,facebook.com over .*youtube.com.*,.*facebook.com.*

[pirate]

Explicit > implicit, lets keep full regex. This also allows us to exclude
extensions like .*+.exe$

[pirate]

Hey @issmirnov, just checking in. I might do this ticket at some point in the next couple months, if you have code you've worked on already you're welcome to share it as a PR and I'll incorporate it into my patch for this feature.

[issmirnov]

Hey @pirate, got swamped by life obligations so didn't have time to finish anything meaningful. All yours - thanks for checking in!

[pirate]

@mlazana this is a great issue to start with. You can practice adding a new config option, going through the archiving/parsing flow to skip certain URLs, and updating the documentation to describe the new config option.

[mlazana]

Great, I'll work on it!

[pirate]

This is done, thanks @mlazana!