Filterlists: add lists' syntaxes and the software that supports them

[collinbarrett]

Adblock lists are only useful in a browser-based solution. Host lists can be used in a browser-based solution as well as in a host file to block entire domains/IPs machine-wide. FilterLists should index lists based on their syntax to assist users.

[collinbarrett]

Possibly the same feature as https://github.com/collinbarrett/FilterLists/issues/11

[bershan2]

This probably should contain the syntax information (as claimed by the authors). In a simplified manner this might look like "syntax" property with possible values "hosts", various revisions of Adblock Plus syntax, and even uBlock Origin own syntax (because it is slightly different from ABP's). Fortunately, this information can be extracted automatically: uBO filters are all under uAssets, Adblock Plus filters mostly contain a line [Adblock Plus x.y] with x.y being the major.minor revisions. If ABP syntax filter does not specify the revision, we could (1) assume the latest revision or (2) use automatic analyzer since ABP syntax is always extended and not substituted (as far as I know).

[collinbarrett]

In building out the next version of the site, I am looking at the following basic syntaxes to use as tags to start. Am I missing anything to your knowledge, @bershan2 or @gorhill ?

Filter List Syntaxes:

• uBlock Origin (Extension of Adblock Plus)
• Adguard (Extension of Adblock Plus)
• Adblock Plus (Extension of Domains)
• Domains
• Hosts
• Exceptions (Whitelist)

*I'm aware that there is overlap between some of these, too, but I'd like to tag each list with a syntax as a tool for users.

Links on syntaxes for my reference:

• https://github.com/gorhill/uBlock/wiki/Static-filter-syntax
• https://adblockplus.org/filter-cheatsheet
• https://adblockplus.org/filters
• https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters

[elypter]

you could also add iptables, hosts.deny, pf and other firewall rule syntaxes

[collinbarrett]

The current working list of syntaxes is as follows. Please provide feedback on additions/modifications.

• Hosts
• Domains
• Adblock Plus
• uBlock Origin
• AdGuard
• Exceptions (Whitelist)
• URLs
• IPs

[elypter]

is Exceptions really a syntax? cant there be whitelists in many different formats? i think it is rather a function than a syntax.

about the syntaxes i know a few but i could not name an example for every one of them. so not sure if those should be put in the list but here they are.

• uMatrix/uBlock dynamic rules example: https://github.com/uMatrix-Rules/uMatrix-Rules-Domain/
• URLs (Wildcard)
• URLs (Regular expression)
• Iptables firewall syntax
• hosts.deny another firewall syntax

there are more but those are probably the most useful ones.

[collinbarrett]

thanks, @elypter . I agree. I wanted some way to distinguish lists that are exceptions/whitelists, but I agree that it is not a syntax. maybe just a boolean flag that says it is a whitelist or not.

thanks for the other suggestions as well. will evaluate.

[elypter]

i would not make it a boolean. i think it should be able to have the values blacklist, whitelist, mixed and none. the last one would be for example for lists that just modify the style of some elements, or for lists that only use the badfilter argument, or anything that contains only metadata.

[collinbarrett]

thanks, @elypter . that makes sense.

[collinbarrett]

Update:

The current working design for this and #11 is that there is a one-to-one relationship between list and syntax and a many-to-many relationship between syntaxes and software that supports it. I am still working through updating the dataset to fill out these relationships. I am currently foreseeing another relationship that will model what lists are shipped with as well as default enabled by each software. But, to me, that's of less importance than for a user to be able to just filter lists by what software it can be used in.

I have not yet implemented @elypter 's suggestion of blacklist/whitelist/mixed/none. Still evaluating how that will work, and I want to find several example lists of each before spending time implementing. But, I agree with the idea.

Data Model Implementation: data and entities.

[elypter]

this is an example of a mixed list https://github.com/collinbarrett/FilterLists/issues/176 which can also be split in a blacklist and a whitelist.

i have no example for "none"
this is an entirely cosmetic list https://github.com/yourduskquibbles/webannoyances/ but cosmetic vs access filters should be a different property if it is needed even if the filters just inject something or change visuals. i cant think of any "none" list yet but i think it is better to have and not need it than the other way round.

[collinbarrett]

closing for now as this is pretty well fleshed out in data model. will add new syntaxes as needed going forward. thanks for everyone's input.