Adguardhome: AdGuard Home's syntax support could need some pretty serious improvements

[DandelionSprout]

Related to, but not the same as, #867.

Greetings, Dandelion Sprout here, maintainer of Dandelion Sprout's Nordic Filters and ~55 other filterlists. Meshkov can vouch for me, if it's needed.

Having now for the first time been able to set up AdGuard Home on an actual home server, there were a number of problems I had to figure out almost on my own.

1) I was pretty disappointed to see that domains lists are not currently supported in AdGuard Home, despite them making up a good 40% of all filter lists that are known to exist. So I hereby request adding support for them.
2) As pointed out in #867, entries with most $ modifier are simply thrown in the bin, instead of being narrowed down to their supported criteria. Thus it's rather weird to see ||example.com$important be supported, whereas ||example.com$empty,important is not, despite how $empty is even an AdGuard-brand-specific modifier. Examples can potentially be made for entries that rely on e.g noop-js and similar super-specific modifiers; those can maybe or maybe not remain ignored.
3) AdGuard Home's hosts file support seems to be identical to that of other hosts-file tools, which leads to the always unfortunate problem that has been inherent to the hosts system for 35 years, in which one needs to know if a domain is prefaced by www. or not, and where 127.0.0.1 www.example.com and 127.0.0.1 example.com are treated separately. I hereby propose supporting or adding an option for treating www. and non-www. hosts entries the same.
4) On top of all of the above three, I propose supporting or adding an option to treat domains rules (example.com, as in section №1) and hosts rules (127.0.0.1 [www.]example.com, as in №3) as if they were ||example.com^$important. For a lot of sites that such rules tend to cover, one would sometimes want or have to block way more broadly than just the front page.

Steps to reproduce

I'll use the extremely safe and ad-free Norwegian newssite nrk.no as the example for this, for convenience's sake.

1. Paste any one of either nrk.no, www.nrk.no, 127.0.0.1 nrk.no, or nrk.no^$empty,important into Custom filtering rules.
2. Try to load nrk.no on a unit that is using such a rule.
3. See that the page loads normally.
4. Paste 127.0.0.1 www.nrk.no or ||nrk.no^$important into Custom filtering rules.
5. See that the page now successfully fails to load.

Expected behavior

Actual behavior

Screenshot:

Unsure what I'd have to take a screenshot of, in that case.

Your environment

| Description | Value |
| -------------- | ------------ |
| Version of AdGuard Home server:| v0.97.1 ARM, seemingly 32-bit
| How did you setup DNS configuration:| System, if I understand the question correctly
| If it's a router or IoT, please write device model:| Raspberry Pi 4
| Operating system and version:| Raspbian "July 2019"
| Client unit that I tested the syntax support on:| Lenovo K5 Vibe, LineageOS 14.1, using DNS Forwarder Pro

[ameshkov]

Hi there!

I was pretty disappointed to see that domains lists are not currently supported

They actually are supported since v0.96. If the rule is just a domain name, it is treated as 0.0.0.0 example.org. This kind is behavior is expected from us by most of the popular hosts lists (like Energized or Malware domains).

Thus it's rather weird to see ||example.com$important be supported, whereas ||example.com$empty,important is not

Generally, our approach is simple: if the modifier is not in the list of supported, the whole rule will be discarded.

The easiest solution would be to add $empty to the list of supported modifiers, but I am not sure if this is the right way. $empty makes AdGuard return an empty response instead of blocking the request. This outcome is different from "blocking" as it does not look like an error to the page's script. My thinking is that when a filter list maintainer opts to use $empty or noopjs, they usually do it because simply blocking the request does not help or not enough.

If you think that there are some other modifiers that can be added to the list of supported, please tell me which ones. I am thinking about $document for both blocking and exception rules.

AdGuard Home's hosts file support seems to be identical to that of other hosts-file tools

But this is the point! These hosts lists are supposed to be used not just with AdGuard Home, but with any other tool that supports this syntax. Changing semantics on our side will make AdGuard Home different from others and unpredictable for those who create these lists.

Also, we do not encourage using hosts files syntax, and we don't use it in the AdGuard SDN filter. It's too limited compared even to the simple basic rules syntax, and the hosts lists tend to be huge as hell.

I hereby propose supporting or adding an option for treating www. and non-www. hosts entries the same.

I guess we could do this for the domains-only lists. Unlike /etc/hosts syntax, we're more or less free in how we interpret them.

[mmotti]

I guess we could do this for the domains-only lists. Unlike /etc/hosts syntax, we're more or less free in how we interpret them

Need to be careful with this. I got caught out with domains such as www.be being replaced with ||be^ which blocks all sorts, such as youtu.be

I ended up running a string substitution for www. only where there is another proceeding .: ^www\.(?=(?:[^.\n]+\.){1,}[^.\n]+$)

[DandelionSprout]

They actually are supported since v0.96. If the rule is just a domain name, it is treated as 0.0.0.0 example.org.

Okay, so now that I saw that answer, I decided to use my main PC as a test client instead of my phone (thinking there could be something wrong with the app I used or with Firefox for Android), as well as testing with multiple sites that my browsers had far smaller local caches for, I can now confirm that domains really are supported. I thank you for clueing me onto it.

If you think that there are some other modifiers that can be added to the list of supported, please tell me which ones.

Off the top of my head, I think $third-party and $~third-party (and uBO's special versions thereof), $script, $generichide, $subdocument, and $image could be worth looking into adding support for.

(The stuff about hosts files)

Good points. I too find the 'hosts' syntax to be woefully inadequate in 2019, but unfortunately a lot of old-timers and Android enthusiasts like Dan Pollock, Steven Black, and Energized base their filterlist making on it.

Need to be careful with this. I got caught out with domains such as www.be being replaced with ||be^ which blocks all sorts, such as youtu.be

A workaround that sounds easy enough to my non-technical ears, could be to only clone(?) www. entries if it has two period dots instead of one.

[mmotti]

The main issue with including the options are that as a standard dns request, it's impossible to distinguish first party from third party, whether it's a script etc.

The other difficulty is that we are applying filters intended mainly to be browser level, network wide. So things such as ||ospserver.net^$third-party may be fine for your browser, but if you have a Samsung phone then it'll block updates.. And this is just one example. I'm sure the image and script options will likely include many otherwise legitimate domains.

Unfortunately DNS based blocking really is rather primitive in comparison to filter lists. You can think of the filter support only really as a way to wildcard blacklist / whitelist.

This feature alone is incredibly helpful though. It cuts down host file sizes to a fraction of the size if you convert everything to wildcards and remove the subdomains.

This is one of the several huge advantages over pihole.

[ameshkov]

Well, on the other hand, it's not too long until we roll out the experimental MITM proxy as a part of AdGuard Home. Unlike the DNS server, it will be able to apply content blocking filter lists properly.

[mmotti]

That certainly sounds interesting. I have however heard of issues with regards to custom certificates / clients and some apps being tricky to trust them without root - Is this similarly the case with the proxy?

Regardless I'd be very eager to give this a try providing my RPI 3b could handle it

[ameshkov]

I have however heard of issues with regards to custom certificates / clients and some apps being tricky to trust them without root - Is this similarly the case with the proxy?

Yeah, there will be some issues on Android, hopefully, we'll find a solution. Things are a little bit easier on iOS.

[DandelionSprout]

I've ended up created something of an intermission non-automatic compilation list of my own, until more $ modifiers become processed or bypassed properly: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/AdGuard%20Home%20Compilation%20List/AdGuardHomeCompilationList.txt

It compiles 6 lists whose blocking rules are currently not particularly supported by AdGuard Home: EasyList, uBlock Filters, uBlock Filters - Badware Risks, Nano Defender Integration, Liste FR, and I Don't Care About Cookies. More lists may or may not be added to it. It'll probably be re-synced with the upstream lists every 1-3 weeks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[DandelionSprout]

Yes, bot, I take the hint. I'll probably create separate issues for some of the $ values soon enough, if I have time and a personal need to do so.

[DandelionSprout]

I've essentially moved the remnants of this to #1026 now, so I presume this issue report can be closed now.