Adaway: Allowed list (from local file) not working

[zgfg]

Thanks for implementing/fixing the previous #2131 about reading a textual list from local storage. Tested with v5.0.7 and Blocked list and it works

However, cannot make it working with Allowed list:

• Create a textual file e.g. myWhiteList.txt on Internal memory, Download folder
• Add an entry like:
  my.allowed.com
• Go to Hosts sources list, press +, select FILE and navigate to the myWhiteList.txt
• By default Apply allowed hosts is enabled (? - IMO, by default most users will be adding Blocked hosts, not Allowed)
• Take Apply Confirmation changes, it doesn't report any error and it says Up-to-date
• Check in the Allowed list, there is no domain my.allowed.com listed
• Make adaway-backup.json, it shows an entry to the local storage file myWhiteList.txt with "enabled" and "allow" both set to true

AdAway 5.0.7
Root or VPN

[PerfectSlayer]

Hi @zgfg

Thanks for this new report. I saw there is some issues with allowed hosts when I rework the source UI but didn't had time to fix it yet.
But you guessed it right, allowed hosts are not working.

How is it supposed to work?

The two _Apply allowed hosts_ and _Apply redirected hosts_ options are hosts file parser options.
Redirected hosts are hosts where IP are not localhost (127.0.0.1, ::1). If you enable the option, the redirection will not be dropped.
Allowed hosts are the trickiest part... It is not supported the _standard hosts format_ (it seems there is no standard for hosts file format…).

So it previously was allow <host_pattern>. For example: allow *.google.com

It is not a whole file that contains hosts to allow or hosts to block.
In summary, you can write in a host file:

127.0.0.1 to-block.com
allow to-allow.*
1.2.3.4 to-redirect.com

The second line will be dropped if _Apply allowed hosts_ is not checked and the third line if _Apply redirected hosts_ is not checked.

The issue

Yeah, I broke the allowed hosts features long time ago when I start enforcing the hosts file format, requiring an IP address as the start of each line (except for comments or empty lines).

For me, the allow <host_pattern> format is not good. It could break some systems if used as is.
What about introducing the a new format based on comments? Like # ALLOW <host_pattern>
Or even # ALLOW <host> and # ALLOW_PATTERN <host_pattern>?

Side note

Yes, it should be false by default in the layout...

https://github.com/AdAway/AdAway/blob/master/app/src/main/res/layout/source_edit_activity.xml#L98

[zgfg]

• Blocked and Redirected domains go (directly) to the hosts file, hence it's logical (and it was implemented that way since ever) that you enter them in the hosts file format:
  (ip-addry) (domain)

However, Allowed domains do not go into the hosts fiole - it's just a note for AdAway that while parsing the hosts sources, to ignore lines with the given domains

Hence it's format can be whatever (even a json like) but only (domain) is needed, all other tags, comments, etc, are obsolete

• If you enter blocked domains through UI, you use their hosts file format

When you have several of them, instead of adding one by one, you copy exact the same lines as used for manual input to your blocked.txt file, making your local input for the Blocked domains

Similarly, if you enter allowed domians through UI, you just use the simple (domain) format (IMO, it was also implemented that way since long ago in the app)

Hence it's logical for me that when you create your allowed.txt file, that you then just similarly copy those same simple domain only lines (as used for UI) into the file

Btw, consider a user who will not read a f... manual - how whould (s)he know that for allowed.txt file he must use nopw some tags like #Allowed or so.

It would be just wasting of time and energy, and possibly prone to errors if he enters the line as #Alowed (single l insteaad of double) or so

• Btw, DNS66 also supports both local Blocked and Allowed lists files

And for blocked.txt it uses the hosts file format (as above, as AdaWay does), while for allowed.txt it uses the same simple domain only format as suggested above

[PerfectSlayer]

Hence it's format can be whatever (even a json like) but only (domain) is needed, all other tags, comments, etc, are obsolete

Yes, it could be whatever we want. But mixing format could introduce complexity.

Btw, consider a user who will not read a f... manual - how whould (s)he know that for allowed.txt file he must use nopw some tags like #Allowed or so.

It will never guess there is a new format...

Similarly, if you enter allowed domians through UI, you just use the simple (domain) format (IMO, it was also implemented that way since long ago in the app)

Originally, it was implemented using allow instead of an ip address if the host file (ex allow *.google.com as hosts file line).

For me, the file format is more for list maintainers and they are more accustomed to hosts file format.
Moreover, if we can agree on custom comment tag, we can bring new features later where a simple text file could not allow new type of feature.

[PerfectSlayer]

_Cross post from XDA_

_(DiamondJohn)_ I agree with your initial proposal above (i.e. no new format, if it aint broke...). As for the example of two keywords for "ALLOW" and "ALLOW_PATTERN", from the users perspective, in what case would you not just process all "ALLOW" lines as "ALLOW_PATTERN"? ie why not use a pattern in the "ALLOW" keyword lines? I am assuming that a "*" is an invalid host name character.[/QUOTE]

Because there is several types of patterns. The one we use is (a poor) glob pattern, with * and ?. But there is also regexp patterns, where * and ? are not wildcards but quantifiers...
So the keywords could be ALLOW followed by a plain host name, ALLOW_GLOB_PATTERN followed by a glob pattern, ALLOW_REGEXP_PATTERN followed by a regular expression pattern, etc...

_(DiamondJohn)_ Arguing the other side, it would allow for future expansion that is just not evident to me today, but could be indispensable in the future. i.e. It would open up for un-foreseen future functionality.[/QUOTE]

We could think about device or priority hints for example to express if an hosts should be applied for mobile or desktop, recommended or mandatory if you want to keep your list small, etc...

_(DiamondJohn)_ The other thing that I am not clear about (although I feel I am also asking the obvious), your above example appears to be mixing a block-list and an allow-list in the same file? I personally would prefer to keep them separate, but I understand, there would of course be nothing stopping a user from doing that anyway. Too many choices![/QUOTE]
If block-list and allow-list are the same format, nothing prevent them to contains mixed blocked and allowed host at the same time.

There is now a settings per hosts file to apply allowed and/or redirected hosts or not. So you don't have to worry about using a hosts source containing allowed hosts you don't want to unblock.
You can enable this settings on your allow-list only and it should work as expected for you ;)

I wish by evolving the hosts file format, we can improve the overall behavior of other ad-blockers too.
I will try to ping the other hosts maintainers to ask them for advice.

[PerfectSlayer]

Hello @jawz101 and @StevenBlack

Can I have your thoughts on the whole subject?
Any idea or contribution is appreciated 🙇

[jawz101]

list A

127.0.0.1 example1.com
127.0.0.1 example2.com
127.0.0.1 example3.com

list B

0.0.0.0 example3.com
127.0.0.1 example4.com
0.0.0.0 example5.com

merged step 1 =A+B

127.0.0.1 example1.com
127.0.0.1 example2.com
127.0.0.1 example3.com
127.0.0.1 example4.com
127.0.0.1 example5.com

list C (a checkbox that says this is a whitelist)

example2.com
example3.com

merged step 2 =A+B-C

127.0.0.1 example1.com
127.0.0.1 example4.com
127.0.0.1 example5.com

I wouldn't support wildcards, json, or any other junk. Host files are just a way to tell your DNS resolver to resolve a DNS request to an IP. Anything fancier and you are just writing your own DNS resolver thingy.

My other stance: Just use better lists. If someone wants to allow something, they are using a bad blocklist.

[PerfectSlayer]

Thanks for the feedback @jawz101 !

So your stance and the @StevenBlack one is:

• Use hosts file for usual configuration and don't touch its format.
• If you want to allow domains, use a dedicated domain list.

That implies:

• Create an other type of source: _the allow list type_.
• Remove the _apply allowed hosts_ from the hosts source settings.

How do you expect the user to create / choose an allow list?
Here is the current process:

1. Click _hosts sources_,
2. Click _Add_,
3. Select type: _URL_ or _FILE_
4. Select the desired options
5. Validate

Bonus question:
Is parsing a new type of file, the allow list, is the opportunity to add uBlock list format? (See #1001)

[jawz101]

I'm confused.

If you do you want to make something more complex than an app that merges host files, anything else is completely different blocking technologies.

Personally, I think what the app does is fine.

[PerfectSlayer]

The question is quite simple: Does a user can import / use an online list of allowed hosts?
Or it is just something he has to input manually on his device?

I don't really understand form where the _list C_ comes from in your comment 😅

[jawz101]

As I understand, the original request was to have an ability to designate some imported lists a exclusionary- in addition to the current ability to manually whitelist individual hosts.

When adding lists, have a toggle to indicate if it is to be treated as an exclusionary/whitelist.

From what @zgfg reported it sounded like they wanted to be able to import allow rules. I suggest that to do so it could only be done if each imported list is designated for either inclusion or exclusion.

List A and B represent include. C is exclude

I keep rewriting the same thing over and over so I'm out of comments. Sorry

[PerfectSlayer]

Don't worry! I just need some time to find the right way to do it.
Plain text host list is obviously the simple and effective way.
I'm just wondering if it could be too simple to block later improvements...

I just take a week off so I hope I will have a clearer mind now 😄

Because I also have in mind a _coming soon_ feature I would like to provide: a "per app list".
It will be lists dedicated to a specific app in order to block its ads/trackers without breaking it.
But I guess it will only contains hosts to allow, never hosts to block...

I need to reach the exodus privacy team first to see if we can do something together first.

[zgfg]

Couldn't that be solved the simolest way as originally suggested in the first post - it was opened a debate, resulting with no conclusion and feature was left not working