Ublock: adblockplus support

[KonoromiHimaries]

add support to other adblocks

https://github.com/azet12/PopupBlocker/issues/222#issuecomment-350392884

https://github.com/AdguardTeam/AdguardBrowserExtension/issues/913

+ reduct the filter lists
+ no conflicts with other adblock lists
+ most developers has be support the ublock

example.

before

spidersweb.pl##.mfp-ready
spidersweb.pl##html:style(overflow: visible!important; margin-right: 0px!important)

after

!!!spidersweb.pl##.mfp-ready
!!!spidersweb.pl##html:style(overflow: visible!important; margin-right: 0px!important)

effect

this list not be longer needed

common filters it's awesome : )

[gorhill]

I think it's a good idea, this would make filter list maintainer's life easier.

[jspenguin2017]

That's a good idea. My only concern is that some people might use !!! as header, like:

!!!example.com
||example.com/ads
||example.com/tracking
!!!other.example.com
||other.example.com/abc

I propose a "preprocessor" style syntax, like:

!#if ubo || adguard_ext
!spidersweb.pl##.mfp-ready
!spidersweb.pl##html:style(overflow: visible!important; margin-right: 0px!important)
!#endif
!#if adguard_app
!!The standalone app (2 ! for comments)
!||example.com/abc$replace=/DOCTYPE/doctype/
!#endif

[mikhaelkh]

The idea is good, but it would be a lot better if uBO and other adblockers would agree to some syntax so you wouldn't have to comment filters to hide it from adblockers that don't support it.
@jspenguin2017 Without syntax highlight it would be easy to misinterpret, one-liners are better:

!#if ubo, adguard_ext: spidersweb.pl##.mfp-ready
!#ifnot abp: spidersweb.pl##html:style(overflow: visible!important; margin-right: 0px!important)
!The standalone app (comment)
!#if adguard_app: ||example.com/abc$replace=/DOCTYPE/doctype/

[jspenguin2017]

@mikhaelkh I got a syntax highlighter, if that's your only worry, then you don't need to worry about that anymore:

[gorhill]

it would be a lot better if uBO and other adblockers would agree to some syntax

No problem with this, I do whatever is beneficial to uBO's users, hence why uBO readily accept hosts files, filter lists from Adguard, and special syntax extensions from ABP.

I meant to notify @ameshkov about how this could best be done to have a shared approach for uBO and Adguard. Regarding ABP, forget about this, they have shown little interest to care about what is done outside regarding filter syntax.

[gorhill]

@jspenguin2017 I am quite impressed by the speed of that syntax highlighter. I clicked on EasyList expecting a noticeable delay and the page appeared immediately all highlighted. Why not have submitted a pull request?

[ameshkov]

Hey guys, I'm glad to see you all:)

This is a nice feature request and that's basically what we tried to achieve with our hints syntax:
https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters#platform-and-not_platform-hints

[ameshkov]

Key points:

1. The proposed syntax should not break ABP so it'd better if it starts with an ! at least until ABP supports it (so, it'll be there for quite some time:)).

   Frankly, I think we'd better reach out to them, I believe they'll understand that the common approach is better for everybody.

2. On the other hand, it should be easy to distinguish these "conditional" rules from regular comments, otherwise, it'd be problematic to implement syntax highlighting.

3. I fully agree with @jspenguin2017's idea of a pre-processor approach. Every blocker knows its own name/identifier and able to distinguish what rules are for it.

Just an idea, but what if we extend it further and add #include pre-processor directive? In this case we can avoid messing with "commented" rules and filters will look cleaner:

!# if ubo_base
!# include https://example.org/ubo_supplement.txt
!# elsif ubo_edge
!# include https://example.org/ubo_for_ms_edge_unbreak.txt
!# elsif adguard_base
!# include https://example.org/adguard_supplement.txt
!# else
!# :’-(
!# endif

[gorhill]

I thought that PLATFORM syntax was to pre-compile the filter lists before publishing them -- so I got this wrong.

I have no problem supporting this syntax -- this will just help uBO discard what is currently not meant to be enforced on a specific platform.

But that does not solve the key issue, which is that Adguard- and uBO-specific filters break ABP when the filter list is fed to ABP, forcing filter list maintainers to create two distinct filter lists. The problem is not us being inflexible, it's ABP being inflexible -- hence why we would need to prefix everything not intended for ABP with !.

[ameshkov]

I thought that PLATFORM syntax was to pre-compile the filter lists before publishing them -- so I got this wrong.

You got it right, it's just the idea is similar. The discussed pre-processor is actually better as it does not depend on a server-side.

[gorhill]

Just an idea, but what if we extend it further and add #include pre-processor directive? In this case we can avoid messing with "commented" rules and filters will look cleaner:

This deserve consideration -- seems to be the cleanest approach to me. We would need to ensure that the component filter lists are all loaded from the same origin as the parent list, maybe even enforcing it through syntax:

!# if ubo_base
!# include ubo_supplement.txt
!# elsif ubo_edge
!# include ubo_for_ms_edge_unbreak.txt
!# elsif adguard_base
!# include adguard_supplement.txt
!# else
!# :’-(
!# endif

I really like this proposition, this way we can move forward without any change from ABP.

[ameshkov]

We would need to ensure that the component filter lists are all loaded from the same origin as the parent list.

Thinking security? IMO, if somebody has access to the filter list, he or she most likely has access to the whole repo and can place the malicious file there and then include it.

[gorhill]

Well filter list maintainers could end up linking to 3rd-party lists for which they have no control. In my opinion it's best that what is loaded only come from the source which the user tacitly trusted.

[ameshkov]

Yup, it makes sense indeed. I guess relative paths should be also supported then. Also, failing to load an inclusion should lead to an update error for the whole filter.

[gorhill]

Also, failing to load an inclusion should lead to an update error for the whole filter.

Yes, agreed.

So for me that settles it, @ameshkov's proposition of using pre-processor directives is the best approach to solve issue raised by OP.

[ameshkov]

Yeah, it sounds like a good idea, but I'd like to sleep on it. Maybe I'll think up something useful by tomorrow.

What do filters maintainers think about it? @mat41997?

[gorhill]

To be clear, a filter list maintainer would merely need to add the following in a base filter list, using case in opening comment:

!# if ext_ubo
!# include ubo_supplement.txt
!# endif

[jspenguin2017]

I really like the # include idea! But in that case, wouldn't it be better to make it one-liner? Like

!# if ext_ubo : include "supplement.txt"

---

Why not have submitted a pull request?

I had the highlighter for well over half a year now and no one said anything about it. My other pull requests and proposals are all ignored, without a fork (Nano, as you might already know) I can't create momentum and my proposals would stay just proposals.

Although it is fast, it does use 3 times more RAM, also, it uses a huge library (over 20k lines) and I thought you don't like libraries. It still has some bugs that I need to clean up, but you are welcomed to import changes from my fork.

[gorhill]

But in that case, wouldn't it be better to make it one-liner?

Fine with me, this make the implementation even simpler for now. We can always become more fancy in the future.

My other pull requests and proposals are all ignored

If you mean the vvvvid.it one, it's because I do worry a lot about growth of resources.txt, and in general I prefer to take my time for large scriptlet useful only on one site. For now there is no way to be selective like filter lists when it comes to loading inline resources. It looks like it's something I have to consider for the future.

For ACE, I did look at it and it's definitely large, but: its purpose is confined to specific panes in the dashboard, so it's loaded only in these cases and all that extra code is removed from memory once these dashboard panes or other auxiliary pages are navigated away. So it's a case where the extra memory seems to be worth it given that it does not affect the core process -- i.e. uBO will still be lean when the dashboard or other auxiliary pages are not opened, or after they are closed.

[jspenguin2017]

I don't see how can that be a problem, modern devices have ton of RAM, the entire Nano Defender is not even 1MB. Beside, you can compress the scriptlets to make them smaller, can easily save 25% just from that.

Something that is heavy is better than something that does not work. Have you counted how many users actually care how much RAM uBO uses? Only a vocal minority talks about that.
I'm a big fan of caching everything in RAM for fast access, CPU performance is way more important than RAM usage. Computer manufacturers have stopped investing in next generation RAM design because 16GB is enough. Why are you so obsessed with RAM usage of uBO's background page?

Like always, I can't convince you until I have shown the potential of my proposals, and that is why Nano exists.
I have proposed a constant definer for months, and you refuse to add that until Nano got it, is that a coincidence?

[ameshkov]

But in that case, wouldn't it be better to make it one-liner?

As I see it:

1. This may hurt us if we ever decide to extend the pre-processors syntax.
2. It will reduce opportunities to including files only. One day when everybody support pre-processor directives, maintainers should be able to write something like that:

!# if abp
@@||example.org^document
!# endif
!# if ubo || some_other_blocker
||example.org^document,important
!# endif

@gorhill btw, I think we should support conditions there (at least "or" conditions).

[gorhill]

This may hurt us if we ever decide to extend the pre-processors syntax.

Ok agreed. So if someone wanted a quick implementation to just solve the case at hand here, this also would work, !# if ... is not mandatory, right?

!# include supplement.txt

[jspenguin2017]

Need to comment out all the rules in between non-abp adblockers:

!# if ubo || some_other_blocker
!||example.org^document,important
!# endif

Also how would mis-matched pre-processor be handled? Filters are all "standalone" lines, but now we are introducing lines that affect other lines. One bad line breaks the rest of the filter isn't really good IMO. I'm thinking of inserting !# endif automatically if a non-commented line is hit.

[gorhill]

Need to comment out all the rules in between non-abp adblockers:

The !# include directive will be used for a list which is incompatible with mainstream blocker. The !# if directive is useful for sections of a list which are meant for filters for a specific purpose, say !# if platform_mobile or !# if browser_blink -- the filters do no breaking harm if they are loaded (hence no need to prefix with ! -- but a blocker can optimize itself by applying the hint. Unmatched !# if should simply apply through end of file.

[ameshkov]

!# if ... is not mandatory, right?

@gorhill yep, I suppose directives should be independent.

One more thing, I think we should also handle pre-processor directives inside of an included file.

Check the example below (plz read the rules comments).

main_filter.txt

...
!#if adguard_base || ubo_base
!#include extended_rules_supplement.txt
!#endif
...

extended_rules_supplement.txt

...
!
! This is just an example
!
!#if adguard_ext_firefox || ubo_firefox
! Only FF can block "blobs" via webRequest listener
|blob:$domain=example.org
!#elsif adguard_ext_chromium || ubo_chromium
! Can't block blobs, blocking webrtc
*$webrtc,domain=example.org
!#else
! Circumventing detection in other browsers
@@||example.org^
!#endif
...

Also, I suggest defining more than one variable.

In the case of uBO it might be:

• ubo_base -- defined in all uBO versions
• ubo_chromium, ubo_edge, ubo_firefox -- depending on the browser
• Maybe you should consider defining ubo_mobile and ubo_desktop to distinguish FF Mobile from others.

[ameshkov]

Unmatched !# if should simply apply through end of file.

This is VERY dangerous. I suggest failing the whole filter update in this case.

[gorhill]

This is VERY dangerous

Why "VERY"? Filter lists are currently wholly read without leaving anything out. Why would that suddenly be such a serious issue?

[jspenguin2017]

Hold on, if rules between an if-case are not commented, how does this help OP's case? Like using extended syntax without having to carry another filter file.

[ameshkov]

Why "VERY"? Filter lists are currently wholly read without leaving anything out. Why would that suddenly be such a serious issue?

In this example, you'll load 1 rule instead of 10k.

#if ubo_base
||example.org
#else
some other rule
! typo in endif
#enduf
Here go 10k other rules

[ameshkov]

Hold on, if rules between an if-case are not commented, how does this help OP's case? Like using extended syntax without having to carry another filter file.

The point is that the !#include directive is commented and maintainers can keep all extended rules in a separate file.

[gorhill]

In this example, you'll load 1 rule instead of 10k.

Sure, but nothing dangerous, just un-optimal at worst. I see the !# if as hints for blockers to optimize themselves, _if they care_.

Edit: ok sorry, I see what you mean. But at some points filter list maintainers are responsible that their list is sound. Forbidding any comment in between !# if-!# endif directives doesn't seem better to me -- this could also create good filters to not be loaded with an erroneous stray comment. Let's not try to second guess filter list maintainers, that's impossible to get it right.

[KonoromiHimaries]

Forbidding any comment in between !# if-!# endif directives doesn't seem better to me

@gorhill i think you can create a new special syntex to any comment in between !# if-!# endif directives.

this could also create good filters to not be loaded with an erroneous stray comment.

this no make a sens if you use that.

example.

!#if adguard_2.7.2_~_firefox_quantum_57 || ubo_1.14.20_~_firefox_quantum_57
!& comment here
normal text or rule without special syntex here to be prove that it is possible
normal text or rule without special syntex here to be prove that it is possible
!& comment here
!& comment here
!# rule here
!# rule here
!# rule here
!#include https://raw.githubusercontent.com/easylist/easylist/master/easytest.txt
!#endif

that's impossible to get it right.

seriously?

[ameshkov]

Sure, but nothing dangerous, just un-optimal at worst. I see the !# if as hints for blockers to optimize themselves, if they care.

1. Typos can happen even if they care
2. Such a mistake may render filter unusable for quite some time
3. This will frustrate casual users and you're the one who they will blame for it as they don't know or care about actual filters maintainers.

[ameshkov]

Forbidding any comment in between !# if-!# endif directives doesn't seem better to me

I am talking about unbalanced #if #endif directives only, there's no need to forbid anything.

[jspenguin2017]

Yea, that's what I meant, if the matching !# endif could not be found, assume it is just before the next non-commented line.
Although this can still cause problems with misspelled directive, it is less likely to swallow the rest of the filter, could potentially break a couple lines, but not the rest of it.

Is there any process for adding new language to GitHub? I think that'll help a lot.

[gorhill]

Ok I get it, do not ignore filters inside non-well formed pre-processor blocks.

[jspenguin2017]

I wouldn't be too worried about 4 or 5 lines inside broken blocks, as you said, you can't get that right.

[ameshkov]

Ok I get it, do not ignore filters inside non-well formed pre-processor blocks.

I guess we can keep it implementation-specific :)

I'd better have the update failed, it'll help maintainers to figure what's wrong and avoid a confusing situation when blocker applies a rule that shouldn't be applied.

[ameshkov]

Is there any process for adding new language to GitHub? I think that'll help a lot.

Talking about syntax highlighting? People often use some editors like Sublime or VS code. They both support custom syntax highlighting via TextMate themes.

[jspenguin2017]

I'd better have the update failed, it'll help maintainers to figure what's wrong and avoid a confusing situation when blocker applies a rule that shouldn't be applied.

I disagree with this, it's not an adblocker's job to tell maintainer their filters are broken, it's like it's not a browser's job to tell developers their HTML is invalid. When you can have a good guess what it should be, go guess it, it will cause less interruption for users.
It is the job for a linter to fail the build when the syntax is invalid.

[ameshkov]

it's like it's not a browser's job to tell developers their HTML is invalid

In my understanding, it's closer to a javascript function with invalid syntax, and browser prevents the whole script from executing.

[jspenguin2017]

The problem is most people consider uBO and Adguard to be error-tolerant, suddenly tossing out entire filter lists like AdBlock and ABP isn't exactly right, but well, your project your decision.

JavaScript is kind of different, as one character can entirely alter the behavior of a script, there is no reasonable way for browsers to guess it. HTML (and filters) are different, even with broken parts, the rest can run just fine.

[gorhill]

Fixed with 6a8c27b6dfb8.

[BilBg]

!#include ../path2/includedfile.txt

@ameshkov @gorhill - please see if include "relative path" can be used from non-own place - add this my list for test:
https://raw.githubusercontent.com/BilBg/BilBg_Test/master/Test_include_relative_path.txt

[BilBg]

For the wanted by some inline (in-the-main-list, just for a few lines) additions of advanced syntax I suggest the prefix !#! which is easy enough to type (and space after it).

! for any "advanced" blocker   !#!
!#! ||example.org^$third-party

! for uBO only   !#!ubo   !#!u
!#!ubo ||google-analytics.com/ga.js$script,redirect=google-analytics.com/ga.js,important
!#!u   ||google-analytics.com/ga.js$script,redirect=google-analytics.com/ga.js,important

! for AdGuard only   !#!adg   !#!g
!#!adg ||example.com^$third-party
!#!g   ||example.com^$third-party

[gorhill]

@BilBg I increased the restrictions as to what can be pulled as a sublist:

• If a sublist contains the substring .., it will be rejected.
• If a sublist is a full URL, it will be rejected.

This is to enforce the following rule: only resources from with the same directory can be pulled as sublists. This still solves the primary issue here, which is what matters.

I think it's bad that one list could pull two sublists which sit in different GitHub repos, despite the same-origin (https://github.com/) not being violated.

[BilBg]

@gorhill - What will happen if:

!#include   /easylist/easylist/master/easylist/easylist_whitelist.txt

Will this start from the current directory or from the root? (https://raw.githubusercontent.com/)

[gorhill]

From the current directory.