Ungoogled-chromium: The future of content filtering (declarativeNetRequest, Manifest v3, and beyond)

[joey04]

This is both an FYI and some follow-on questions.

The FYI -- For those who haven't yet heard, Google is planning a severe reduction in extension capability for blockers and other content filters. Their current draft of changes includes deprecating the webRequest API and replacing it with declarativeNetRequest. This will only permit a fixed number of purely declaritive rules in a single json list. Moreover, the browser will have final say on what requests get blocked or redirected.

For more info, see this uBlock Origin thread. It has all of the relevant links.

I have two follow-on questions:

1. Is there any interest from the developers here to integrate a filtering engine? I realize this is a big request, but the new API will eliminate the possibility of uBlock Origin, uMatrix, and other robust extensions. Since this project recommends those on the wiki, it will impact this project and its users. My idea is to modify Brave's engine, which is already integrated but has only AdBlock Plus level functionality.

2. If not, can you recommend how to undertake such an effort? I have experience working with large C++ codebases, but nothing browser-related. Any tips, gotchas, etc. would be appreciated.

[Eloston]

@virtadpt asked something similar on Gitter:

Given that Chrome and seemingly Chromium-mainline are going to lose the use of the webRequest API, which is going to break many ad-blocking extensions (https://www.howtogeek.com/fyi/chrome-may-get-faster-ad-blocking-while-breaking-ublock-origin/), are there plans to keep it in Ungoogled Chromium? Or would this be technically infeasible due to ongoing maintenance concerns?

My response:

Google has stated that their draft is subject to change, so I won't plan anything unless it is certain to become a reality.

That being said, I have nothing against such patch as long as it's implemented well. If the patch becomes too difficult to maintain, we could look into other options like Brave's native blocking implementation. We will discuss this on GitHub Issues if it ever gets to that point.

[joey04]

I advise planning for this sooner rather than later. I have little hope that Google will make any serious concessions on this matter. (Folks can read the thread I linked to, the threads on Reddit, etc. to understand why.)

I have several points to cover in this post, but first a little about me since I'm new here.

I'm a Firefox guy who has never built a browser from source. I've been using Chrome on Windows and Chromium on Linux for a few years as a supplement, but I'm much more familiar with the Mozilla way of doing things. As for content filtering, I've twice forked uBlock Origin (uBO), first for XUL (aka Firefox legacy) and now for WebExtensions. It's been a while since I've used regular uBO :)

I obviously know a lot more about filtering than modifying Chromium. But I've now spent several hours looking into code possibilities. From the readme here I found Bromite with its integrated blocker inherited from the defunct NoChromo code. Csagan5 has done some nice refactoring work, including moving the code from the net component to the chrome component. However, he still has the very simplistic hard-coded header file of 50K+ rules, apparently translated via script from EasyList. (Upon seeing this, I couldn't help but laugh at the irony of Google going this direction with extensions being limited to a single json file of rules.)

On the other hand, Brave has a full-fledged filtering engine (though not robust enough for me) but with a heavyweight integration into Chromium. (Understandably, given their grand ambitions of an entirely new online advertising model, along with other features they've added.)

For me, assuming my big learning curve and setup of a build environment goes well, I'll probably go with a more Bromite way of doing things to start. Eventually I'd incorporate more robust capabilities, possibly borrowing from the Brave code. Lots of details TBD, of course, but I mention it as food for thought here.

---

For those reading this and fretting about their browsing future, I too have some anxiety about the end of robust extensions. (Not just in Chrome; I would expect Mozilla to follow suit eventually, which is another reason I'm here.) It'll be at least a year before the current extension API is phased out of Chromium, so there's plenty of time to prepare. While there won't be any options as desirable as the robust extensions we're accustomed to, we won't be restricted to just a Google or Mozilla browser with crappy filter extensions.

I advise not putting all your eggs in one basket. A custom Chromium appears to be the best choice to transition to for my primary browser, given its best-possible web compatibility. But another possibility is Firefox forks. I used Pale Moon for several years and can attest to the quality work of Mark Straver and his helpers. He has a clear vision of what he wants his browser to be and has done a good job making it happen. There are inherent drawbacks, unfortunately, like web compatibility, but it's still worth considering, especially since it will always maintain robust XUL extension support. (It'll be easy for me to revive my original uBO fork.)

I expect I'll continue to have a multi-browser setup. It's TBD what will be primary (for the lion share of browsing), but that's what I intend to figure out in the coming months. Worst-case, I can use regular Chromium with a very limited v3 extension as a supplemental browser starting next year. Like I said before, there are a number of options worth considering.

[mikhoul]

I'll probably go with a more Bromite way of doing things to start. Eventually I'd incorporate more robust capabilities, possibly borrowing from the Brave code. Lots of details TBD, of course, but I mention it as food for thought here.

Where I could follow your progress on a new Chromium Fork ?

Like you know from the other Nano thread I'm really interested by a new Chromium fork that would stay retro-compatible with V2 Manifest but that would also be compatible with the future V3 manifest.

Regards :octocat:

[joey04]

Where I could follow your progress on a new Chromium Fork ?

Nowhere. This is a preliminary exploratory time. I can't guarantee I will do anything on Chromium. (The way I wrote it yesterday didn't properly convey that. Sorry about that.)

As I wrote yesterday, I've never built a browser. Chromium is a massive codebase with millions of lines of code. And one very important item I didn't mention before is that I would be purchasing a new machine to do this work.

My current PC is a Windows 8.1 with hyper-threaded dual-core and 8 GB of RAM. So it's capable of building Chromium, but doing so would abuse the hell out of my hardware. I really like my highly-customized Windows setup, in large part because 8.1 is the last version I can control. In fact, I've been intending for several years to transition to a Linux distro for my next PC because I hate Windows 10. But I'm in no hurry to do this and would prefer to continue using my current PC indefinitely. Running it hot for several hours (at least) for an initial build of a Chromium pull is the last thing I'll be doing to it.

At this point, undertaking a very time-consuming Chromium modification project is literally a thousand times more involved for me compared to simply installing and configuring one of Straver's browsers (Pale Moon or Basilisk) as a backup option to my current primary browser of Firefox ESR. (I probably won't do that just yet, because I'm content with my ESR setup, but it wouldn't take me more than a few hours to have a Basilisk with Classic Theme Restorer configured to my liking.)

I mention all of this for context. I'm still interested in modifying Chromium because it will have the best web compatibility in the long run. But, practically-speaking, so long as Firefox or Straver's forks are a viable primary browser for me, I would prefer to stick with them. There are a number of things I prefer about the Firefox way, including privacy settings and the quality of Gecko rendering, compared to Chromium.

Finally, I'll add that while I suspect Mozilla will eventually conform to Google's V3 API at some point, that wouldn't be for at least another year. So there's no urgency for me here.

[joey04]

There's one more point I'd like to make. Please don't think I'm a Mozilla advocate here. That's not my intention.

In fact, I don't like the Mozilla organization. Nor do I trust them to make good or even ethical decisions. But, as I've stated, its ESR browser is my best choice as primary right now.

I must state that my ESR setup is highly customized to the point that the browser does not emit a single packet unless I expressly want it to. (I've verified it with Wireshark.) This took a number of hours to do, using the thorough documentation of the ghacks-user project to disable a bunch of settings. I honestly consider it an "UnMozillad Firefox" :)

It helps a lot that all of my extensions are either my own creation or modified by me. So they're all unsigned, of course, and I had to fully disable add-on updates to not have the browser ping Mozilla's add-on servers.

One advantage of Straver's forks is that I got this same level of control much easier. He doesn't do any shady crap like Mozilla. Seriously, default Firefox settings result in almost Chrome level of phoning home.

(No way I can completely UnGoogle my Chrome install on Windows, of course, but I have done so to the extents possible in settings. As a supplemental browser, I only use Chrome for a few websites, so I'm okay with it.)

[joey04]

It just occurred to me, perhaps there's a way to do Chromium development without need of my own build machine.

Ideally, I could just pull the code here on GitHub and run a Travis build. But I don't know much about the particulars.

Question: Does anyone know if this is feasible? Or is there another online service I could use?

[Eloston]

Thanks for describing in depth your problem and motivations. I believe I understand the situation better now.

We would all appreciate new feature development, even if it isn't immediately useful, or even if it doesn't end up useful at all. While I can't guarentee that I will personally contribute code, I would at least like to provide guidance and support for you and this feature.

It looks like you have a vision of what you want this feature to become, so I won't push back with feature requirements or the sort at this time. I think this'll be a great way to let you familiarize yourself with the Chromium codebase, and allow us to form better ideas based on actual work.

You said you've worked on large codebases before, so here are some brief tips I can give (in no particular order of importance):

• Take a look through Google's design docs in chromium.org and in the Chromium source tree under docs/. At least skim them to get an idea of what's there and to set your bearings straight while in the code.
• Have a look through ungoogled-chromium's patches to get an idea of how certain components of Chromium work.
• Try reading through the code mentioned by the design docs or patches. Especially code comments, like those in the header files. Reading helped me get an intuitive "feel" for Chromium's code style and structure.
• There are several major differences between ungoogled-chromium's build process and Chromium's build process. There are several reasons for doing so, but they should make more sense as you go through them.

I realize that working on Chromium, or any new large codebase, is a daunting task. But like any other large codebase, no one person will know how all of Chromium works, or even what all the individual components are. Thankfully, there is a nice coherent structure and style throughout that makes reading Chromium code a lot easier, even satisfying at times. Reading the design docs covering overall motivations, structures, and processes helped quite a bit too (albeit it is usually a bit outdated). While the amount of code you will need to look through will not be insignificant, it shouldn't be too overwhelming either.

To keep track of your work, I think it is best to start a fork; perhaps we could maintain a Pull Request here to increase visibility.

Finally, to wrap up with a question you had:

Question: Does anyone know if this is feasible? Or is there another online service I could use?

Have a look through #17. Basically, Travis won't work. You could try OpenSUSE Build Service or maybe something like an Amazon EC2 instance.

[joey04]

@Eloston thanks for your thoughtful reply. I appreciate your advice. In fact, I suggest adding it to your developing doc.

I read several of the Chromium docs last week when I first looked into this. I was pleasantly surprised how helpful they are. It was a good first impression.

But, all things considered, I probably won't be interested in using a modified Chromium until uBO becomes unviable in Chrome. I'm guessing that will be in the next 18 months, depending on Google's TBA timeframe for phasing out webRequest.

This doesn't mean I'm not interested in exploring the integration of a custom filter engine. I'm actually going to continue looking into that in the near future. I already have very clear notions of what a custom filter engine should be; for me, that's the easy part because I've tailored my current uBO fork to be just that. Of course there are other essential considerations, which is what I'll be focusing on as I look at relevant portions of the code. But without a build machine of my own or a pressing need to make these changes, it'll just be a learning exercise for now.

[gmt]

FYI the chromium build requirements are just crazy. If you want to hack on the chromium source, you really need a beefy machine with a lot of ram, the more cores the better. On my old 8-core-8-thread-32G system it takes about 12 hours to build, and I have to turn off debug symbols or else I run out of ram. On my new 16-core-32-thread-64G system, it takes two or three hours without symbols and three or four with debug symbols.

On the relatively rare occasions I do chromium-related development I usually am worried about packaging for Gentoo, not chromium itself, per-se. Which means, my workflow is quite different from the standard one. But the standard workflow, which I do also have limited experience with, is considerably more, not less, resource intensive.

My point is, you are going to need some horse-power, probably more than you'd like to pay Amazon for, unless you're made of money. You might look into if the gcc compute cluster, a free resource hosted by FSF France for open-source projects needing access to a bunch of heterogeneous computer systems, could help you.

There are some bureaucratic hurdles to getting access (human review is involved) but they deliver a fair amount of compute for free. If you go this route, please mind your resource consumption; those are shared compute resources like in the bad old days, and there is an assumption that folks will not gobble up all the resources on those machines, so you need to play nice.

For the record, I landed here because I share your concerns about this manifest v3 business, and have come to the same (or worse) conclusions as you about the likely outcomes. In my (hopefully paranoid) imagination, their incentive is to create as much "counter-fud," to coin a phrase, as possible, essentially stringing everyone along with vague hopes that the inconceivable will not happen until it's too late, and
coordinated code-drops and a simultaneous web-store revamp drop this on the unsuspecting public like a bomb.

Hopefully I'm wrong. I definitely have no insider knowledge and only a very cursory understanding of the technical stuff involved. But I'm not optimistic at all, personally -- it seems to me they have decided that, with the public increasingly concerned about privacy, now is the time to monetize their investment in web-client software, leveraging it as a means to limit the threat of the privacy arms race to their future revenue.

[Eloston]

FYI the chromium build requirements are just crazy. If you want to hack on the chromium source, you really need a beefy machine with a lot of ram, the more cores the better. On my old 8-core-8-thread-32G system it takes about 12 hours to build, and I have to turn off debug symbols or else I run out of ram. On my new 16-core-32-thread-64G system, it takes two or three hours without symbols and three or four with debug symbols.

I'm not sure what your setup is like, but that sounds excessive. I have been using a laptop for all my recent ungoogled-chromium builds; a Skylake i5 (dual core, hyper-threaded) with 16 GB of RAM. I can put the entire build tree on tmpfs (no swap), and it compiles in roughly 3-6 hours (time varies due to the particular build configuration and Chromium version). For my Debian builds, the entire build tree takes somewhere between 8 GB to 10 GB, and the compiler and linker are just fine with the remaining amount of RAM (though, I do have to log out of GNOME 3 to prevent an OOM issue during linking).

EDIT: I should note I usually build in release mode, which is why I'm able to fit the build tree into tmpfs.

[gmt]

My old workstation is pretty darn slow (bulldozer) and largely built with -Og -ggdb3 which leads to high build-time memory demands. Also maybe I'm exxagerating the numbers a little bit... definitely in that ballpark but I never measured scientifically.

Btw I've noticed for my chromium builds ccache always got huge "hit" percentages compared to other projects I've tried it with, when I activate it. This can be a lifesaver under the right circumstances (or lead to segfaults in the wrong ones).

[pipboy96]

There's way too much off-topic conversation here. @Eloston can you hide all these comments (mostly about hardware "needed" to build Chromium) using GitHub's new feature (like this one) and if possible make an official comment, even if it's just:

Right now, there are no concrete plans related to this issue.

[SuperRobinHood]

I've always wanted a browser with uMatrix intergrated in by default. I wonder if ungoogled-chromium could be that browser? It would probably require a change of policy from just being a simple Chromium fork with the spying stuff removed, to being its own browser. But Google and Mozilla are falling deep into the censorship rabbit hole (see the Dissenter extension removal), so we can't rely on them for long. Expect extensions to become more and more gimped as time goes on.

What do you think?

[pipboy96]

@SuperRobinHood I also think baseline stuff such as blocking and HTTPS Everywhere could be simply integrated into the browser.

[Technically-Alexander]

https://groups.google.com/a/chromium.org/forum/m/#!topic/chromium-extensions/veJy9uAwS00/discussion

I've started becoming aware of the progress of this situation and it appears as if Google has expressed no intention of lifting the 30k limit to filters, according to the active thread on the topic above.
The other changes they plan will force list updates to require whole extension updates, which, with ungoogled-chromium's divorce from extension update infrastructure, would break a significant amount of current adblock functionality and the train is full speed ahead for destroying uBlock Origin with no sign of change in the future. If anything they seem explicitly dedicated to destroying uBO the way Google staff keep making excuses for breaking its functionality.

I just want to bring this to awareness of the development team, and others. I will keep a close eye on progress of this situation and try to learn more about the project and about the browser's development as well.

Thank you.

[pipboy96]

@Eloston Need your opinion since the situation had changed.

[Technically-Alexander]

edited, wrong link
https://groups.google.com/a/chromium.org/forum/m/#!topic/chromium-extensions/WcZ42Iqon_M

Sorry posting from mobile has been giving me trouble I may send too many notifications.

"""Increased Ruleset Size: We will raise the rule limit from the draft 30K value. However, an upper limit is still necessary to ensure performance for users. Block lists have tended to be “push-only”, where new rules are added but obsolete rules are rarely, if ever, removed (external research has shown that 90% of EasyList blocking rules provided no benefit in common blocking scenarios). Having this list continue to grow unbounded is problematic."""

February 15th, 2019, Google Chromium dev posts this, specifically stating they are going ahead with most of their plans, they may implement Google approved top domains for remote requests and specific types of requests, but in this quote they specifically express intention to bring an end to EasyList and its additional filters, as they consider it "problematic"

This is Googles response to the endless stream of complaints in the original thread posted in my previous comment.

[Eloston]

In that case, we should find a solution that allows us to keep using uBlock Origin with full functionality. My question is: What kind of options do we have? Should we remove the 30k limit (is that even a good idea)? Is it feasible to adopt Firefox's webRequest API (of which has advantages mentioned here)? I'm open to suggestions.

[pipboy96]

@Eloston Maybe implement basic blocking as part of our patches? So even if uB0 is discontinued we can still use the lists.

[mikhoul]

https://twitter.com/BrendanEich/status/1134141335881912320

https://old.reddit.com/r/brave_browser/comments/buhq20/chrome_to_limit_full_ad_blocking_extensions_to/epdmuk5/

Brendan Eich is an American technologist and creator of the JavaScript programming language. He co-founded the Mozilla project, the Mozilla Foundation and now he is the CEO of Brave Software.

@Eloston I'm pretty sure it's the way to go, just base Ungoogled Chromium on Brave instead of Chromium, this way you will benefit from the work of Brave developers and focus on the things that make you différents from Brave/Chrome i.e. the things that make you unique.

Regards :octocat:

[Eloston]

@mikhoul That's a good suggestion, but I don't want to rush in until we see how they implement it. Not only that, but rebasing ungoogled-chromium based on Brave means we implicitly include all the kinds of changes that fit their goals and agends; I don't know Brave well enough to be comfortable depending on them. For now, I'd rather just cherry-pick changes from them like we've been doing with other projects.

[emanruse]

Off-topic just to clarify (perhaps worth documenting?):

I also think baseline stuff such as blocking and HTTPS Everywhere could be simply integrated into the browser.

I notice that unlike Firefox recent Chrome versions default to HTTPS when you type an URL. So in regards to that there is no need for HTTPS Everywhere at all (which itself has anti-privacy implications just like any extension which requires list updates from remote hosts however promising those hosts may pretend to be).

Also if you use uMatrix with setting "Forbid mixed content" you are protected from the non-obvious insecure HTTP requests.

To harden this even further you can use flags:

chrome://flags/#enable-mark-http-as = Enabled (mark as actively dangerous)
chrome://flags/#enforce-tls13-downgrade = Enabled
chrome://flags/#disallow-unsafe-http-downloads = Enabled

[emanruse]

@Eloston

Why don't you cooperate with @gorhill and make your browser do what uMatrix does without requiring extensions? I have always wanted a browser which would allow me to block 3rd party requests etc. and enable them selectively. If it is built-in rather than an extension, it can be very fast and optimized.

[pipboy96]

@emanruse Not quite true, there are cases when simply doing s/^http:/https:/ causes redirect to wrong/broken page.

[xEIkiFo]

@Eloston

Please keep in mind that uBlock Origin is not the only extension affected by these changes. There are many extensions which rely on the blocking and modifying abilities of the webRequest API and a lot of them will have their functionality reduced and some will not be able to exist anymore because of these changes.

The main aim here should be to preserve the webRequest API to its full extent. Expanding the API or adopting Firefox's version would be an improvement on Chrome's webRequest API but depends on whether you or someone else could implement them.

I don't think it's a good idea to base on Brave either. At least not yet.

[emanruse]

@emanruse Not quite true, there are cases when simply doing s/^http:/https:/ causes redirect to wrong/broken page.

Which part exactly is not true?
And where are you doing this replacement?

[pipboy96]

@emanruse

So in regards to that there is no need for HTTPS Everywhere at all

[emanruse]

@pipboy96 I don't know what you are talking about. In any case this is off-topic, so perhaps open a separate issue to discuss there.

[mikhoul]

The main aim here should be to preserve the webRequest API to its full extent

Amen ! Also keep extension like Userscript/Userstyle Managers to download external ressources (userscripts & userstyles).

Backgound page swill also be deprecated.

[pipboy96]

@mikhoul A side effect of background page deprecation is that <script type="module"> will no longer work.

[atg]

Before thinking about writing a native code uBO/uMatrix replacement, please read uBlock's source code. It tens of thousands of lines of JavaScript, and would be an immense undertaking in C++, months or years working full time.

uBlock Origin is already using WASM in the hot parts, so the assumed big performance benefit to using C++ might in reality be small.

In other words, the easiest and quickest solution is to concentrate on restoring the webRequest _interface_ between uBO/uMatrix and Chromium after Google severs it. Hopefully it's just a compile-time flag since Google is keeping it around for enterprises, but even they were to completely remove it, restoring the interface is going to be significantly easier than rolling your own adblocker.

[pipboy96]

@atg What do you think about implementing it in Go instead of C++?

[emanruse]

It tens of thousands of lines of JavaScript, and would be an immense undertaking in C++, months or years working full time.

Optimize: https://dilbert.com/strip/2007-09-03

:)

What do you think about implementing it in Go instead of C++?

https://developers.slashdot.org/story/19/05/24/2349236/is-go-googles-programming-language-not-ours

[pipboy96]

@emanruse Go and its standard library are released under BSD license; if you don't agree with something, you can fork, it's not like Google is going to sue you to death for doing this. If you don't like Go, you can use Rust, but it's "owned" by Mozilla. It's also licensed under MIT or Apache (the choice is yours).

[emanruse]

@pipboy96 Just sharing a recent article on the topic as it seems somewhat related to ungoogling things. Forgive me to provide an answer to a question you put (maybe you expected answer from one particular person only).

[pipboy96]

@emanruse Can you give a tl;dr of that article?

[emanruse]

@emanruse Can you give a tl;dr of that article?

By your question I understand that you replied (twice) without even reading and still refuse to read.

[pipboy96]

@emanruse I need more information to decide if I want to read this article or not. 🤦‍♂

[jneidel]

I need more information to decide if I want to read this article or not.

@pipboy96 the article is already a tl;dr of another article, if you'd opened it even once you'd know that...

And this is way off topic by now.

[pipboy96]

@jneidel I agree re. off-topic.

Re. the article, I just read it now. All my points (which I made before reading it, see https://github.com/Eloston/ungoogled-chromium/issues/662#issuecomment-500244350) still stand.

[medavox]

Concerned uBO user here, weighing in while checking my options going forward.

@atg is right, the hard part wouldn't be keeping the v2 support code intact; it would be filtering and modifying commits that remove its usage from the rest of Chromium.

Preserving an existing implementation will always be easier than rewriting it. Or porting it
(unless said existing implementation is godawful).

The biggest limiting factor to open-source fightback on issues like these is a lack of time: most developers don't have enough time to spare on volunteering in open-source, because they need money to live. Google (and every other major software company out there) know this, and leverage their superior developer-hour resources to outpace those trying to defend against their for-profit actions.

There's a reason a lot of new open-source projects are started by university students: their living costs are (often or usually) already accounted for during their time at uni (either because it's state-funded, or the Bank of Mum and Dad)

@pipboy96 I recommend learning Go and C++, then having a go at porting yourself.

[emanruse]

"Despite sharing a common Chromium codebase, browser makers like Brave, Opera, and Vivaldi don't have plans on crippling support for ad blocker extensions in their products -- as Google is currently planning on doing within Chrome."

The full article:

https://soylentnews.org/article.pl?sid=19/06/10/1113239

[xEIkiFo]

It should be relatively easy to keep the full webRequest API right now because Google has said they aren't removing them for enterprise users. Yet the only Chromium derivative who has clearly said they will keep webRequest fully is Brave. The rest of the them have not definitively said they will keep webRequest. They have only said that they don't have plans to cripple ad blocking. If you read their actual statements they are being vague on whether that means they will definitely be keeping webRequest fully. They all conveniently mention that their own built in ad blockers will still keep working even though all of them are inferior in capabilities to the popular extensions that use webRequest.

I don't think Ungoogled-Chromium / @Eloston should rely too much on other Chromium derivatives even Brave on their word that they will keep the webRequest API fully. When Google eventually removes the webRequest API completely even for enterprise users, we will then know who really means what they say.

[emanruse]

Web Request and Declarative Net Request: Explaining the impact on Extensions in Manifest V3:

https://blog.chromium.org/2019/06/web-request-and-declarative-net-request.html

[joey04]

Before thinking about writing a native code uBO/uMatrix replacement, please read uBlock's source code. It tens of thousands of lines of JavaScript, and would be an immense undertaking in C++, months or years working full time.

I've already done so, having thoroughly altered uBO for my own personal preferences. In summary, I've removed of hundreds of KB from the uBO codebase and added a few new features of my own. In fact, a significant portion of this work was only undertaken after I started this thread, in part because I needed to learn gorhill's code better to fully understand what I would have to do for a C++ equivalent.

Depending on what particular features you want for in-browser content filtering, it doesn't necessarily have to be more than a couple months of a dedicated hobby effort. For me, the GUI wouldn't be as nice as uBO's most likely, but the actual filtering would be as thorough as the current Chrome verison (i.e. no HTTP response body filtering).

I write all of this because I agree with what @xEIkiFo wrote two posts ago. I personally think it best to take the bull by the horns instead of trying to preserve the v2 API that Google will eventually fully remove. That's why I started this thread the way I did.

(But I'm in no way committing to an in-browser project of my own or helping out with someone else's. For now, I'm just giving advice on how to approach this. The big picture hasn't changed for me from what I wrote 5 months ago, and I hope Mozilla continues to support their current API for at least a few more years.)

[emanruse]

Some fresh news:

https://soylentnews.org/article.pl?sid=19/08/01/2024202

[pipboy96]

@emanruse Direct link to post by EFF:

https://www.eff.org/deeplinks/2019/07/googles-plans-chrome-extensions-wont-really-help-security

[pipboy96]

See the issue in HTTPS Everywhere's repository for statements recently made by browser vendors: https://github.com/EFForg/https-everywhere/issues/17268.

[emanruse]

Perhaps the future is to see some actual cleaned up forks of Chromium and Firefox, not just re-branded patched ones. Or hopefully someone can start working on a new browser without all the spying of Google and the "privacy" lies of Mozilla.

Some interesting starting points could be:

https://surf.suckless.org/
https://www.uzbl.org/
https://www.netsurf-browser.org/

[csagan5]

Google is planning a severe reduction in extension capability for blockers and other content filters.

I have discussed this a bit here, in particular its relationship with the introduced NetworkService.

From the readme here I found Bromite with its integrated blocker inherited from the defunct NoChromo code. Csagan5 has done some nice refactoring work, including moving the code from the net component to the chrome component. However, he still has the very simplistic hard-coded header file of 50K+ rules, apparently translated via script from EasyList. (Upon seeing this, I couldn't help but laugh at the irony of Google going this direction with extensions being limited to a single json file of rules.)

@joey04 please feel free to mention me next time (unless specifically you prefer not to); yes, that engine is now gone (since v77) and Bromite is now using the Chromium subresource filter without limitations and including a much more extended filter file.
I also noticed that Chromium is now using basically a binary AdBlockPlus format, with really little or no changes (less features supported), thus the migration to start using the subresource filter instead of the old NoChromo engine was smooth (except for the necessity of downloading filters without extensions, which I had to write from scratch).

Please note that even when using the NoChromo engine I had always planned to have binary-compact filters to be downloaded instead of an embedded list, I could never do it for lack of development skills and time; but upstream now delivered it.

I think another possibility (not mentioned here, if I am correct) is to extend the subresource filter with features users might desire, or to work on one of the other engines. I maintain a partial list here: https://github.com/bromite/bromite/wiki/AdBlocking

The biggest limiting factor to open-source fightback on issues like these is a lack of time: most developers don't have enough time to spare on volunteering in open-source, because they need money to live. Google (and every other major software company out there) know this, and leverage their superior developer-hour resources to outpace those trying to defend against their for-profit actions.

That is sadly true.

[travankor]

https://www.bleepingcomputer.com/news/software/google-begins-testing-extension-manifest-v3-in-chrome-canary/

[aluo-x]

@Eloston

So it seems like Google is intent on delivering Manifest V3 now, rendering extensions like ublock-origin and uMatrix far less powerful.
It seems like there are a few possible choices, in terms of difficulty:

1. Do nothing, and in the future limited adblocking can happen with declarativeNetRequest with up to 30,000 static rules, and 5,000 dynamic rules. I suspect in the future most people will migrate to this option.

2. Port over the adblock engine used by Bromite @csagan5, with additional modifications to allow for filter subscriptions etc. Seems like a fairly reasonable patch, and self contained.

3. Port over the adblock engine used by Brave, which seems to support easylist, and ublock-orign style filter lists, and seems to include for subscriptions. Doesn't look too bad, with around 10K lines of code in the core, and maybe 1K lines in the C++ ffi. Some sort of UI will likely be needed, but that could potentially happen outside the scope of this project.

4. Use any Brave patches once released to restore Manifest V2, .since they seem to indicate they will patch it out eventually. However, it is uncertain if extensions for uBbock-origin will continue to be developed for such a small minority.

[Eloston]

Option 1 is viable. But this will happen by default, so it isn't really worth discussing.

Option 2 and 3 could also work. But first, I'd like to know more about Bromite's adblock engine so we can compare them.

I'm also fine with Option 4 if it comes to fruition and it doesn't break any existing functionality.

[emanruse]

Having merely "ad" blocking is not enough for privacy and security. uBO does more than that. uMatrix can do even more than uBO along the lines of blocking 3rd party requests and type based ones. That is what we need to have in future too.

IOW: simply supporting "ad blocking rules" (which some other browsers do) will not be enough to preserve uMatrix functionality.

Long term: I wonder how hard it would be to "translate" uMatrix to C/C++ in order to have a browser which has that functionality built-in. Then that would not depend on what is available to extensions (which may be even worse in future). Additionally it would be more efficient (CPU wise) as the functionality would not have to pass through layers of JavaScript etc.

[Eloston]

Long term: I wonder how hard it would be to "translate" uMatrix to C/C++ in order to have a browser which has that functionality built-in.

Probably difficult. I suspect that it took Brave developers a few weeks worth of full-time work to implement the first iteration of their general-purpose blocking engine at the very least.

Then that would not depend on what is available to extensions (which may be even worse in future).

It might be easier to maintain custom extension APIs, but that depends on how much of the underlying components Google will be changing. However, would it be useful to augment Chromium's extension APIs to match Mozilla's WebExtensions? I don't know what Firefox's extension ecosystem is like.

[emanruse]

@Eloston

It might be best to consult @gorhill.

[csagan5]

I am following this discussion with interest however I do not think content filtering is something that ungoogled-chromium should focus on: it can, but then it would be best to start a separate project just for that part as it is potentially huge compared to the rest of the patches.

The approach I chose for Bromite is the least painful/more practical to maintain: to follow upstream. They have developed a content filtering engine (please go make some reading about it, it's quite advanced) and any alternative will always have less manpower behind it. It is also possible (and not surprising) that the subresource filter engine will be radically changed in the future, for example once upstream takes over the "game" of content filtering, having practically destroyed all extensions' competition.
The current iteration of this engine can be used without any privacy compromise (it is not tightly integrated with cloud services etc) and that is why I chose it; if this fact would change (overal several development cycles) then Bromite would have to go back to the drawing board in a discussion similar to this one.

If I were to look for a content filtering solution (either for Bromite in the future, given the above-described unfavourable scenario or for ungoogled-chromium now) I would stay with the Chromium-based engine which has the best maintenance/support.

I made a list some time ago here: https://github.com/bromite/bromite/wiki/AdBlocking

The choice seems to be between SWE WebRefiner and Brave; they both are very unfriendly to integrate because of the commits mess (no clean patches), and the former was abandoned IIRC.

If someone knows about other adblock engines for Chromium please edit that wiki page and mention them in this discussion.

In conclusion: picking Brave's engine, or any other, or developing hybrid solutions as mentioned above (unrealistic) are IMO all worth of a separate project or sub-project.

I suggest "project" because you would absolutely want contributions if something like this was ever started, and it would be best that multiple FOSS browsers use it rather than a single one.

What would such project cover?

If using the subresources filter engine:

• (desktop) patches for the UI to allow adding filters in AdBlockPlus format
• (android) patches for the UI to allow managing filters (I developed a very simple one, with one URL edit field, for Bromite)
• (core) patches to convert them on-the-fly to the subresource filter binary format
• possibly other patches to add functionality, you can look at Brave's features for example

If using Brave's engine:

• basically cleanup of Brave's patches so that they can be used for desktop/android

@thestinger what is your opinion about "framing" the development of what is being discussed here? I assume it's also not something strictly on-topic for Vanadium but my thinking is that if there was a separate project about this, multiple browsers would be able to benefit from it.

I'd like to know more about Bromite's adblock engine so we can compare them.

@Eloston the engine used in Bromite is exactly the one used by Chromium upstream, with little changes. The biggest change I had to develop from scratch was a mechanism to download filters without using extensions, you can find relevant information in the URLs I have put in previous comments.

This was also documented here and here at time of release.

• Port over the adblock engine used by Bromite @csagan5, with additional modifications to allow for filter subscriptions etc. Seems like a fairly reasonable patch, and self contained.

@aluo-x there is no Bromite adblock engine, see above link. The patch is only for downloading updates in a simple and secure way without using the extensions system. The modifications I did to the subresources filter engine are minimal (removing unnecessary safe browsing integrations and other types of navigation throttles).

[emanruse]

I do not think content filtering is something that ungoogled-chromium should focus on

ungoogled-chromium is about privacy.

Content filtering/hiding does not (necessarily) improve privacy. It may be cosmetic or partial.

However request blocking can give privacy and that aligns with this project's overall goal.

[csagan5]

@emanruse by the same reasoning we could add VPN or TOR capabilities to ungoogled-chromium. My point is that it is a non-trivial feature to develop/support/maintain and cannot be simply added as a "patch" or a few patches; see my previous comment for the rationale.

[emanruse]

by the same reasoning we could add VPN or TOR capabilities to ungoogled-chromium

I use ungoogled-chromium with Tor proxy all the time. Yes, it would be quite good to have more control like in Tor browser (switching identity etc) and better anti-fingerprinting. What is wrong with that reasoning?

My point is that it is a non-trivial feature to develop/support/maintain and cannot be simply added as a "patch" or a few patches; see my previous comment for the rationale.

Ungoogling Chrome is itself a non-trivial task (although it may look like a patch work). Still that doesn't mean it is worthless, irrational or that it should be easily denied because it requires effort.

If your point is to say that the primary goal should be just content filtering (and not request blocking) because it is easier to do - that would be a completely worthless effort because Falkon browser already has built-in support for ad blocking rules + it was not developed by Google in the first place. But Falkon cannot even dream of what uMatrix can do.

[gmt]

Sorry to add more noise here. But just a quick note about the economics of critical mass and open source:

When some hypothetical abstract fork maintainer merges X->Y patches, because, even though they don't like it, "critical mass" is going to be lacking for feature X, what's really going on? Let's assume, all things being equal, we would prefer to fork and maintain the code, except for the concern that feature "X" will no longer have critical-mass adoption levels and therefore will represent wasted labor or marginalize our fork.

There is a game-theoretic problem here. Feature "X" may stand a chance of surviving upstream's "X"->"Y" move, if enough forks keep feature "X" alive. If a bunch of small forks play 'follow the leader' because of 'critical mass' reasoning, like the above, then upstream wins by default due to a lack of challengers. But if a bunch of forks stay with "X", upstream may be forced to revert their decision, or may simply become marginalized due to declining user-base. It would seem that somehow upstream has tricked down-streams into acting against their collective better interest in such a situation.

In this case, the "X" is a bunch of content filtering functionality with entire ecosystems built around it. It is threatened by "Y," a proposed inferior feature which upstream claims is "for the children" (correction: "for security"). Upstream's project-owner has a revenue-model-scale conflict-of-interest and nobody but upstream seems to want "Y".

Normal people might not know about this yet. A lot of them might find out the hard way when this stuff lands. At that time, the critical mass thing will probably get decided. Are convenient migration paths available to them, that don't force to make a decision like "I guess I have to take this abuse, because, convenience?"

Some people finding themselves in that position will/should not move to ungoogled-chromium because it lacks the Google-cloud-sync they rely on and/or want. Perhaps they will have to find some other fork or make a compromise and adopt "Y-brand" content-filtering.

But other people are not using those features. For them, ungoogled-chromium offers, potentially at least, the same or better value proposition as upstream. It seems to me it would be a shame for ungoogled-chromium not to be an option for disgruntled users of ublock0 and other plugins, at that moment, when a plurality of chrome users will have decided to reconsider what web-client they are going to use, going forward.

[csagan5]

@gmt my comment is about developing a solution to the "banishing" of ad blocker extensions (declarativeNetRequest, Manifest V3 etc) as a stand-alone, cross-browsers project.
(I am frankly surprised about how it could be read in any other way...)

For the records, I agree with what you wrote and that is exactly the reason why I would give the community effort more success chances if it were to be coordinated with other open source projects and exist as a stand-alone, cross-browsers project instead of being developed within ungoogled-chromium or Bromite or Vanadium (GrapheneOS) or Inox or Iridium or any other open source project which has privacy in mind - but would not be able on its own (because of resources and because of the issues that arise when trying to properly define the scope) to tackle the problem being discussed on this issue.

If it is successful enough it might even set a "de facto standard" and facilitate projects forking Chromium.

[gmt]

@gmt my comment is about developing a solution to the "banishing" of ad blocker extensions (declarativeNetRequest, Manifest V3 etc) as a stand-alone, cross-browsers project.
(I am frankly surprised about how it could be read in any other way...)

My rant above isn't in response to anyone in particular... more of a coffee-fueled non sequitur attacking the argument (which may not even be seriously proposed above) that there's no point to this discussion, because upstream has a (decreasingly) benevolent dictatorship over the browser plugin ecosystem.

(tbc: in general Google has dropped a lot of really great open source into the AOSP/chromium repos resulting in great benefits to lots and lots of people. But perhaps they sometimes forget that in FOSS, you can't just unilaterally decide to remove something people don't want removed (well you _can,_ but ...))

If it is successful enough it might even set a "de facto standard" and facilitate projects forking Chromium.

Your line of thinking above, iiuc, takes my point as a given, but is focused instead on _how_ to fix the problem, not only on day 1, but in a more sustainable/maintainable way, i.e., where we don't end up with as many content filtering frameworks as browsers. I think it's a fantastic idea. No doubt, the devil's in the details but I can't think of any reason those particular devils couldn't be managed if enough people worked together on it.

Which brings us to the biggest problem with both our remarks and many others here, namely that our time is clearly better spent coding up solutions than hanging out here bikeshedding :)

[csagan5]

our time is clearly better spent coding up solutions than hanging out here bikeshedding :)

I consider the time spent discussing possibilities of open source projects coordination extremely well invested; this is my opinion based on the direct experience with the maintenance burden of Chromium patches. I can roughly estimate how wasteful it is for multiple projects to do that, often for the same patches/purposes.

[emanruse]

I consider the time spent discussing possibilities of open source projects coordination extremely well invested; this is my opinion based on the direct experience with the maintenance burden of Chromium patches. I can roughly estimate how wasteful it is for multiple projects to do that, often for the same patches/purposes.

The most efficiently spent time and effort would be if all those projects' intelligent developers join forces to work on a new browser instead of patching existing ones in which corporations dictate everything. Then that community would have the power to influence web standards too. Unfortunately that is not happening and the behemoths are ruling.

[emanruse]

BTW something interesting (perhaps a good starting point for further development?):

https://tech.slashdot.org/story/19/11/24/2055219/new-tracking-prevention-in-microsofts-edge-will-hit-google-the-hardest

[Eloston]

I think our first step is to implement something simpler, like Bromite's filtering approach (maybe not exactly, because Bromite is under GPLv3 and we're under BSD 3-clause. Would need @csagan5's input). That way, we will have a solution by the time Manifest v3 comes around.

In the long term, I think the best option based on what's currently available is to use Brave's filtering enigne. Brave has a solid implementation that will be well-supported for the foreseeable future. However, there might be a few licensing hurdles with Brave's code (I would need to know the details of MPL 2.0 to be sure).

Based on what I've read so far, I think the ideal solution would be to bring back Manifest V2 or add support for WebExtensions. Extensions provide a layer of abstraction and decoupling that makes the whole system more robust; i.e. it's easier to develop and test extensions than code in the browser. And with features like WebAssembly, the runtime performance loss should be marginal. Not to mention that there is already a large extension ecosystem we can use.

I am also open to participating in a collaborative effort to develop a new filtering engine. This is not necessarily to the exclusion of the previous two options though; if we have enough manpower, I'd like to investigate all of these long-term solutions a bit more to scope them out and assess them properly.

[travankor]

I doubt that copyleft licensing is a concern for the average user of ungoogled chromium -- at least it isn't for me. That said, you could make it optional if it's a problem.

By the way, MPL 2.0 is also a copyleft license. It's similar to the LGPL but a bit less strict when linking with proprietary projects AFAIU.

From my limited knowledge of web/content blocking, uMatrix/uBo is still the most powerful of all the available options. I think all the other options are inferior (either in quality of blocking or UI/UX or number of toggleable blocking features), but I haven't found a comprehensive comparison and analysis on the privacy effectiveness of various blockers. An analysis of the privacy goals required from a blocker would be a good way to frame the discussion IMO. There are probably university papers in this area, so maybe something to look for.

[Eloston]

I doubt that copyleft licensing is a concern for the average user of ungoogled chromium -- at least it isn't for me. That said, you could make it optional if it's a problem.

I was referring to licensing problems for ungoogled-chromium development. We don't use a copyleft license (we use BSD 3-clause), so it may pose some challenges in how we include features from copylefted projects.

[black-ish]

It has been a while and wanted to ask what is the current status of this? Is Manifest V3 already in effect and is the API removed?

[jstkdng]

@black-ish
https://developer.chrome.com/extensions/migrating_to_manifest_v3

the old api wont be removed anytime soon