Brave-browser: Provide a Brave-specific User Agent for some sites

[bsclifton]

Updated description

There should be a way to share a Brave-specific User Agent

We currently do this with our "site hacks":
https://github.com/brave/brave-core/blob/66ceaf3491778f7d9a62f215c8fb47b318c9b181/browser/net/brave_site_hacks_network_delegate_helper.cc#L62-L76

User Agent example format:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Brave Chrome/79.0.3945.117 Safari/537.36

This user agent is only shared with a few hard-coded sites, current list found here:
https://github.com/brave/brave-core/blob/4621aa3f71758f3bd890f381f5774d91a0338b2e/common/shield_exceptions.cc#L17-L29

The scope for this issue would be to expand to our list to be dynamic (remove hard-coded entries) and match the list we provide for our supported publishers (already fetched on browser launch).
https://laptop-updates.brave.com/promo/custom-headers

Original issue description

Carried over from https://github.com/brave/browser-laptop/issues/3693
Related https://github.com/brave/brave-browser/issues/4641

Brave should show some bravery of its own and own up to having its own User-Agent. I'm aware of and acknowledge this issue, but without identifying itself – brave will be counted in online statistics as just Chrome. Who knows? Maybe Brave will be more popular than Chrome one day, but website owners will not be prompted to sit up and take notice of Brave if they can't identify any traffic from the browser and its users.

The original issue report is also misleading. Of course the User-Agent would be unique in that test if no one else yet have run the test. With more users, it's uniqueness for tracking will be reduced over time.

• Original implementation and test plan can be found in https://github.com/brave/brave-core/pull/2442
• This implementation was reverted with https://github.com/brave/brave-core/pull/2583

We may need these initial overrides to start:
UA overrides in Edge: edge://compat/useragent

[tomlowenthal]

As a general policy, we don't want the user-agent string to make Brave users much more distinctive. Our general ballpark plan is to switch to a Brave UA when users on that platform will have an anonymity set of about a million.

[grahamperrin]

FWIW, a couple of cases where a distinctive string _might_ be problematic:

• Discord
• Slack

Related, in the Waterfox area:

• https://www.reddit.com/r/waterfox/comments/9drjqx/user_agent/e66hdxs/
• https://www.reddit.com/r/waterfox/comments/9drjqx/user_agent/e66h85p/

[sairam]

From the results I see for the browser - https://panopticlick.eff.org/results?aat=1#fingerprintTable

The UserAgent string displays the following:

1. OS with version
2. make of the processor

A simpler way to customize this User Agent string to mask this information could be a starting point for customization.

[bbondy]

Updated the title. We feel like we have a big enough user base now to go ahead with adding Brave to the end of the User-Agent.

So instead of:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36

We'd show:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36 Brave

[da2x]

@bbondy for a slew of historical reasons the Brave component should have a version number to be compatible with existing User-Agent parsing libraries (and regex soups). Some strict profiles of web application firewalls (things like mod_security) block unusually looking User-Agents; like UAs with unversioned components. You can either use Brave’s own marketing version number, or just copy the major version or full version from the Chrome component.

E.g. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36 Brave/74

[bbondy]

Thanks @da2x we were considering just doing Brave/1.0 in a side discussion but maybe the Chrome version is better.

[Fornax96]

Using Brave/1.0 will probably break a few very badly written web apps. But I think it might be worth it in order to stop all this User Agent madness. Having five browser names and version numbers in your user agent is completely unnecessary nowadays (and humiliating too).

+1 for Brave/1.0 so we can begin to end the user agent war.

[da2x]

@bbondy More reasons to copy the Chr version: HTTP/2 uses header compression and repeated [version] strings compress better than unique ones. 😋 It also doesn't leak any additional data.

@Fornax96 Have a look at the never ending battle over this in the WebCompat database. It's not that easy. Product/Version would be technically nice but it doesn't work. Too many services are completely unmonitored and unmaintained, and require clients to deal with the legacy of the web. You still lose access to portions of the web without the Mozilla/5.0 prefix.

That being said, Brave could probably go with just Brave/Version on HTTP2. At least as an experiment.

[Fornax96]

I like the HTTP/2 idea.

Brave could also add a user agent switcher like Opera has. There will always be some bad seeds left among web developers. For instance a while ago I had to use a user agent switcher in chromium because Whatsapp locked me out.

EDIT: Actually it turns out whatsapp still blocks users based on their user agent. This is what happens when I change my agent to Brave/1.0

I sent them an email about it a year ago but they never replied.

[iefremov]

@bbondy @bsclifton @tomlowenthal
I would say it is not that hard to foresee potential web compatibility problems that this change could bring, see for example the comment above.
Probably we want to implement some kind of UA spoofing that will allow us to rollback the UA for problematic domains (this list of course should be able to update on the fly)

[da2x]

All other browser have a webcompat component that does UA overrides for specific domains. Opera has their browser.js, Firefox and Safari has webcompat.js, Edge has DomainActions, ... . Brave can potentially source these lists, assuming compatible licenses, for troublesome domains.

[Fornax96]

I used a plugin to change my user agent to Brave/1.0 for an entire day. I'd say about 40% of the websites I visited were broken. Having the Mozilla/5.0 and Chrom(e/ium)/version signatures are definitely important for compatibility.

I suggest including Chromium's user agent into Brave's user agent, se we can still advertise some degree of freedom.

[da2x]

@Fornax96 I told you so. 😜 I guess reCAPTCHA and Cloudflare were particularly unhappy with you? (They don't like it when your navigator.userAgent and the HTTP User-Agent don't match.) If you care about _User-Agent discrimination_, then please contact the websites in questions first and then report the issues to the WebCompat project. Do attempt to reach out and make first contact with the site before reporting, though.. You'll have more success if you accept that Mozilla/5.0 is just going to be required and report sites that block Mozilla/5.0 (Linunx) Brave/1.0 (or Windows|Mac OS X) instead. That is about as compact as you can get it and still expect to get about on the web.

[Fornax96]

All Google products showed me javascriptless versions, or they just refused to let me use them. Graph plotting libraries were throwing errors. Whatsapp wouldn't let me in. And some sites showed me a very badly maintained plain text version.

I understand why they do it though. There are people around who are using text based browsers, or windows XP who actually need those watered-down versions of those websites. The Mozilla/5.0 and chrome version number are a pretty good indicator of what a browser supports (and if it's even graphical or not)

Websites also use the OS part of the string to choose between mobile and desktop versions of their site, so you really need to have Ubuntu, Linux, Android or iOS in there

[iefremov]

@da2x @Fornax96 In this issue we are appending Brave/1.0 to the existing UA, not replacing the entire string. But this also can be a problem, indeed.

[cwbriscoe]

I agree there should be an option in Brave Shields dropdown to use the current string in case a site doesn't like Brave/1.0 on the end. Some sites may go out of their way to block Brave/1.0 because they know it is probably blocking their precious ads.

[bsclifton]

Re-opening after code was reverted with https://github.com/brave/brave-core/pull/2583

[bsclifton]

Quick update...

After this revert due to web compatibility issues, the new plan is:

• create a new branch and re-land the UA changes of https://github.com/brave/brave-core/pull/2442 and https://github.com/brave/brave-core/pull/2515 (leaving the header parts alone, for the time being)
• continue to flush out UA exceptions as part of brave/brave-browser#4641; we'd want a list similar to https://github.com/brave/autoplay-whitelist. If items are on this list, we send Chrome UA
• re-submit work as pull request which closes this issue and brave/brave-browser#4641

[bsclifton]

Dropped down to a P3
Blocked by https://github.com/brave/brave-browser/issues/4641

[pietsnot123]

Identifying as Brave instead of generic Chrome would make my browser much more unique. In fact, I consider even leaking my operating system (Linux) as a problem as it has a market share below 5%. I currently work around this using a header modifyer-extension to identify as the most normal Chrome/Windows combination.

Brave should have an option to always serve a User-Agent string that pretends to be Chrome on Windows.

[bsclifton]

This is something we will be revisiting - stay tuned!

[FatBirdie]

-tuned!-

[jzstern]

Following for updates. I'm using a webm background on my site that doesn't autoplay unless the user knows to click always allow. Would like to show a toast message for Brave users to enable it and get the full experience (:

[FatBirdie]

This is something we will be revisiting - stay tuned!

Is there any ETA for this update?

[FatBirdie]

Provide a Brave-specific User Agent for some sites?
Was not the idea to provide Brave user agent by default with a site exclusion list?

[bsclifton]

@FatBirdie that is correct; maybe the right thing to do would have been to create a new issue instead of changing this existing one

For those following, the current plan is to do the following:

1. Provide a Brave-specific UA for sites in an existing publishers list
2. Switch to Brave-specific UA for ALL sites and then use an exclusion list

cc: @rebron, in case you wanted to provide an update

I'm going to start dev on this issue and am shooting for 1.5 (currently on Nightly)

[FatBirdie]

@bsclifton Hm, I must have missed something. What will be the difference between user agent for sites on a publishers list and "ALL" sites? So there will be 3 different categories of UAs in total?

[bsclifton]

@FatBirdie UA would be the same, when brave-specific UA is presented

Think of this as a gradual move towards always presenting Brave-specific UA. We'll start with presenting only for a small list of known sites (which grows), covered by this issue.

Then (as we can ensure experience is good / we have more confidence) we'll move towards always presenting, unless it's in an exclusion list. We'll need a new issue filed for this

For sites breaking when Brave-specific UA is present (why we reverted this change), we'll have to work with them long-term to fix and temporarily add them into this exclusion list. WhatsApp broke completely for example. We may also allow user to configure the UA behavior (needs more discussion)

[FatBirdie]

@bsclifton Okay, I see. So for the (small) site list of Brave specific UA, will it be possible to add sites to it on Github or will it be closed?

[pietsnot123]

Google has announced to phase out the whole User-Agent starting March 2020. By September 2020, all desktops are to show the same User-Agent, no matter what version of Chome or what OS you're using.
Maybe Brave should follow this as well. That would significantly improve privacy.

https://www.zdnet.com/article/google-to-phase-out-user-agent-strings-in-chrome/

[FatBirdie]

It would be a shame if Brave doesn't display their UA for the time being that there is a referral program in place for new users. I'm currently on-boarding hundreds of new users to brave each month through one of my websites, and if there were an option to actually know who is running Brave and not those numbers could be improved even further, of this I'm sure.

Personally I wouldn't mind what you're mentioning @pietsnot123 but for the time being, I do think there are many website owners that are on-boarding new users to Brave who would oppose it.

[diracdeltas]

We are actually very interested in adding Brave/... to the UA but hesitant to do so immediately because we don't know if sites will break because of this (both desktop and mobile). If anyone has thoughts on how to test this lmk.

[da2x]

@diracdeltas install Opera and look for browser.js in the profile directory. It contains a list of websites that need UA cloaking. The same list of websites causing problems for OPR/ will probably cause problems for Brave/. There are 42 sites on the list.

[da2x]

@diracdeltas The Web Compatibility Project maintains a list of issues with the uaoverride label which should also be an excellent resource.

[bsclifton]

Closing in favor of https://github.com/brave/brave-browser/issues/8216

• solution in https://github.com/brave/brave-browser/issues/8216 will allow detection of Brave
• Chromium is looking at Freezing the UA. More info available:
  
  
  
  • https://www.chromestatus.com/feature/5704553745874944
  
  
  • https://bugs.chromium.org/p/chromium/issues/detail?id=955620
  
  
• no inclusion/exclusion list needed
• not modifying UA lowers web compatibility risks