Uassets: Unblock PerimeterX Bot Defender

Created on 13 Nov 2019  路  35Comments  路  Source: uBlockOrigin/uAssets

URL(s) where the issue occurs

https://www.invaluable.com/
https://www.bloomberg.com/
https://seatgeek.com/
https://seekingalpha.com/

Describe the issue

PerimeterX Bot Defender is a website security service used by hundreds of popular sites across the web. It is using Javascript to profile user behavior to identify malicious behavior. more details can be found at https://www.perimeterx.com. PerimeterX does not collect PII data or track the users. The information is only used to decide if to grant access to the site.

Some rules from the EasyPrivacy list are blocking our ability to run our code and causing normal users to be blocked.

Examples from the conversations with the EasyList team:
https://github.com/easylist/easylist/pull/3723#issuecomment-508870872
https://github.com/easylist/easylist/commit/0f70a983eff105a88780f6fc6d3fbfac3478ae12
https://github.com/easylist/easylist/pull/3724

We are asking to create a filter override for the blocked perimeterx.net domain and for /api/v2/collector

Screenshot(s)

https://github.com/easylist/easylist/commit/6489947a49ea29002534a1c064c695e2fb0eee5f#commitcomment-35818636

Versions

  • Browser/version: all
  • uBlock Origin version: all

Settings

  • none

All 35 comments

Really? I understand that the nature of bot detection involves fingerprinting, but that is something that the EasyPrivacy list explicitly blocks. https://github.com/easylist/easylist/commit/6489947a49ea29002534a1c064c695e2fb0eee5f#commitcomment-35818463

PerimeterX is not a tracking service, we have strict privacy policy and these rules are causing issues to users on hundreds of web sites (as you identified on the thread on easyprivacy).
The fingerprinting and other mechanisms rendered by easyprivacy as trackers are used solely to secure access to the site, and not correlating any data to PII.
This isn't different from recaptcha that is whitelisted (and is tracking users), and the result of blocking the script will result in the site visitors forced to solve recaptchas.

Let us know what you recommend in order to resolve this, as currently this is negatively impacting the experience of uBlockOrigin users on many sites.

This isn't different from recaptcha

Sure, it is, reCaptcha is unavoidable and results in site breakage, yours does not.

as currently this is negatively impacting the experience of uBlockOrigin users on many sites.

and what about Adguard/ABP/Adblock/Ghostery users ? Why do you care about uBlock Origin users only ?

  • ABP no use default EasyPrivacy - if that list spoils pages.

  • AdBlock by 尾:fish: have only EasyList Lite where put block rules / exceptions (as if they were trying to overwrite something).
    https://cdn.adblockcdn.com/filters/easylist_lite.txt
    Default no use EasyPrivacy.

  • For AdGuard possible report anonymously: https://reports.adguard.com/en/new_issue.html, and recommend use AdGuard lists, default without EasyPrivacy or AdGuard Tracking Protection enabled.

We are working to resolve this issue on each extension using the EasyPrivacy list.

While PerimeterX uses fingerprinting technology to identify threats, the usage of this technology is not to track users and it's never used or available in order identify, target or track users.

There is a negative impact to users of uBlockOrigin as they will see more catpchas on hundreds of sites. The PerimeterX tool is not as unavoidable as reCaptcha, but it deployed on some of the largest and most popular sites on the internet today.

?

https://github.com/uBlockOrigin/uAssets/commit/ebec09871ec0accafbd37fe932a67c70dad6eb80 will help unblock the user from solving captcha (which is great in itself), but it will not help with avoiding getting a captcha. The purpose of this thread was to unbreak the root cause, thus avoiding the user seeing any impact.

@ZaphodBeebblebrox, I'd appreciate reopening this issue.

We could add
@@||perimeterx.net/api/*/collector$xhr

What do you think @okiehsch @gorhill @gwarser

Refreshing the page works and captCha can be avoided like that once it occurs.

It seems that there is no agreement to address the problem, closing

Narrow to @@||*/captcha/PX*/captcha.js$script,1p ?

@@/captcha/PX*/captcha.js$script,1p

works identical?

Narrow to @@||*/captcha/PX*/captcha.js$script,1p ?

The captcha is not broken on every site using that script, atleast on my end.
For example
https://twentytwowords.com/kylie-jenner-and-travis-scott-are-planning-on-having-another-child-together/`
with the script blocked.
image

They started asking people to subscribe to this list on captcha page: https://abp.perimeterx.com/whitelist.txt

https://abp.perimeterx.com/whitelist.txt
We could badfilter them. 馃榿

Then these users wouldn't be able to use the site... That was the only option we could think of to resolve the issue.

@omriiluz seekingalpha has been fixed
other sites with captcha broken ?

https://www.bloomberg.com/

@llacb47 https://github.com/uBlockOrigin/uAssets/issues/6819#issuecomment-573462497

again I have troubles with: https://www.bloomberg.com/europe

obraz | obraz
--- | ---
\|\|cedexis.net^$3p | \|\|cedexis.net^$3p,badfilter
layout is broken | layout looks fine

filter log:

| | | | | | | | |
|:--- |:--- |:--- |:--- |:--- |:--- |:--- |:--- |
| +1 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 0,3 | script | https://2-01-3073-0019.cdx.cedexis.net/ | aliasURL=https://www.bloomberg.com/service-worker.js |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/BWHaasGrotesk-55Roman-Web,BWHaasGrotesk-75Bold-Web,Bloomberg/font-face.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/fence/v4/styles.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/css/module/story_package/_variation/six_up_images-98a2c8aaf3.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/css/module/story_package/_variation/3_up_autoplay-54196db3b8.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/bbnav/public/stylesheets/bb-global-footer-7fce47d03a.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/Bloomberg/font-face.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/css/components/dataStrip/dataStrip-a3c93caf50.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/vendor/css/reg-ui-client-162e0da6e2.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/css/components/masthead/masthead-63fcec205f.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/Bloomberg%20Logotypes-Medium,BWHaasGrotesk-75Bold-Web,BWHaasGrotesk-95Black-Web,BWHaasGrotesk-55Roman-Web/font-face.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0019.cdx.cedexis.net/ | aliasURL=https://www.bloomberg.com/tophat/assets/v2.4.0/that.css |
| +1 | \|\|cedexis.net^$3p | -- | behind-the-scene | 0,3 | xhr | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/AvenirNextPForBBG:400,600/styled-font-face.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/images/favicon-black-63fe5249d3.png |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/images/apple-touch-icon-180x180-c1a237984e.png |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/module/newsletter/newsletter_signup-187d8919e5.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/module/newsletter/NewsletterView-c75216c0e4.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/module/story_package/StoryPackageView-86101ff87c.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/module/single_story/SingleStoryView-9964bee019.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/lazyLoad/lazyLoad-57e6ac79dd.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/auto_page_reload/auto_page_reload-8ba99caff1.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/continuous_client/prev_viewed-78792dbc70.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/geoip/geoip-e5fab131a4.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/lazy_images/lazy_images-95fa8c8eb4.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/dependencies/dependencies-cbd0a4f059.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/initialize_pianola/initialize_pianola-08be4cc87d.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/timestamp/timestamp-c932de4e4c.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/abba_variants/abba_variants-9fdac72210.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/abba/abba-4de8a9504e.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/polyfills/polyfill-60a44ed0c6.sync.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/event_bus/event_bus-a9f299e831.sync.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/js/header-d7841082fc7455378b3b.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0019.cdx.cedexis.net/ | aliasURL=https://www.bloomberg.com/tophat/assets/v2.4.0/that.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iM1_7u4Hydjg/v1/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iwaNv4EATB1k/v1/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iF0X4nuARsvE/v0/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iIpxs4bO4b30/v0/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iqHc7JLeH1Hg/v0/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iuaRT6QalS0E/v0/800x600.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/fence/v4/app.bundle.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/BWHaasGrotesk-55Roman-Web,BWHaasGrotesk-75Bold-Web,Bloomberg/font-face.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/fence/v4/styles.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/polyfills/polyfill-60a44ed0c6.sync.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/js/event_bus/event_bus-a9f299e831.sync.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | script | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/js/header-d7841082fc7455378b3b.js |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iM1_7u4Hydjg/v1/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iwaNv4EATB1k/v1/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iF0X4nuARsvE/v0/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iIpxs4bO4b30/v0/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iqHc7JLeH1Hg/v0/25x19.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | image | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iuaRT6QalS0E/v0/800x600.jpg |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/css/module/story_package/_variation/six_up_images-98a2c8aaf3.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/javelin/public/hub/css/module/story_package/_variation/3_up_autoplay-54196db3b8.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/bbnav/public/stylesheets/bb-global-footer-7fce47d03a.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/Bloomberg/font-face.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/css/components/dataStrip/dataStrip-a3c93caf50.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/vendor/css/reg-ui-client-162e0da6e2.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/s3/navi/css/components/masthead/masthead-63fcec205f.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/Bloomberg%20Logotypes-Medium,BWHaasGrotesk-75Bold-Web,BWHaasGrotesk-95Black-Web,BWHaasGrotesk-55Roman-Web/font-face.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0019.cdx.cedexis.net/ | aliasURL=https://www.bloomberg.com/tophat/assets/v2.4.0/that.css |
| +0 | \|\|cedexis.net^$3p | -- | www.bloomberg.com | 3 | css | https://2-01-3073-0012.cdx.cedexis.net/ | aliasURL=https://assets.bwbx.io/font-service/css/AvenirNextPForBBG:400,600/styled-font-face.css |

@mapx- we are verifying one by one and will respond later today

@krystian3w is bloomberg affected by perimeterX ?!

Off-topic! I'm sure @omriiluz has better things to do than read about how CNAME uncloaking breaks Bloomberg.com.

@mapx- these are not showing the captcha element due to the block:

https://www.hotelscombined.com/?_pxhc=1
https://creativemarket.com/?_pxhc=1

fixed hotelscombined , I get no captcha for creativemarket

Despite multiple issues to fix Bing Ads, there has been no response whatsoever and agitation from repo maintainers:
So, referencing my issue here:
https://github.com/uBlockOrigin/uAssets/issues/6887#issuecomment-578509029

Let's name and shame ublock origin until their ego issues get resolved.

@mapx- broken on bloomberg.com

image

image

Needing additional whitelisting @mapx-
image
@@||pxchk.net/api/v2/collector/ocaptcha$xhr

Due to ||pxchk.net^$third-party in EasyPrivacy

Since I was blocking px-cdn.net I needed to add @@||px-cdn.net/api/v2/collector/ocaptcha$xhr as well.

I cannot reproduce the captcha thing

Also uses paywall - maybe JS conflit like "frozen" nodes/objects.

but they are probably checking if object is frozen, and if, then they not show captcha

I get a paywall on my side, _"Read with Free Trial"_.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

Jose1971AB picture Jose1971AB  路  3Comments

terrorist96 picture terrorist96  路  3Comments

melnation-com picture melnation-com  路  4Comments

Jose1971AB picture Jose1971AB  路  3Comments

patrickdrd picture patrickdrd  路  4Comments