Uassets: nytimes.com detects private browsing

[josephcsible]

URL(s) where the issue occurs

https://www.nytimes.com/2019/05/23/science/spacex-launch.html

Describe the issue

A few seconds after the page loads, the site will detect that you're visiting it in private browsing mode and will hide all of the content.

Versions

• Browser/version: Firefox 67.0
• uBlock Origin version: 1.19.6

Settings

• All defaults

Notes

nytimes.com##+js(abort-on-property-read.js, window.indexedDB) fixes it for me, but interestingly, nytimes.com##+js(set-constant.js, window.indexedDB, undefined) does not.

[okiehsch]

I can't reproduce.

[josephcsible]

Hmm, I can reproduce it on my Linux box but not my Windows box (both same browser and uBO versions). I'll poke around a bit to try to figure out why.

[josephcsible]

I created a completely new profile in Windows, installed uBO without doing anything else, then went there in private browsing mode, and then I could reproduce it there.

[okiehsch]

I can't reproduce with Linux or Windows on my end.

[josephcsible]

Did you try with a fresh profile?

[okiehsch]

With Firefox, I only use it to test filters anyway, so it is a default installation + uBO.

I can't reproduce with Chromium either, though I have not tried a fresh profile.

[okiehsch]

Let's wait and see if other users can reproduce,

[josephcsible]

I think they might be doing some kind of A/B testing or something, where not every request has the malicious code. I just opened my original Windows Firefox profile again (the one that I didn't repro on before), and now I do repro there.

[josephcsible]

This is definitely still happening. Can it be reopened?

[yourduskquibbles]

I could reproduce on Test Firefox profile on Windows 10 with only uBO & stock filters.

[mapx-]

does the OP's filter fix the issue ?
nytimes.com##+js(abort-on-property-read.js, window.indexedDB)

[okiehsch]

I still can't reproduce.
When the private browsing mode of Firefox is enabled, the IndexedDB throws an InvalidStateError because it is not available in private browsing mode, so the mentioned filter probably works.

[yourduskquibbles]

@josephcsible nytimes.com##+js(abort-on-property-read.js, window.indexedDB) partially works for me and allows the article to load but it also seems to break embedded objects within the page with that filter as well.

I see a broken twitter embed, a missing youtube video and a missing nytimes.com embedded frame in the article page with that filter. Are you getting those page embedding breaking issues as well?

[okiehsch]

Interestingly nothing is broken on my end with that filter, I see the youtube video, is it always broken on your end - with that filter in place - or only if the private browsing detection kicks in?

[yourduskquibbles]

Interestingly nothing is broken on my end with that filter, I see the youtube video, is it always broken on your end - with that filter in place - or only if the private browsing detection kicks in?

Yes broken for me on regular or private browsing with nytimes.com##+js(abort-on-property-read.js, window.indexedDB) as my only filter besides stock filters.

Below is a screenshot from a non-private browsing session with logger & example link side by side. The nytimes.com##+js(abort-on-property-read.js, window.indexedDB) filter looks like it is being applied against the nytimes.com embeded frame first on my side which then maybe cripples the twitter and youtube embed as well?

The page looks the same with broken embeds in private browsing mode as well but doesn't break the embeds when I remove nytimes.com##+js(abort-on-property-read.js, window.indexedDB)

[spirillen]

Mostly a note....

I was trying to reproduce this issue, but couldn't, however I coulnd't help myself from noting that out of 77 queries the 37 (~50%) of them is tracking / spam .... Shouldn't nytimes.com be a "newspaper" protecting democracy? or is it just me as a European who have misunderstood the concept of "_American_" democracy!!

[Jap2-0]

I can reproduce on both Firefox and chrome on Windows 7. I'm attempting to work on a filter to get around it, but it's going rather slowly.

*note: I'm using more than stock filters

[irlcatgirl]

Happens for me in Brave Browser (Chromium fork)

[in4u]

Same issue using Firefox 67.0.4 on 64-bit Windows 10 as well as Windows 7. Tested both filters shared by @josephcsible.

Tried nytimes.com##+js(set-constant.js, window.indexedDB, undefined) first and it worked just one time and never again.

Tried nytimes.com##+js(abort-on-property-read.js, window.indexedDB) and it's been working so far.

However, the embedded video and twitter feed also gets hidden as a side effect. With the filter enabled, this happens in both normal as well as private browsing mode. Without filter, the embedded items don't get hidden in normal mode.

[gorhill]

Disabling JavaScript works fine if you just want to read an article.

[numbermaniac]

I can reproduce this both with uBO turned off, and while it is on.

Firefox 67.0.4, uBO 1.20.0, macOS 10.14.4

[okiehsch]

Does
nytimes.com##^script:has-text(NYTD.Abra)
work for you?

[numbermaniac]

@okiehsch Yes, that fixed it for me.

[okiehsch]

@numbermaniac just a heads up if you add
nytimes.com##^script:has-text(NYTD.Abra)
to your filter list it will also cause some site breakage, for example it breaks the youtube video at
https://www.nytimes.com/2019/05/23/science/spacex-launch.html
and embedded tweets will not show up.
The filter is only meant to be used if you just want to read an article in incognito mode.

For Chrome/Chromium users
nytimes.com##+js(set-constant.js, NYTD.Abra, noopFunc)
works on my end without causing any noticable breakage.
The filter does not work using Firefox.

[llacb47]

May be of some relevance: Bypassing anti-incognito detection in Google Chrome

And a page where you can test detection

Bug filed on Chromium issue tracker

[llacb47]

Here's the code the NYT uses to detect private browsing -- from webpack:///./src/shared/utils/privateBrowsingDetect.js. I can reproduce in Chrome and Firefox.

/* eslint-disable consistent-return */

let on;
let off;

const isSafari = () => {
  if (Object.prototype.toString.call(window.HTMLElement).indexOf('Constructor') > 0) {
    return true;
  }

  if (
    // eslint-disable-next-line no-undef
    (!window.safari || safari.pushNotification).toString() === '[object SafariRemoteNotification]'
  ) {
    return true;
  }

  try {
    return window.localStorage && /Safari/.test(window.navigator.userAgent);
  } catch (e) {
    return false;
  }
};

const isMozilla = () => 'MozAppearance' in document.documentElement.style;

const Webkit = () => {
  if (window.webkitRequestFileSystem) {
    window.webkitRequestFileSystem(window.TEMPORARY, 1, off, on);
    return true;
  }
};

const Mozilla = () => {
  if (isMozilla()) {
    const db = indexedDB.open('test');
    db.onerror = on;
    db.onsuccess = off;
    return true;
  }
};

const Safari = () => {
  if (isSafari()) {
    // iOS 11
    // Origin: https://gist.github.com/cou929/7973956#gistcomment-2272103
    try {
      window.openDatabase(null, null, null, null);
    } catch (e) {
      on();
      return true;
    }

    // Older Safari
    try {
      if (localStorage.length) off();
      else {
        localStorage.x = 1;
        localStorage.removeItem('x');
        off();
      }
    } catch (e) {
      // Original gist: https://gist.github.com/jherax/a81c8c132d09cc354a0e2cb911841ff1

      // Safari only enables cookie in private mode
      // if cookie is disabled then all client side storage is disabled
      // if all client side storage is disabled, then there is no point
      // in using private mode
      navigator.cookieEnabled ? on() : off(); // eslint-disable-line no-unused-expressions
    }

    return true;
  }
};

const IE10Edge = () => {
  if (!window.indexedDB && (window.PointerEvent || window.MSPointerEvent)) {
    on();
    return true;
  }
};

export const checkPrivate = (onCb, offCb) => {
  on = onCb || (() => {});
  off = offCb || (() => {});
  Webkit() || Mozilla() || Safari() || IE10Edge() || off(); // eslint-disable-line no-unused-expressions
};

export const checkPrivateWhitelist = (onCb, offCb) => {
  const whitelistOn = onCb || (() => {});
  const whitelistOff = offCb || (() => {});

  const privateButAllowed = [
    'FBAV',
    'FBAN',
    'FBIOS',
    'FBBV',
    'FBDV',
    'FBMD',
    'FBSN',
    'FBSV',
    'FBSS',
    'FBCR',
    'FBID',
    'FBLC',
    'FBOP',
    'Twitter for iPhone',
    'TwitterAndroid',
    'nytios',
    'nytiphone',
    'nytipad',
    'nyt-android',
    'AppleNews',
    'Flipboard',
  ];

  const re = new RegExp(privateButAllowed.join('|'), 'i');
  if (re.test(navigator.userAgent)) {
    whitelistOn();
    return true;
  }

  whitelistOff();
  return false;
};

[llacb47]

I can reproduce Washington Post private mode detection https://www.washingtonpost.com/business/2019/07/30/trump-backpedals-china-threats-trade-deal-shows-signs-slipping-away/ in Chrome, but not Firefox in neither browser.
WaPo's private mode detection code is obfuscated... and inside an inline script.

"condition_isPM": function() {
    return new Promise(function(_0x5c8e98) {
        var _0x3054d7 = !![];
        if (navigator[_0x2ec0("0x1a")]["indexOf"](_0x2ec0("0x1b")) >
            -1) return (new Promise(function(_0x5c8e98) {
            var _0x36aeee = window["RequestFileSystem"] || window[_0x2ec0("0x1c")];
            if (_0x36aeee) _0x36aeee(window[_0x2ec0("0x1d")], 100, function() {
                _0x3202b0["pv"] = ![];
                _0x5c8e98({
                    "type": "pv",
                    "result": _0x3202b0["pv"],
                    "overlay": _0x3054d7
                })
            }, function() {
                _0x3202b0["pv"] = !![];
                _0x5c8e98({
                    "type": "pv",
                    "result": _0x3202b0["pv"],
                    "overlay": _0x3054d7
                })
            })
        }))["then"](function(_0x48925e) {
            _0x5c8e98(_0x48925e)
        });
        else if (navigator[_0x2ec0("0x1a")][_0x2ec0("0x1e")]("Safari") > -1) try {
                var _0x50cda8 =
                    _0x2ec0("0x1f");
                window[_0x2ec0("0x20")][_0x2ec0("0x21")](_0x50cda8, _0x2ec0("0x22"));
                window[_0x2ec0("0x20")][_0x2ec0("0x23")](_0x50cda8);
                window[_0x2ec0("0x24")][_0x2ec0("0x21")](_0x50cda8, _0x2ec0("0x22"));
                window["localStorage"][_0x2ec0("0x23")](_0x50cda8);
                window[_0x2ec0("0x25")](null, null, null, null);
                _0x3202b0["pv"] = ![]
            } catch (_0x151b90) {
                _0x3202b0["pv"] = !![]
            } else if (navigator[_0x2ec0("0x1a")][_0x2ec0("0x1e")](_0x2ec0("0x26")) > -1) _0x3202b0["pv"] = !navigator[_0x2ec0("0x27")];
            else if (window[_0x2ec0("0x28")] ||
            window[_0x2ec0("0x29")]) _0x3202b0["pv"] = !window[_0x2ec0("0x2a")];
        else _0x3202b0["pv"] = ![];
        _0x5c8e98({
            "type": "pv",
            "result": _0x3202b0["pv"],
            "overlay": _0x3054d7
        })
    })
}

[llacb47]

Since the WaPo anti-adblock and private mode detection are both in the same inline script, the filter washingtonpost.com##+js(abort-current-inline-script, Promise.all, adblocked) from https://github.com/uBlockOrigin/uAssets/commit/f0ade04a56a09472c2715e4525f19598c3490a9f takes care of both.

[llacb47]

These filters work in Chrome (not Firefox) but they mess up fonts:

nytimes.com##+js(abort-current-inline-script.js, NYTD.Abra, indexedDB)
nytimes.com##+js(set-constant.js, NYTD.Abra, noopFunc)

Edit: tested in Chrome and Firefox, these filters don't seem to mess up fonts or anything else. Can you test these @okiehsch?

nytimes.com##+js(set-constant.js, webkitRequestFileSystem, noopFunc)
nytimes.com##+js(abort-current-inline-script.js, NYTD.Abra, indexedDB)
nytimes.com##+js(abort-on-property-read.js, indexedDB.open)

Test link with video: https://www.nytimes.com/2014/01/07/us/burglars-who-took-on-fbi-abandon-shadows.html
Test link with fonts: https://www.nytimes.com/2019/07/30/magazine/deported-mother.html
Test link with third-party embeds: https://www.nytimes.com/2019/05/23/science/spacex-launch.html

[okiehsch]

https://chromereleases.googleblog.com/2019/07/stable-channel-update-for-desktop_30.html
Fixes the issue for Chrome users.

nytimes.com##+js(set-constant.js, webkitRequestFileSystem, noopFunc)
works on my end using Firefox.

[llacb47]

Will filters to fix this still be added? Not everyone who uses uBO has an up-to-date Chrome.

[llacb47]

So this issue will not be fixed in older Chrome versions?

[numbermaniac]

Is there a reason you're not updating Chrome?

[okiehsch]

With Chrome 76+ the issue "disappears".
https://www.zdnet.com/article/google-to-clamp-down-on-incognito-mode-detection

[llacb47]

I have an old version of Chromium to avoid a redesign, but at some point I will probably upgrade due to security concerns. However, not everyone uses the latest version of Chrome, and if a filter can be added for that, that would be appreciated.

[llacb47]

The filter nytimes.com##+js(set-constant.js, webkitRequestFileSystem, noopFunc) works in Firefox and Chrome. This filter doesn't break anything in either browser, as webkitRequestFileSystem is only used for private mode and third party cookie detection by nytimes.com. Please remove the pre-processing directive !#if env_firefox ... !#endif.

[josephcsible]

On Chrome 76.0.3809.100 and uBO 1.21.6, I still get caught by this, even with the change made in commit 18955b6.

[okiehsch]

Yes, they changed it again to circumvent
https://www.blog.google/outreach-initiatives/google-news-initiative/protecting-private-browsing-chrome/
and our filter will also not work anymore.

[okiehsch]

nytimes.com##+js(set-constant.js, NYTD.Abra, trueFunc)
works if you just want to read the articles.
It will break the homepage.

[krystian3w]

Install chrome 76 and test again?

• https://nakedsecurity.sophos.com/2019/07/22/chrome-76-blocks-websites-from-detecting-incognito-mode/
• https://www.ghacks.net/2019/06/14/chrome-76-blocks-sites-from-detecting-incognito-mode/
• https://www.bleepingcomputer.com/news/software/chrome-76-released-with-blocked-flash-incognito-detection-fix/

[okiehsch]

Install chrome 76 and test again?

[llacb47]

https://mishravikas.com/articles/2019-07/bypassing-anti-incognito-detection-google-chrome.html

[josephcsible]

@llacb47 Yep, that's what they're doing. Here's the new malicious code snippet (they even left a link in the comment pointing to the site you mentioned):

/* eslint-disable consistent-return */

let on;
let off;

/**
 * Quota for an incognito Chrome window is a fraction (10%) of the device memory with an upper limit of 120MB.
 * More info: https://mishravikas.com/articles/2019-07/bypassing-anti-incognito-detection-google-chrome.html
 */
export const INCOG_MAX_QUOTA = 120;
export const MB_IN_BYTES = 2 ** 20;

const isSafari = () => {
  if (Object.prototype.toString.call(window.HTMLElement).indexOf('Constructor') > 0) {
    return true;
  }

  if (
    // eslint-disable-next-line no-undef
    (!window.safari || safari.pushNotification).toString() === '[object SafariRemoteNotification]'
  ) {
    return true;
  }

  try {
    return window.localStorage && /Safari/.test(window.navigator.userAgent);
  } catch (e) {
    return false;
  }
};

const isMozilla = () => 'MozAppearance' in document.documentElement.style;

const Webkit = () => {
  if (window.webkitRequestFileSystem) {
    window.webkitRequestFileSystem(window.TEMPORARY, 1, off, on);
    return true;
  }
};

const Mozilla = () => {
  if (isMozilla()) {
    const db = indexedDB.open('test');
    db.onerror = on;
    db.onsuccess = off;
    return true;
  }
};

const Safari = () => {
  if (isSafari()) {
    // iOS 11
    // Origin: https://gist.github.com/cou929/7973956#gistcomment-2272103
    try {
      window.openDatabase(null, null, null, null);
    } catch (e) {
      on();
      return true;
    }

    // Older Safari
    try {
      if (localStorage.length) off();
      else {
        localStorage.x = 1;
        localStorage.removeItem('x');
        off();
      }
    } catch (e) {
      // Original gist: https://gist.github.com/jherax/a81c8c132d09cc354a0e2cb911841ff1

      // Safari only enables cookie in private mode
      // if cookie is disabled then all client side storage is disabled
      // if all client side storage is disabled, then there is no point
      // in using private mode
      navigator.cookieEnabled ? on() : off(); // eslint-disable-line no-unused-expressions
    }

    return true;
  }
};

const IE10Edge = () => {
  if (!window.indexedDB && (window.PointerEvent || window.MSPointerEvent)) {
    on();
    return true;
  }
};

export const Chrome = () => {
  return Promise.resolve()
    .then(() => {
      if (navigator.storage && navigator.storage.estimate) {
        return navigator.storage.estimate();
      }
      return undefined;
    })
    .then(estimate => {
      if (estimate && estimate.quota) {
        const { quota } = estimate;
        const limit = INCOG_MAX_QUOTA * MB_IN_BYTES;
        if (quota <= limit) {
          return true;
        }
      }
      return false;
    })
    .catch(() => false);
};

export const checkPrivate = (onCb, offCb) => {
  on = onCb || (() => {});
  off = offCb || (() => {});

  Chrome().then(isPrivate => {
    if (isPrivate) {
      return on();
    }
    return Webkit() || Mozilla() || Safari() || IE10Edge() || off();
  });
};

export const checkPrivateWhitelist = (onCb, offCb) => {
  const whitelistOn = onCb || (() => {});
  const whitelistOff = offCb || (() => {});

  const privateButAllowed = [
    'FBAV',
    'FBAN',
    'FBIOS',
    'FBBV',
    'FBDV',
    'FBMD',
    'FBSN',
    'FBSV',
    'FBSS',
    'FBCR',
    'FBID',
    'FBLC',
    'FBOP',
    'Twitter for iPhone',
    'TwitterAndroid',
    'nytios',
    'nytiphone',
    'nytipad',
    'nyt-android',
    'AppleNews',
    'Flipboard',
  ];

  const re = new RegExp(privateButAllowed.join('|'), 'i');
  if (re.test(navigator.userAgent)) {
    whitelistOn();
    return true;
  }

  whitelistOff();
  return false;
};

[josephcsible]

Adding this rule fixes it for me: nytimes.com##+js(set-constant.js, navigator.storage.estimate, undefined)

[okiehsch]

Thanks, that works on my end.

[krystian3w]

https://github.com/AdguardTeam/AdguardFilters/issues/39685

[Yuki2718]

It seems only one of the two rules is enough to avoid detection on each platform like https://github.com/uBlockOrigin/uAssets/issues/7058
i.e.
##+js(set-constant.js, navigator.storage.estimate, undefined)
for Chromium and
nytimes.com##+js(set-constant.js, webkitRequestFileSystem, noopFunc)
for Firefox.

[Yuki2718]

Not sure if it should be addressed in uBlock Annoyances but if you access the site on Android (with or without the Annoyances filter enabled) you'll see leftover of modal popup only for several seconds.

SS was taken on PC for convenience.
nytimes.com##.ReactModalPortal

[krystian3w]

Locked scroll?

PS. trojmiasto.pl also detect private mode in browser

[Yuki2718]

Locked scroll?

PS. trojmiasto.pl also detect private mode in browser

Yeah, scroll is locked but I'm not sure whether a style rule is necessary as the scroll recovers just 2-3 seconds after page load.
Why don't you create a new issue for that?

[krystian3w]

Because you haven't separated into a new one with a locked-up pop-up rewind (scroll).

And this thread could be changed to "pages with private mode detection".

[Yuki2718]

Ah, I'm still not sure if that leftover should be addressed in uBlock Annoyances as it doesn't require uBO syntax. Given @okiehsch didn't add this I guess it's not.

Personally I agree, but since we don't have such thread as "pages with right-click disabled", maybe we're still required to report them one by one with each URL.

[okiehsch]

Given @okiehsch didn't add this I guess it's not.

I can't reproduce.

[Yuki2718]

Given @okiehsch didn't add this I guess it's not.

I can't reproduce.

Okay, will make a new issue. I'm currently a bit busy so may be late.

[llacb47]

Is this the modal? Reproduced with Firefox emulating Android. Just like you said (only for several seconds) the modal goes away after a few seconds.

[llacb47]

This filter gets rid of that: nytimes.com##.ReactModalPortal:has(.welcomeAd)

I assume this fits the criteria for annoyances.

Here's the full DOM structure of the popup, it's hard to look at since it is removed from the DOM after a few seconds: https://pastebin.com/hFG45Jqt

[Yuki2718]

@llacb47 ty for confirmation and more, yeah, that one! I found I see that one on default settings rather than the leftover I posted on medium mode settings.

SS