Puppeteer-extra: Page redirect/ different render when using puppeteer-extra vs without

Created on 17 May 2020 · 6Comments · Source: berstend/puppeteer-extra

Hello, I am really grateful for the plugin, but have couple of issues using it.
I am testing it with couple websites: https://www.whatismyip.com/ , https://whatismyipaddress.com/

The code I am using

// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality
const puppeteer = require('puppeteer-extra')

// add stealth plugin and use defaults (all evasion techniques)
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())

// puppeteer usage as normal
puppeteer.launch({ headless: false }).then(async browser => {
  console.log('Running tests..')
  const page = await browser.newPage()
  await page.setViewport({ width: 1366, height: 768});
  await page.goto('https://whatismyipaddress.com/')
  await page.waitFor(5000)
  await page.screenshot({ path: 'testresult.png', fullPage: true })
  await browser.close()
  console.log(`All done, check the screenshot. ✨`)
})

Without stealth :
testresult

With Stealth :
testresultWithExtra

It seems that I get redirect to page with AdSense ad. When I comment
//puppeteer.use(StealthPlugin())
It is working normally.
Quite strange, can I get a hand on this issue, thank you

bug stealth ㊙️ prs-welcome

Source

Bobchev

👍2

Most helpful comment

I solved the issue in my case by editing the evasions/iframe.contentWindow/index.js file:

          const contentWindowProxy = {
            get(target, key) {
              // Now to the interesting part:
              // We actually make this thing behave like a regular iframe window,
              // by intercepting calls to e.g. `.self` and redirect it to the correct thing. :)
              // That makes it possible for these assertions to be correct:
              // iframe.contentWindow.self === window.top // must be false
              if (key === 'self') {
                return this
              }
              // iframe.contentWindow.frameElement === iframe // must be true
              if (key === 'frameElement') {
                return iframe
              }
+++              if (key === 'document') {
+++                return iframe.contentDocument;
+++              }
              return Reflect.get(target, key)
            }
          }

Phyks on 17 Jun 2020

👍2

All 6 comments

I have the same issue on a different site which is resolved either by not using the stealth plugin or by using the adblocker plugin in conjuction. However, with the adblocker plugin enabled I run into #90/#91 and the suggestions on those issues (using version 2.4.5 or disabled accept-language evasion) do not resolve the stream of error messages.

cam-perry on 22 May 2020

@Bobchev I experienced the same problem. I removed the evasions one at a time and found that iframe.contentWindow causes the problem. Namely it finds the first iframe in the document and navigates to it. Since ads are often served through iframes, you'll often see ads. If you launch with headless: false you can watch it happen.

So you'll need to disable the evasion. I think there is a way you can disable a single evasion, but I did this:

puppeteer.use(stealthPlugin({
    enabledEvasions: [
        'chrome.runtime',
        'console.debug',
        // as of 6/11/2020 the iframe.contentWindow evasion causes navigation to the first iframe in the document
        // 'iframe.contentWindow',
        'media.codecs',
        'navigator.languages',
        'navigator.permissions',
        'navigator.plugins',
        'navigator.webdriver',
        'user-agent-override',
        'webgl.vendor',
        'window.outerdimensions'
    ],
}));

I figured opting into evasions is better anyway because I'd want to retest my scripts if any new evasions are added in future versions of the stealth plugin.

The code for the iframe.contentWindow evasion is pretty complex so I don't know how to fix it

kensnyder on 12 Jun 2020

I solved the issue in my case by editing the evasions/iframe.contentWindow/index.js file:

          const contentWindowProxy = {
            get(target, key) {
              // Now to the interesting part:
              // We actually make this thing behave like a regular iframe window,
              // by intercepting calls to e.g. `.self` and redirect it to the correct thing. :)
              // That makes it possible for these assertions to be correct:
              // iframe.contentWindow.self === window.top // must be false
              if (key === 'self') {
                return this
              }
              // iframe.contentWindow.frameElement === iframe // must be true
              if (key === 'frameElement') {
                return iframe
              }
+++              if (key === 'document') {
+++                return iframe.contentDocument;
+++              }
              return Reflect.get(target, key)
            }
          }

Phyks on 17 Jun 2020

👍2

Confirmed that without the above change it performs a redirect and with the fix it seems to behave as expected.

Just unsure if this change will cause any side effects?