Puppeteer-extra: [Bug] bet365.com stopped working 2 days ago

[apulidoc]

// edit by @berstend:

See here for a workaround if you're affected by this: https://github.com/berstend/puppeteer-extra/issues/399#issuecomment-759387597
and another workaround: https://github.com/berstend/puppeteer-extra/issues/399#issuecomment-770155239

Original issue:

Describe the bug

• Im trying to scrap bet365 website
• Expect open the website bet365.com without problems.
• I was using puppeteer-extra-plugin-stealth for this purpose but is not working anymore from 2 days ago.

Code Snippet

const fs = require('fs')
const json = require('JSON')
const puppeteer = require('puppeteer-extra')
const { Bot } = require('tgapi')
const mysql = require('mysql')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')

var dbConfig ={
  host     : 'localhost',
  port     : '3307',
  user     : 'xxxx',
  password : 'xxxx',
  database : 'mydb'
}

let connection = mysql.createPool(dbConfig);
connection.setMaxListeners(20)

puppeteer.use(StealthPlugin())

//Ejecucion de Puppeteer
puppeteer.launch({ args: ['--start-maximized'], headless: false }).then(async browser => {
  console.log('Ejecutando script...')
  const page = await browser.newPage()

  await page.goto('https://www.bet365.es/#/IP/B1');

  etc...

Versions

System:
OS: Windows 10 10.0.18363
CPU: (8) x64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Memory: 8.31 GB / 15.86 GB
Binaries:
Node: 12.17.0 - C:\Program Files\nodejs\node.EXE
npm: 6.14.10 - ~\AppData\Roaming\npm\npm.CMD

[Skllipzgn]

I observe the same thing. There is such an error in the console

[hrehman200]

Same thing here ... can't load bet365.com

[prescience-data]

Which Chrome version are you using?

[hrehman200]

Chromium the one which is installed alongside puppeteer.

[apulidoc]

Which Chrome version are you using?

Chromium Versión 88.0.4298.0

Also i downgraded to 86.0.4240.0 but still is not working

[apulidoc]

Removing stealth plugin and adding '--disable-blink-features=AutomationControlled' its working again

[hrehman200]

Thanks @apulidoc, its working.

[Niek]

The bet365 site has very strict IP checks. If you open the site with an:

• IP from a country that is not supported: you get this error - https://www.bet365.com/Members/helpers/error.aspx
• IP from a datacenter range: they load a JS script that is doing a bunch of calls to localhost which are blocked by Chrome. I tested a non-puppeteer Chrome and Safari but I have the same issue there.
• IP from a residential range: it hangs loading when using puppeteer, no matter if stealth is enabled or not.

Some more observations:

• Using the bundled Chromium from puppeteer directly works fine:

./node_modules/puppeteer/.local-chromium/mac-818858/chrome-mac/Chromium.app/Contents/MacOS/Chromium https://www.bet365.com/ --no-first-run --user-data-dir=$(mktemp)

• Connecting puppeteer to an existing running browser (started with --remote-debugging-port=9992) with puppeteer.connect({browserURL: 'http://localhost:9992'}): it's again stuck on loading the site. This means they detect something in the way the page is opened.
• If you open an existing page (so not opened by puppeteer) in a browser opened with stealth (to avoid the webdriver property) it works fine - obviously page-level stealth evasions are not applied but it's working fine. Here's working sample code:

const puppeteer = require('puppeteer-extra')
puppeteer.use(require('puppeteer-extra-plugin-stealth')())

puppeteer.launch({ headless: false }).then(async browser => {
  const page = (await browser.pages())[0];

  await page.goto('https://www.bet365.com/');
});

We need to investigate what they detect exactly in newly opened pages.

[prescience-data]

We need to investigate what they detect exactly in newly opened pages.

My immediate thoughts is it must be one of the evasions leaking / mismatching if it's occurring only on newPage() and passing on bundled Chromium?

(ie something is out of alignment with real-Chrome)?

[Niek]

I'm in favor of deleting those evasions, since they do more wrong than good. I think @berstend was already planning to remove them in automation-extra, but it might be a good idea to also remove them in the existing code.

[berstend]

Seems like the issue it was targeting was solved in recent releases of pptr or it was fixed in chrome itself.. If the TS says it broke 2 days ago

Do you have any link to that?

I think we might be confusing different issues here:

• https://github.com/puppeteer/puppeteer/issues/1106

  • This is the srcdoc iframe bug that's the reason for iframe.contentWindow to exist

• https://github.com/puppeteer/puppeteer/issues/3667

  • This is another longstanding bug that affects implicitly created targets
  • Chromium Bug with some recent activity: https://bugs.chromium.org/p/chromium/issues/detail?id=1070568

[berstend]

Workaround for the time being:

const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

[berstend]

I confirmed through local testing that the bug with iframes is still present. There's potential to improve the iframe.contentWindow evasion though, so it doesn't break e.g. bet356

[kingname]

@berstend As your code is based on puppeteer, if I want to use Selenium, what is the equivalent of stealth.enabledEvasions.delete in Selenium?

[berstend]

@berstend As your code is based on puppeteer, if I want to use Selenium, what is the equivalent of stealth.enabledEvasions.delete in Selenium?

Selenium is not supported by us - stealth.enabledEvasions is a Map() which is used by the stealth plugin (.delete is just a native method of Maps). This won't help you as the puppeteer-extra as well as the stealth plugin are not written to support selenium. :-)

[berstend]

Closing for now, as iframe.contentWindow issues are quite well known by now and a workaround is mentioned above. Eventually we improve the evasion so it doesn't cause as much breakage.

[kasd7]

It seems like Bet365 has improved its detection method, and now the workaround doesn't work.

Example of use:

const puppeteer = require('puppeteer-extra')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')

const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

puppeteer.launch({ args: ['--no-sandbox','--start-maximized'], headless: false }).then(async browser => {
  const page = await browser.newPage()

  await page.goto('https://www.bet365.com/');
  await page.waitFor(50_000)
})

@berstend maybe we should reopen the issue?

[migue74]

Yup, bet365 broke puppeteer again :(

[helderppb]

Yes! It has stopped just today for me. Does anyone has a new workaround to solve it?

[apulidoc]

if anyone find a solution please post here! =)

[vladtreny]

Hello friends,

Here I've created a temporary fix for bet365

Run this and it should work

        await page.evaluateOnNewDocument(() => {
            Object.defineProperty(navigator, 'maxTouchPoints', {
                get() {
                    return 1;
                },
            });

        });

Will take a closer look on free time

[helderppb]

Do you have the complete code that you're using? I'm using the code below and it's not working.

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {
console.log('Running tests..');
const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');

});

[vladtreny]

Do you have the complete code that you're using? I'm using the code below and it's not working.

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {
console.log('Running tests..');
const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');

});

Do not forget the rest stuff

const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

[helderppb]

It’s not working for me. Can you share the code you’re using, please?

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {

console.log('Running tests..');
const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');

});

[vladtreny]

It’s not working for me. Can you share the code you’re using, please?

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {

console.log('Running tests..');
const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');

});

You need to run await page.evaluateOnNewDocument(() => { before await page.goto('https://bet365.com');

[apulidoc]

Im having troubles with this workaround...

When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website...

Anyone has this trouble?