Puppeteer-extra: Puppeteer detected via variable name

[prescience-data]

This very slick detection just popped up, and my tests with both base Puppeteer and Puppeter-Extra with Extra-Stealth plugin are failing it.

https://github.com/digitalhurricane-io/puppeteer-detection-100-percent

Would be great to get that variable renamed to something opaque...

[StevenVeshkini]

Yikes...how would you do that though? Wouldn't you need to recompile puppeteer from source?

[prescience-data]

The only workarounds I have so far are:
a) forking original puppeteer, or;
b) running something like this:

const files = [
    '../../node_modules/puppeteer/lib/ExecutionContext.js',
    '../../node_modules/puppeteer/node6/lib/ExecutionContext.js',
];

const fs = require('fs');

files.forEach(file => {
    fs.readFile(file, 'utf8', function (err, data) {
        if (err) {
            return console.log(err);
        }
        let result = data.replace(/__puppeteer_evaluation_script__/g, '__jquery__');

        fs.writeFile(file, result, 'utf8', function (err) {
            if (err) {
                return console.log(err);
            }
        });
    });
});

[m4eba]

You could proxy the websocket connection that puppeteer uses to control browser and modify the sourceURL there.
I made proof of concept, it's ugly but works:
https://github.com/m4eba/puppeteer-detection-100-percent

But that is out of scope for this plugin, the proxy needs to go into the core library and then activated with the requirements variable, maybe?

[prescience-data]

Hopefully if this gets implemented it may fix it:

https://github.com/puppeteer/puppeteer/issues/2671

[kensnyder]

Adding this bit of trickery avoids detection:

await page.evaluateOnNewDocument(() => {
  const errors = { Error, EvalError, RangeError, ReferenceError, SyntaxError, TypeError, URIError };
  for (const name in errors) {
    globalThis[name] = (function(NativeError) {
      return function(message) {
        const err = new NativeError(message);
        const stub = {
          message: err.message,
          name: err.name,
          toString: () => err.toString(),
          get stack() {
            const lines = err.stack.split('\n');
            lines.splice(1, 1); // remove anonymous function above
            lines.pop(); // remove puppeteer line
            return lines.join('\n');
          },
        };
        if (this === globalThis) {
          // called as function, not constructor
          stub.__proto__ = NativeError;
          return stub;
        }
        Object.assign(this, stub);
        this.__proto__ = NativeError;
      };
    })(errors[name]);
  }
});

[a10kiloham]

This is a good addition to the extra stealth plug in. Has anyone done a PR yet? Might try and find time this week otherwise

[digitalhurricane-io]

@kensnyder That is some nice trickery!

One suggestion. You're missing part of the stack trace with this code though, as written. That would be pretty easy to detect.

I would suggest adding these 2 lines after "'lines.pop(); // remove puppeteer line"

const inRange = (max, min) => (Math.round(Math.random() * (max - min) + min)).toString();
lines.push(`    at jQuery.js:${inRange(5, 200)}:${inRange(1, 80)}`);

[berstend]

What's the latest on this? I'm a bit worried that simply replacing __puppeteer_evaluation_script__ with something else doesn't fix the issue and the other side will simply add whatever string we use/generate to their blacklist. :-)

Has someone checked out what vanilla Chrome is using in their stack traces so we can emulate this properly?

An expected vs actual comparison would be very useful here.

[digitalhurricane-io]

@berstend Every script that runs on the page has a name. I don't think you can really blacklist something like "main.js" as that's pretty generic. However there may be better solutions that forking the library.

[prescience-data]

What's the latest on this? I'm a bit worried that simply replacing __puppeteer_evaluation_script__ with something else doesn't fix the issue and the other side will simply add whatever string we use/generate to their blacklist. :-)

Has someone checked out what vanilla Chrome is using in their stack traces so we can emulate this properly?

An expected vs actual comparison would be very useful here.

After working / researching this over the last month it's actually a deeper problem than just the string. It's the ability of the detection script to monitor execution.

I've got a solution working on Puppeteer 1.19 as mentioned above that rewrites some core Puppeteer logic to force all Puppeteer execution to run within it's own context, meaning even if the detection script is attempting to override functions, they can't access the isolated world, so it's not even able to be aware of the script.

https://github.com/berstend/puppeteer-extra/issues/224

The only way I figure they could detect execution is to watch the DOM for unexpected changes, but even then, they would get a ton of false positives from things like password managers etc writing DOM for functionality.

Thoughts?

[berstend]

@berstend Every script that runs on the page has a name. I don't think you can really blacklist something like "main.js" as that's pretty generic. However there may be better solutions that forking the library.

This is what I was hinting at, we should be able to derive the proper "originator" script dynamically and use that instead of something hardcoded. But this is all still too theoretical, a proper comparison of vanilla vs. puppeteer behavior would add a lot to the discussion.

edit: For better or worse puppeteer-extra seems to be used as a benchmark by the other teams, similar to how we use their various bot-detection test pages. We can't really hardcode anything and expect to get away with it. E.g. if my site isn't using jQuery.js that makes it very simple to detect the presence of a headless browser in disguise. :-)

edit2: Out of curiosity, has anyone checked how Playwright behaves in these scenarios?

[berstend]

Fixed in #273

[berstend]

Fix published in [email protected]