When bundling I get this error:
WARNING in ./node_modules/puppeteer-extra/dist/index.esm.js 294:22-35
Critical dependency: the request of a dependency is an expression
@ ./src/processor.js
@ ./src/index.js
@ multi @babel/polyfill ./src/index.js
And then at run-time:
A plugin listed 'puppeteer-extra-plugin-stealth/evasions/chrome.runtime' as dependency,
which is currently missing. Please install it:yarn add puppeteer-extra-plugin-stealth Note: You don't need to require the plugin yourself, unless you want to modify it's default settings.Error: Cannot find module 'puppeteer-extra-plugin-stealth/evasions/chrome.runtime'
Of course, puppeteer-extra-plugin-stealth is already in the package.json.
Work-around is to import and apply the plugins manually:
import puppeteerVanilla from 'puppeteer';
import { addExtra } from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
import AcceptLanguagePlugin from 'puppeteer-extra-plugin-stealth/evasions/accept-language';
import ChromeRuntimePlugin from 'puppeteer-extra-plugin-stealth/evasions/chrome.runtime';
import ConsoleDebugPlugin from 'puppeteer-extra-plugin-stealth/evasions/console.debug';
import IFrameContentWindowPlugin from 'puppeteer-extra-plugin-stealth/evasions/iframe.contentWindow';
import MediaCodecsPlugin from 'puppeteer-extra-plugin-stealth/evasions/media.codecs';
import NavigatorLanguagesPlugin from 'puppeteer-extra-plugin-stealth/evasions/navigator.languages';
import NavigatorPermissionsPlugin from 'puppeteer-extra-plugin-stealth/evasions/navigator.permissions';
import NavigatorPlugins from 'puppeteer-extra-plugin-stealth/evasions/navigator.plugins';
import WebdriverPlugin from 'puppeteer-extra-plugin-stealth/evasions/navigator.webdriver';
import UserAgentPlugin from 'puppeteer-extra-plugin-stealth/evasions/user-agent';
import WebglVendorPlugin from 'puppeteer-extra-plugin-stealth/evasions/webgl.vendor';
import WindowOuterDimensionsPlugin from 'puppeteer-extra-plugin-stealth/evasions/window.outerdimensions';
async () => {
const puppeteer = addExtra(puppeteerVanilla);
const plugins = [
StealthPlugin(),
AcceptLanguagePlugin(),
ChromeRuntimePlugin(),
ConsoleDebugPlugin(),
IFrameContentWindowPlugin(),
MediaCodecsPlugin(),
NavigatorLanguagesPlugin(),
NavigatorPermissionsPlugin(),
NavigatorPlugins(),
WebdriverPlugin(),
UserAgentPlugin(),
WebglVendorPlugin(),
WindowOuterDimensionsPlugin(),
];
const browser = await puppeteer.launch();
for (const plugin of plugins) {
await plugin.onBrowser(browser);
}
const [ page ] = await browser.pages();
for (const plugin of plugins) {
await plugin.onPageCreated(page);
}
// ...
};
Hmm, interesting. May I ask why you opted to use Webpack for a NodeJS project?
If you could provide a minimal webpack based project with that issue that'd be great, as I then can take a quick look and see how to best fix this. :)
I use Webpack with Node because it's a simpler way to use Babel, bundle node_modules and minify code.
webpack.config.js:
const path = require('path');
const TerserPlugin = require('terser-webpack-plugin');
const { env } = process;
const isProduction = env['NODE_ENV'] === 'production';
const mode = isProduction ?
'production' :
'development';
console.log({ mode });
module.exports = {
entry: [
'@babel/polyfill',
'./src/index.js',
],
target: 'node',
devtool: isProduction ? false : 'source-map',
mode,
output: {
path: path.join(__dirname, 'build'),
filename: 'index.js'
},
module: {
rules: [
{
test: /\.m?js$/,
exclude: /(node_modules|bower_components)/,
use: {
loader: 'babel-loader',
options: {
babelrc: true,
},
},
},
{
test: /\.js$/,
loader: 'unlazy-loader'
}
],
},
optimization: {
minimize: isProduction,
minimizer: [
new TerserPlugin(),
],
},
resolve: {
alias: {
'pg-native': path.resolve(__dirname, 'aliases/pg-native'),
},
},
};
Thanks for providing the config, I'll look into it when I find time.
I use Webpack with Node because it's a simpler way to use Babel, bundle node_modules and minify code.
So it's to use new ES language features? Bundling and minifying shouldn't matter on backend code (hence my question). I personally used Babel with NodeJS back in the day to be able to use ES6 imports but eventually stopped doing that as I noticed issues with that and figured it's not worth it and causes more problems than it does good :)
Yes I use the latest ES6 features.
I also use loaders for other languages (such as Fable) and data (such as JSON, CSS).
Bundling everything can also be useful for targets like AWS lambda.
I think as rule-of-thumb, only using ES6 import / export will always give a library that works with Webpack, Rollup, etc.
Got it :) Anyway, puppeteer-extra should still be able to work with Webpack. It might be that the internal dependency system is just not aware of the bundler already taking care of the dependencies and a flag to disable the internal dependency resolution is sufficient.
Couple of avenues I'll explore to fix this:
disableInternalDependencyResolution: trueesm rollup build to disable the internal dependency management (not sure about that, as regular NodeJS will soon(?) use esm exports as well?)What is the purpose of the internal dependency management?
Could a simpler design work like this?
import puppeteerExtra from 'puppeteer-extra';
import MyPlugin from 'my-plugin';
const puppeteer = pupeteerExtra({
plugins: [
MyPlugin(),
],
});
async () => {
const browser = await puppeteer.launch();
// etc...
};
The above already exists (with puppeteer.use()) :)
The idea behind the dependency plugin system was to make it easy to re-use plugins within plugins. E.g. the stealth plugin needs to anonymize the UA and instead of copy pasting that code we just load the anonymize-ua plugin internally as a dependency.
The code for that is here: https://github.com/berstend/puppeteer-extra/blob/master/packages/puppeteer-extra/src/index.ts#L326-L334
Thinking of it now (I wrote this in the very first version) I think we don't need to handle that within puppeteer-extra, but can just use the native package.json methods to declare them.
What I currently don't understand: This should still work with Webpack regardless, as they should transform the require() statements to point at bundled resources instead. I need to take a closer look to see what's going on there.
So yeah, it's just webpack being bad at dynamic imports.
This fix should work:
https://github.com/webpack/webpack/issues/4175#issuecomment-450746682
So basically adding this to the webpack module.rules (untested):
{
// regex for the files that are problematic
test: \.\/node_modules\/puppeteer-extra\/dist\/index\.esm\.js,
loader: 'string-replace-loader',
options: {
// match a require function call where the argument isn't a string
// also capture the first character of the args so we can ignore it later
search: 'require[(]([^\'"])',
// replace the 'require(' with a '__non_webpack_require__(', meaning it will require the files at runtime
// $1 grabs the first capture group from the regex, the one character we matched and don't want to lose
replace: '__non_webpack_require__($1',
flags: 'g'
}
}
also you will need to install string-replace-loader: https://github.com/Va1/string-replace-loader
The above already exists (with puppeteer.use()) :)
I meant this usage but with internal dependency management removed entirely.
The idea behind the dependency plugin system was to make it easy to re-use plugins within plugins. E.g. the stealth plugin needs to anonymize the UA and instead of copy pasting that code we just load the anonymize-ua plugin internally as a dependency.
The code for that is here: https://github.com/berstend/puppeteer-extra/blob/master/packages/puppeteer-extra/src/index.ts#L326-L334
Thinking of it now (I wrote this in the very first version) I think we don't need to handle that within puppeteer-extra, but can just use the native package.json methods to declare them.
So the plugins would declare the other plugins that they depend on using package.json dependencies? Sounds like the best solution to me.
What I currently don't understand: This should still work with Webpack regardless, as they should transform the require() statements to point at bundled resources instead. I need to take a closer look to see what's going on there.
string-replace-loader looks like a good work-around. Thanks!
string-replace-loader looks like a good work-around. Thanks!
Would be great to hear if it fixes the webpack issue, then I can add it to the documentation :)
So the plugins would declare the other plugins that they depend on using package.json dependencies? Sounds like the best solution to me.
Unfortunately that won't fix the issue with dynamic imports + webpack, we still need to do dynamic require() under the hood (which works fine, just not with webpack). :)
we still need to do dynamic require() under the hood
Why is this?
we still need to do dynamic require() under the hood
Why is this?
It's always good to question assumptions, I'm a big believer in that :)
Let's take the stealth plugin as an example: it comes with a set of "evasions" (which are just regular plugins) and acts as an "umbrella" plugin, so the user doesn't need to add the specific evasions one-by-one.
One feature is that the user can add or remove evasions on that list, before puppeteer-extra will require these files (and thereby code mods).
I'm not aware of a way to accomplish that without dynamic require(). :)
Assuming the fix mentioned in https://github.com/berstend/puppeteer-extra/issues/93#issuecomment-564185702 works I'm gonna demote this issue to "nice to have in a future version". :)
we still need to do dynamic require() under the hood
Why is this?
It's always good to question assumptions, I'm a big believer in that :)
Let's take the
stealthplugin as an example: it comes with a set of "evasions" (which are just regular plugins) and acts as an "umbrella" plugin, so the user doesn't need to add the specific evasions one-by-one.One feature is that the user can add or remove evasions on that list, before
puppeteer-extrawill require these files (and thereby code mods).I'm not aware of a way to accomplish that without dynamic
require(). :)
The stealth plugin could require all evasions statically. This would cover most use-cases.
Users who need something more custom could manually import the ones that they need. This wouldn't be much code and the full import list could be copied from the stealth plugin then edited.
I got this working locally but when I used webpack to bundle it and send it over to aws lambda, this line
StealthPlugin();
results to the error below. Adding "kind-of": "^6.0.2" to the project's package.json does not resolve the problem.
Error: Cannot find module 'kind-of'
at a (/var/task/index.js:145:1835)
at Function.o [as typeOf] (/var/task/index.js:145:1440)
at i (/var/task/index.js:145:854)
at e.exports (/var/task/index.js:145:371)
at new n (/var/task/index.js:139:191)
at new i (/var/task/index.js:133:12241)
at e.exports (/var/task/index.js:133:12910)
at Object.startBrowser (/var/task/index.js:127:99655)
at Runtime.t.handler (/var/task/index.js:127:86330)
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
code: 'MODULE_NOT_FOUND'
}
I got this working locally but when I used webpack to bundle it and send it over to aws lambda, this line
StealthPlugin();results to the error below. Adding "kind-of": "^6.0.2" to the project's package.json does not resolve the problem.
Error: Cannot find module 'kind-of'
at a (/var/task/index.js:145:1835)
at Function.o [as typeOf] (/var/task/index.js:145:1440)
at i (/var/task/index.js:145:854)
at e.exports (/var/task/index.js:145:371)
at new n (/var/task/index.js:139:191)
at new i (/var/task/index.js:133:12241)
at e.exports (/var/task/index.js:133:12910)
at Object.startBrowser (/var/task/index.js:127:99655)
at Runtime.t.handler (/var/task/index.js:127:86330)
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
code: 'MODULE_NOT_FOUND'
}
I had the same issue. unlazy-loader solved this for me...
@levz0r Interesting. Would you mind providing a full example using unlazy-loader for others? :)
@levz0r Interesting. Would you mind providing a full example using
unlazy-loaderfor others? :)
Hey, sorry for the long delay...
Nothing special actually... Just add
rules: [
{
test: /\.js$/,
use: "unlazy-loader"
}
]
To webpack.config.js. That's it.
Hope it helps.
I'm getting a similar problem to this with serverless-bundle. Any ideas how to work around this with serverless-bundle?
Hello,
Any update on this issue ?
I tried all the workaround combinations here (painfully).
One did work but seems very hacky :
Import files like described here : https://github.com/berstend/puppeteer-extra/issues/93#issuecomment-563159177
But i had to comment out AcceptLanguagePlugin and UserAgentPlugin (maybe they don't exist anymore ?)
// import AcceptLanguagePlugin from 'puppeteer-extra-plugin-stealth/evasions/accept-language';
// import UserAgentPlugin from 'puppeteer-extra-plugin-stealth/evasions/user-agent';
But I also had to use the solution described here : https://github.com/berstend/puppeteer-extra/issues/93#issuecomment-575600329
When building I'm getting these warnings :
warning in ./node_modules/puppeteer-extra/dist/index.esm.js
Critical dependency: the request of a dependency is an expression
warning in ./node_modules/puppeteer-extra/dist/index.esm.js
Module not found: Error: Can't resolve 'puppeteer-core' in '...\node_modules\puppeteer-extra\dist'
But puppeteer opens and seems to work.
Unfortunately this more elegant solution didn't work :
https://github.com/berstend/puppeteer-extra/issues/93#issuecomment-564185702
Would be nice to clarify the best way to make stealth-plugin work with webpack, and maybe document it.
Thanks
Same issue over here.
@celicoo did you try this? https://github.com/berstend/puppeteer-extra/issues/93#issuecomment-564185702
@celicoo for me I had to add both solutions mentioned here. In second rule I had to update regex in test:
{
test: /\.js$/,
use: "unlazy-loader"
},
{
// regex for the files that are problematic
test: /node_modules\/puppeteer-extra\/dist\/index\.esm\.js/,
loader: 'string-replace-loader',
options: {
// match a require function call where the argument isn't a string
// also capture the first character of the args so we can ignore it later
search: 'require[(]([^\'"])',
// replace the 'require(' with a '__non_webpack_require__(', meaning it will require the files at runtime
// $1 grabs the first capture group from the regex, the one character we matched and don't want to lose
replace: '__non_webpack_require__($1',
flags: 'g'
}
},
I got this working locally but when I used webpack to bundle it and send it over to aws lambda, this line
StealthPlugin();
results to the error below. Adding "kind-of": "^6.0.2" to the project's package.json does not resolve the problem.
Error: Cannot find module 'kind-of'
at a (/var/task/index.js:145:1835)
at Function.o [as typeOf] (/var/task/index.js:145:1440)
at i (/var/task/index.js:145:854)
at e.exports (/var/task/index.js:145:371)
at new n (/var/task/index.js:139:191)
at new i (/var/task/index.js:133:12241)
at e.exports (/var/task/index.js:133:12910)
at Object.startBrowser (/var/task/index.js:127:99655)
at Runtime.t.handler (/var/task/index.js:127:86330)
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
code: 'MODULE_NOT_FOUND'
}I had the same issue. unlazy-loader solved this for me...
Doesn't work for me.
Any fix for this?
On a more general note:
@berstend I am getting the error:
Error: Cannot find module 'kind-of'
at webpackEmptyContext (D:\node_projects\vueapp\dist_electron\index.js:34338:10)
at Function.getter [as typeOf] (D:\node_projects\vueapp\dist_electron\index.js:91020:29)
at cloneDeep (D:\node_projects\vueapp\dist_electron\index.js:34370:17)
at mergeDeep (D:\node_projects\vueapp\dist_electron\index.js:100090:16)
at new PuppeteerExtraPlugin (D:\node_projects\vueapp\dist_electron\index.js:159981:22)
at new StealthPlugin (D:\node_projects\vueapp\dist_electron\index.js:159809:5)
at defaultExport (D:\node_projects\vueapp\dist_electron\index.js:159905:31)
at Module../src/main/gen.js (D:\node_projects\vueapp\dist_electron\index.js:221678:15)
at __webpack_require__ (D:\node_projects\vueapp\dist_electron\index.js:20:30)
at Module../src/main/main.js (D:\node_projects\vueapp\dist_electron\index.js:221764:62)
This is my vue.config.js
module.exports = {
configureWebpack: {
// Configuration applied to all builds
target: "electron-renderer",
devtool: 'source-map',
module:{
rules:[
{
test: /\.js$/,
use: "unlazy-loader"
},
{
// regex for the files that are problematic
test: /node_modules\/puppeteer-extra\/dist\/index\.esm\.js/,
loader: 'string-replace-loader',
options: {
// match a require function call where the argument isn't a string
// also capture the first character of the args so we can ignore it later
search: 'require[(]([^\'"])',
// replace the 'require(' with a '__non_webpack_require__(', meaning it will require the files at runtime
// $1 grabs the first capture group from the regex, the one character we matched and don't want to lose
replace: '__non_webpack_require__($1',
flags: 'g'
}
}
]
}
},
pluginOptions: {
electronBuilder: {
builderOptions: {
// options placed here will be merged with default configuration and passed to electron-builder
}
}
}
}
babel.config.js:
module.exports = {
presets: [
'@vue/cli-plugin-babel/preset'
]
}
This only happens if i use StealthPlugin() code
@Nisthar did you try adding this:
import kindOf from 'kind-of';
Tried it just now. doesn't make any difference unfortunately.
Tried everything on the list. Using Next.js api routes https://nextjs.org/docs/api-routes/introduction
Using unlazy-loader and/or string-replace-loader did not work.
However, the plugins array workaround was what I needed. Updated 19 October 2020:
Handler.ts
// handler.ts
import { NextApiRequest, NextApiResponse } from "next";
import chromium from "chrome-aws-lambda";
import { addExtra } from "puppeteer-extra";
import AdblockerPlugin from "puppeteer-extra-plugin-adblocker";
import StealthPlugin from "puppeteer-extra-plugin-stealth";
import ChromeAppPlugin from "puppeteer-extra-plugin-stealth/evasions/chrome.app";
import ChromeCsiPlugin from "puppeteer-extra-plugin-stealth/evasions/chrome.csi";
import ChromeLoadTimes from "puppeteer-extra-plugin-stealth/evasions/chrome.loadTimes";
import ChromeRuntimePlugin from "puppeteer-extra-plugin-stealth/evasions/chrome.runtime";
import IFrameContentWindowPlugin from "puppeteer-extra-plugin-stealth/evasions/iframe.contentWindow";
import MediaCodecsPlugin from "puppeteer-extra-plugin-stealth/evasions/media.codecs";
import NavigatorLanguagesPlugin from "puppeteer-extra-plugin-stealth/evasions/navigator.languages";
import NavigatorPermissionsPlugin from "puppeteer-extra-plugin-stealth/evasions/navigator.permissions";
import NavigatorPlugins from "puppeteer-extra-plugin-stealth/evasions/navigator.plugins";
import NavigatorVendor from "puppeteer-extra-plugin-stealth/evasions/navigator.vendor";
import NavigatorWebdriver from "puppeteer-extra-plugin-stealth/evasions/navigator.webdriver";
import SourceUrlPlugin from "puppeteer-extra-plugin-stealth/evasions/sourceurl";
import UserAgentOverridePlugin from "puppeteer-extra-plugin-stealth/evasions/user-agent-override";
import WebglVendorPlugin from "puppeteer-extra-plugin-stealth/evasions/webgl.vendor";
import WindowOuterDimensionsPlugin from "puppeteer-extra-plugin-stealth/evasions/window.outerdimensions";
export async function getOptions() {
const isDev = !process.env.AWS_REGION;
const exePath =
process.platform === "win32"
? "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe"
: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
let options;
if (isDev) {
options = {
args: [],
executablePath: exePath,
headless: true,
};
} else {
options = {
args: chromium.args,
defaultViewport: chromium.defaultViewport,
executablePath: await chromium.executablePath,
headless: chromium.headless,
};
}
return options;
}
const puppeteerExtra = addExtra(chromium.puppeteer);
const plugins = [
AdblockerPlugin(),
StealthPlugin(),
ChromeAppPlugin(),
ChromeCsiPlugin(),
ChromeLoadTimes(),
ChromeRuntimePlugin(),
IFrameContentWindowPlugin(),
MediaCodecsPlugin(),
NavigatorLanguagesPlugin(),
NavigatorPermissionsPlugin(),
NavigatorPlugins(),
NavigatorVendor(),
NavigatorWebdriver(),
SourceUrlPlugin(),
UserAgentOverridePlugin(),
WebglVendorPlugin(),
WindowOuterDimensionsPlugin(),
];
export async function handler(req: NextApiRequest, res: NextApiResponse) {
const options = await getOptions();
const browser = await puppeteerExtra.launch(options);
for (const plugin of plugins) {
// @ts-ignore: Expected 2 arguments, but got 1.
await plugin.onBrowser(browser);
}
// https://github.com/berstend/puppeteer-extra/issues/88
const page = await browser.newPage();
for (const plugin of plugins) {
await plugin.onPageCreated(page);
}
await page.goto("https://bot.sannysoft.com/");
const file = await page.screenshot({
type: "jpeg",
quality: 90,
fullPage: true,
});
await page.close();
await browser.close();
const filename = 'screenshot.jpg';
// https://stackoverflow.com/a/41996347
res.statusCode = 200;
res.setHeader("Content-Disposition", "attachment; filename=" + filename);
res.setHeader("Content-Transfer-Encoding", "binary");
res.setHeader("Content-Type", "application/octet-stream");
return res.send(file);
}
// TODO: cache browser instead of launching and closing per request
@jstormail2 thanks for sharing your workaround 馃憤
I must admit listing the plugins manually isn't the most prettiest thing in the world but great if it fixes the webpack issue. :-)
cc @Nisthar
I should note that while working on the playwright-supporting rewrite I realized that for improved type-safety it would make sense to ditch the dynamic custom imports, so I'm most likely doing that. This would mean that this webpack issue should evaporate in the next major release. 馃槃
@berstend None of the solutions above worked for me. Do you have any update on the rework ?
@berstend None of the solutions above worked for me. Do you have any update on the rework ?
Did you try https://github.com/berstend/puppeteer-extra/issues/93#issuecomment-563159177 ? It is highly likely to work because it is pure ES6 with static imports.
thanks @njlr pretty sure your workaround would still work. @Seblor just verify that you cover all evasion imports (check the index.js file in the stealth repo for all currently loaded evasions).
After many hours, I managed to make it work, albeit only halfway.
I had to use @njlr's linked solution in addition to unlazy-loader.
However even though it works nicely on bot.sannysoft.com (it even has one more passing criteria that my browser), it still gets detected by reCaptcha. Possibly something to do with it being in an iframe ?
Interestingly enough, removing puppeteer-extra (and all plugins) and simply adding the --disable-blink-features=AutomationControlled flag (and no other "stealth" mechanisms) is enough to fool reCaptcha. Which is weird since I saw in an other issue that this flag was supposed to be included in the puppeteer-extra stealth plugin.
Thank you for the quick replies, though.
@Seblor this is out of the scope of this webpack issue but the iframe.contentWindow evasion is known to cause issues, more context and workaround here: https://github.com/berstend/puppeteer-extra/issues/137#issuecomment-685680787
Which is weird since I saw in an other issue that this flag was supposed to be included in the puppeteer-extra stealth plugin.
It is, as you can see in the code of the navigator.webdriver evasion. Make sure you use the latest versions of everything.
Most helpful comment
Work-around is to import and apply the plugins manually: