Puppeteer-extra: Working for only first link, not for the links that are opened in new tabs (via website).

[millionwords]

Hello,

I've started using your packages, but it seems to be working on the first request only if there is any link opening in a new tab, the requests are getting bypassed.

Can you help me out with this?

Thank you!

[berstend]

Hey @millionwords could you provide a code example of the issues you have?

The following simple test case with multiple pages works as expected:

const puppeteer = require("puppeteer-extra")

const UserAgentPlugin = require("puppeteer-extra-plugin-anonymize-ua")
const uaPlugin = UserAgentPlugin({
  customFn: ua => ua.replace("Chrome", "Beer")
})

puppeteer.use(uaPlugin)

const checkPageForBeer = name => async page => {
  await page.goto("https://httpbin.org/headers", {
    waitUntil: "domcontentloaded"
  })
  const content = await page.content()
  console.log(`- ${name}: User agent includes beer:`, content.includes("Beer"))
}

puppeteer.launch({ headless: true }).then(async browser => {
  const page1 = await browser.newPage()
  await checkPageForBeer("page1")(page1)

  const page2 = await browser.newPage()
  await checkPageForBeer("page2")(page2)

  await browser.close()
})

Output:

❯❯❯ node newtab.js
- page1: User agent includes beer: true
- page2: User agent includes beer: true

Are you by chance dealing with popups? I recall that there where several issues surrounding popup handling in pptr, but it's been a while. 😄

[millionwords]

yes, It's a popup in a new tab, the headers do not work well :(

[gentlyawesome]

I have the same issue. The first page is okay with all the settings but after clicking a link and opening to a new window. it is not configured as the first page.

[berstend]

Back when I did the bulk of the code, popups where a known issue with puppeteer. A lot has happened since then and I'm gonna check if we aren't able to find a workaround here. 😄👍

[RainUUU]

I also have the same question.
like this

` const page = await browser.newPage();

await page.goto('https://example.com/');
await page.evaluate(() => {
    var a = document.createElement('a');
    a.href = 'https://example.com/?404';
    a.target = '_blank';
    a.innerHTML = 'Click me';
    a.id = 'click';
    document.body.appendChild(a);
});

await page.waitFor(1000);

await page.click('#click');`

the new page is opend by a[target='_blank']，then puppeteer-extra-plugin-stealth does not work.
so is the puppeteer's bug? has other method to hack this problem?

[felyxjet]

This is a known limitation of puppeteer https://github.com/GoogleChrome/puppeteer/issues/3667

[berstend]

I'm working on a fix for this using new CDP hooks (https://github.com/puppeteer/puppeteer/issues/3667#issuecomment-562280230).

Right now I got a PoC working which will intercept implicitly created targets but I currently need to reload the popup/page to make it continue loading (which is less than optimal). I'll continue researching this, also happy to discuss potential ways to fix that.

But all in all it should be possible to fix this behaviour in the not too distant future :)

[JazyCruice]

Hey @berstend !
You are doing a great job here
It would be great if you can share any progress connected with this issue

[jozsi]

FYI, this is on the list for Playwright. I'm eager to see how they manage to resolve it.

[arjunattam]

FYI, this is on the list for Playwright. I'm eager to see how they manage to resolve it.

I'm from the Playwright team. Playwright supports emulation properties at a browser context level, which enables all pages within the context to work with the said properties. These pages can be opened through browserContext.newPage() or as a result of page navigations.