Puppeteer: userDataDir + headless = lost authorization

[vsemozhetbyt]

• Puppeteer version: v0.12.0-alpha
• Platform / OS version: Windows 7 x64
• URLs (if applicable): any that needs authorization

1. Create a test script and an empty folder test-profile-dir:

'use strict';

const puppeteer = require('puppeteer');

(async function main() {
  try {
    const browser = await puppeteer.launch({
      headless: false,
      userDataDir: 'test-profile-dir',
    });
    const page = await browser.newPage();

    await page.goto('https://twitter.com/');

    console.log(await page.evaluate(() => document.title));
    console.log(await page.evaluate(() => document.cookie));

    // await browser.close();
  } catch (err) {
    console.error(err);
  }
})();

You will see something like this:

Twitter. It's what's happening.

personalization_id=...; guest_id=...; ct0=...

Sign in into Twitter and close the browser.

1. Run the same script second time (await browser.close(); can be uncommented). You will see something like this:

Twitter

personalization_id=...; guest_id=...; ct0=...; ads_prefs=...; remember_checked_on=...;
twid=...; tip_nightmode=...; _ga=...; _gid=...; dnt=...; lang=...

1. Run the same script, but with headless: true, The output is the same as before authorization:

Twitter. It's what's happening.

personalization_id=...; guest_id=...; ct0=...

I have tried various sites, all of them seem to lose authorization in headless mode.

Some more notes:

• response.request().headers does not contain cookies in both headless: false and headless: true modes.

• console.log(await page.cookies('https://twitter.com/')); contains many cookies in the headless: false mode. In the headless: true mode it gives an empty array [].

[chuckchamberland]

I can duplicate this on MacOS 10.10.5 using 0.11.0 and facebook.

I'm also having trouble with an internal site I'm trying to automate, but also seeing it in headful mode. I only mention it here because of the note about 'response.request().headers' not containing cookies.

[klaus82]

Hello,
made some test.
In the both cases of the test added (page and page2), into the function launch of Launcher.js the userDataDir are correct, the folder logged is the same.

[aslushnikov]

For the record: upstream bug is filed as https://crbug.com/771830

[vsemozhetbyt]

@aslushnikov Is it connected? In OP's case, two first runs are with headless: false — so the issue is not in closing with headless: true, it is in not sending cookies with headless: true that have been successfully saved by previous runs.

Did you mean https://github.com/GoogleChrome/puppeteer/issues/918 ?

[fhmd4k]

The chromium can't save and restore storage and cookies in headless mode. It's chromium bug.

[fhmd4k]

It's chromium bug. It's not puppeteer bug.
On windows, you can run fiddler capture url cookies and then run chrome:(test some have cookie site)
chrome --user-data-dir=C:\Users\xxx\Desktop\testud --headless -- "https://www.example.com"

run some times, you can find the cookie is different in every times(same userDataDir)

[aslushnikov]

This is actually not fixed by #1028.

[vsemozhetbyt]

Refs: https://github.com/GoogleChrome/puppeteer/pull/1055#issuecomment-337326682

[JoelEinbinder]

This should be fixed as of #1063. @vsemozhetbyt can you confirm? Thanks

[vsemozhetbyt]

@JoelEinbinder Unfortunately, exactly this case from OP still is not fixed: logging in in headful mode is not restored in headless mode. I did not test if cookies are stored and restored in completely headless mode — this can be fixed, but for me, this is rather a speculative use case.

[aslushnikov]

For the record: this is tracked by upstream bug https://bugs.chromium.org/p/chromium/issues/detail?id=775703

[vsemozhetbyt]

Seems to be fixed upstream.

[vsemozhetbyt]

@aslushnikov @JoelEinbinder Unfortunately, https://github.com/GoogleChrome/puppeteer/pull/1259 does not fix the case in the OP: the outputs of the examples still differ and the console.log(await page.cookies('https://twitter.com/')); still displays many cookies in the headless: false mode and much fewer cookies in the the headless: true mode.

[aslushnikov]

@vsemozhetbyt we're very sorry; it indeed doesn't work! We're on it.

[aslushnikov]

For the record: upstream patch is up for review https://chromium-review.googlesource.com/c/chromium/src/+/752743

[vsemozhetbyt]

@aslushnikov Sorry, https://github.com/GoogleChrome/puppeteer/commit/2f7c77875ec56f20b7064704d91f002115a4efef still does not fix the case in the OP, the status quo from the https://github.com/GoogleChrome/puppeteer/issues/921#issuecomment-341574397 remains.

[aslushnikov]

@vsemozhetbyt just checked, it worked for me on Mac OS X. I'll check on Windows shortly. Meanwhile, are you sure you ran the 'npm install' so that pptr downloaded a new chromium binary?

[vsemozhetbyt]

@aslushnikov Yes, I've run npm install https://github.com/GoogleChrome/puppeteer and checked the version in the .local-chromium (is 64.0.3262.0 / 514418).

[JoelEinbinder]

I can reproduce this on windows, although the test headless should be able to read cookies written by headful passes for me.

[vsemozhetbyt]

Is there any progress plan for this issue? Still not working with the last tip of the tree.

[aslushnikov]

@vsemozhetbyt we'll have it fixed, but there's no good ETA. Headless team is aware about it

[vsemozhetbyt]

@aslushnikov Thank you!

[sarfraznawaz2005]

I am also having the same issue on Windows 8. In headless mode, session/cookie info is lost.

I am using puppeteer 0.13.0, node 8.9.1 and chrome that bundles with puppeteer is 64.0.3264.0 (Developer Build) (64-bit).

[jimthedev]

@aslushnikov Thanks for keeping on this one. Is there a chrome bug I can track?

[bondansebastian]

Hello, is there any updates for this issue ?

I'm also experiencing this problem using puppeteer ^1.6.1, bundled chromium 69.0.3494.0, and Windows 10 OS

[GHXST01]

Also having this issue! Launching a new driver in headless with the same user-data-dir does not remember any of the profile details like auth, cookies etc.

[Nicolas-vrcc]

Hello, same issue :p

[nicoandmee]

https://gist.github.com/nicoandmee/e6f9a0b839f713c53ac1674537195537
Not sure if this helps but I use this & headless chrome (latest pptr)

[getuliojr]

@nicoandmee can you show a more specific example in how to use this function ?

[Ron-Burg]

I found a way to get around this issue!
giving the puppeteer.launch args for the userDataDir will keep it both on headless true/false.
for example, I had issue while using this:
const browser = await puppeteer.launch({headless: true, userDataDir: "myDataDir")
but when using like that:
const path = require('path');
const browser = await puppeteer.launch({headless: true, args: [`--user-data-dir=${path.resolve(__dirname, 'myDataDir')}`] });
It works perfectly 💯 (note that I had to use absolute path)

Please let me know if that works for you

[vsemozhetbyt]

@Ron-Burg Thank you. Currently, I failed to test due to https://github.com/GoogleChrome/puppeteer/issues/3452 and https://github.com/GoogleChrome/puppeteer/issues/3453 . Will try to come up with other sites.

[sarfraznawaz2005]

@Ron-Burg Awesome, it works, I also didn't use absolute path, works with relative path too.

[vsemozhetbyt]

Thank you! https://github.com/GoogleChrome/puppeteer/pull/3481 does fix this issue for me.

[pswzyu]

in my experiment, userDataDir only works if I use absolute path, using a relative path will only save things related to "crashpad"? into the folder, but all others are lost

I'm using windows 10
@sarfraznawaz2005 hope this can help you

[sarfraznawaz2005]

3490

@pswzyu Thanks for reply. Tried using userDataDir: "C:\\Dev\\tmp" too but in this case, response is <html><head></head><body></body></html> and screen generated is white empty one.

[vsemozhetbyt]

See also https://github.com/GoogleChrome/puppeteer/issues/3453

[pswzyu]

@vsemozhetbyt thanks for sending this link. I was indeed using the bundled chromium under headless mode with a relative userDataDir. I did not pay attention to the cpu usage, but I did experience the hang. I did not know there is a connection between relative dir and the hang.

@sarfraznawaz2005 I also need to mention that after being not successful with the bundled chromium came with the puppeteer install, I switched to a chrome installed on my machine. Please try.

var chromeConfig = {
executablePath: 'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe',
headless: true,
userDataDir: "C:\xxxxxxxxxxxxxxxxx\tmpUserProfile",
args: [
'--remote-debugging-port=9222',
'--disable-gpu',
'--disable-infobars',
'--window-position=0,0',
'--ignore-certificate-errors',
'--ignore-certificate-errors-spki-list'
]
};

[sarfraznawaz2005]

@pswzyu In that case unfortunately it thinks it is not chrome browser even though I specified original chrome binary in executablePath, here is screenshot generated. Tried setting user-agent with no luck: await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36');

[marqmarti]

@sarfraznawaz2005 Same problem here. It seems that the problem is that setUserAgent is overridden when userDataDir data is loaded

[bp12345]

Was there a solution to this?

[onderceylan]

This is still an issue and it seems like providing userDataDir as launch arg does the trick: https://github.com/puppeteer/puppeteer/issues/921#issuecomment-433599870

[paragbaxi]

confirmed on mac os 10.15.1, puppeteer 2.0.0

[shirshak55]

hey guys today i found one solution to issue. persisting cookies seems to take like 60 seconds. So what i did was not closed browser for and delay 60 second. And found it works perfectly after that.

[ahvahsky2008]

Hey guys. Pls solve issue

[shirshak55]

@ahvahsky2008 whats your issue. I do see cookie being persisted properly in headless?

[lonecrane]

Confirmed on win 10 and chrome 80.0.3987.132.
Was there a solution to this?

[Vasile-Peste]

Up?

[Vasile-Peste]

I found a way to get around this issue!
giving the puppeteer.launch args for the userDataDir will keep it both on headless true/false.
for example, I had issue while using this:
const browser = await puppeteer.launch({headless: true, userDataDir: "myDataDir")
but when using like that:
const path = require('path');
const browser = await puppeteer.launch({headless: true, args: [`--user-data-dir=${path.resolve(__dirname, 'myDataDir')}`] });
It works perfectly 💯 (note that I had to use absolute path)

Please let me know if that works for you

For everyone having this issue: this solution is working. Thank you!

[sydney-d]

hello @Vasile-Peste what version are you using ?
It's not working for me, with v3

[Vasile-Peste]

hello @Vasile-Peste what version are you using ?
It's not working for me, with v3

2.1.1. You set the absolute path through path.resolve?

[sydney-d]

I'm downgraded to version 2 and it's working.
Cookies were lost with version 3 (and absolute path set through path.resolve)

Thanks

[Llorx]

Why is this closed?

[sathio]

I'm having the same issue, any way to have it working?

[vincentheet]

I'm also seeing this issue on Windows 10. Cookies aren't persisted when running in headless mode since v2.1.0. It worked fine in v2.0.0. The issue is not present on macOS, version v2.1.1 and v3.0.2 work as expected on the mac. The fix suggested by @Ron-Burg to set the user data dir as an argument does not work for me either.

On thing I noticed is that when running in headless mode true the Cookies table in the Cookies file (SQLite database) stays empty. When running with headless mode false the Cookies table does get populated.

[bozdoz]

Tried every suggestion here. Can't get cookies to persist from headless to headful. Even between headless sessions, the cookies are lost.

[Llorx]

So the solution is to revert to v2.0.x if headless cookies are critical to us?

[shanesreal]

I've downgraded puppeteer from v3.0.4 to v2.1.1 which didn't work and then finally downgraded to v2.0.0 and cache is now working properly in headless mode.

[peixotorms]

why is this closed?
Still not working.

[ArtixZ]

Is this issue really fixed in Chrome version 64.0.3262.0? I still have this issue in Chrome version 84.

Tried in Windows 10. userDataDir with headless doesn't store my login session.

[rubengmurray]

I'm adding this for reference as a few people have mentioned macOS in this thread: I've been able to use cookies for a couple of years without issue in puppeteer using the chrome-cookies-secure npm package.

[mikelpr]

workaround? (not cookies but whole user profile, including localstorage)
it is definitely not fixed. maybe a regression. really terrible in any case

[marqmarti]

Hi,

I had this problem about 1 year and a half ago and I submitted a comment in this thread. The same problem was happening to me a month ago and the solution I found was including the full path in the userDataDir parameter.

Example:

const browser = await puppeteer.launch({
    userDataDir: '/var/www/html/nodetest/whateverfolder',
    headless: true,
    args: ['--no-sandbox']
});

[kajojify]

Hello guys. Still face this on Chrome 85.0.4183.121 (Official Build) (64-bit) for Windows 10.

[Zogoo]

Is this issue solved or still pending? Because, it's still happening on v5.4.0

[shirshak55]

I just use playwright and there is no such issue + playwright is maintained by original authors. This issue will probably never be fixed because I don't see same progress in puppeteer as it used to be.

[anaskasmi]

I can't believe This issue is still persisting since 2017 and it is almost 2021 :/
puppeteer: "^5.3.1",
version of Chromium : 86.0.4240.0 (Developer Build) (64-bit)
OS : Windows 10 Pro

[coelho-faminto]

This is not a "bug" really.
The developers could have fixed this long ago but they wont. That is because saved sessions can defeat google recaptcha ;)

The only way to fix that is by recompiling chromium, this is not a puppeteer issue, you can easily reproduce this behavior by running the chromium/chrome command line passing --user-data-dir and --headless to confirm that when --headless is present the user-data-dir will be ignored and the sessions will not be retrieved.

This is by design...

[sathio]

and this is a very lame design I must say

[coelho-faminto]

Exactly... as a malicious software developer, for me this does not represent a limitation since I can bypass it perfoming some runtime tricks on the targeted software. But for those legit users who don't have the ($) time and motivation necessary to get through this, it is just lame design, as you say.

In the end they are blocking the legit users while the malicious developers are still taking full advantage from the debugging tools :+1:

If anyone here reading this is using chromium to defeat captcha on botnets: yay!

If you are not but you would like to you just have to think about one thing: The behavior of chromium-like browsers is completely different when --headless is enabled. So think carefully about it, all you really want is to do everything without the (victim) (client) user seeing anything. You don't need headless... Keep your software away from this option and you will be fine.