Archivebox: Archive Method: CHROME_USER_DATA_DIR is being ignored, authenticated site archiving fails

[jamelait]

Hi,
There is a website that requires me to be logged in that I want to archive.

I have logged in using chromium-browser (launched from bash) and I made sure to set the env (with export command) for:
CHROME_BINARY is set to value returned by the command "which chromium-browser"
CHROME_USER_DATA_DIR is set to value that I saw in "chrome://version"

I then do

echo 'https://members.website.com/my-account' | ./archive

The command runs fine but it looks like my credentials are not used to access the website (a message saying that i should be logged in is shown in the archived html).

What could be the issue?

chromium-browser --version : Chromium 73.0.3683.75 Built on Ubuntu , running on Ubuntu 18.04

[pirate]

Huh, strange. Try setting CHROME_HEADLESS=False so you can watch the browser UI as it archives, it may reveal some problem.

You can also try running the chrome command manually like this (try it both with and without --headless, also replace that user-data-dir path with your correct one if different):

chromium-browser --headless --timeout=60000 --user-data-dir=~/.config/chromium --window-size=1440,2000 --screenshot http://example.com

[jamelait]

Setting CHROME_HEADLESS=False did reveal the problem: it was using the wrong user data directory.
I was setting the env variable like this: CHROME_USER_DATA_DIR=/home/jamel/.config/chromium/Default
But it seems that in that configuration, chromium creates a new profile directory so the new user-data-dir becomes /home/jamel/.config/chromium/Default/Default.
So of course I wasn't logged in.

Problem solved!

I did notice a weird thing: it seems that /ArchiveBox/output/archive/.../members.website.com/my-account/index.html was not accessed with a logged in profile but /ArchiveBox/output/archive/.../output.html was.

[pirate]

Ah yes, this problem is common, I’ve made the same mistake of using the default directory before too.

The Index.html output is generated using Wget, not chrome, to make that one be logged in you have to pass a COOKIES_FILE=... parameter.