Archivebox: Bugfix: UnhandledPromiseRejectionWarning with singlefile attempt

[drpfenderson]

Describe the bug

Decided to move forward with a clean archive, and keep the old one as a historical snapshot. Added a new list of links to archive, but after 6 or 7, almost all of the tasks start erroring out in cascade fashion, starting with singlefile step. Error is TimeoutExpired Command for chromium-browser, but it gives me a command to run, which also errors out with (node:162485) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected number in JSON at position 2

Steps to reproduce

1. Installed with pip install archivebox
2. Changed config to disable MEDIA and timeout set to 180 seconds.
3. Added list of links with archivebox add ./list.txt.
4. Starts adding things correctly, but after link 5 or 6 starts to error out.

Screenshots or log output

Original error

[i] [2020-09-02 17:41:18] ArchiveBox v0.4.21: archivebox update
    > /mnt/volume/.archivebox-output/new-archive


[*] [2020-09-02 17:41:21] Writing 33 links to main index...
    √ /mnt/volume/.archivebox-output/new-archive/index.sqlite3
    √ /mnt/volume/.archivebox-output/new-archive/index.json
    √ /mnt/volume/.archivebox-output/new-archive/index.html

[▶] [2020-09-02 17:41:23] Collecting content for 8 Snapshots in archive...

[√] [2020-09-02 17:44:28] "NYPD used facial recognition to track down Black Lives Matter activist - The Verge"
    https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram
    √ ./archive/1598996327.937664
      > singlefile
        Extractor failed:
            TimeoutExpired Command '['/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file', '--browser-executable-path=chromium-browser', '--browser-args="["--headless", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"', 'https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram', '/mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664/singlefile.html']' timed out after 180 seconds
        Run to see full output:
            cd /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664;
            /mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file --browser-executable-path=chromium-browser "--browser-args="["--headless", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"" https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664/singlefile.html

      > pdf
        Extractor failed:
            Exception Failed to chmod: output.pdf does not exist (did the previous step fail?)
        Run to see full output:
            cd /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664;
            chromium-browser --headless "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36" --window-size=1440,2000 --timeout=180000 --print-to-pdf https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram

When I run

cd /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664;
/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file --browser-executable-path=chromium-browser "--browser-args="["--headless", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"" https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664/singlefile.html

Output:

(node:162485) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected number in JSON at position 2
    at JSON.parse (<anonymous>)
    at getBrowserOptions (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/back-ends/puppeteer.js:63:51)
    at Object.exports.initialize (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/back-ends/puppeteer.js:36:35)
    at initialize (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file-cli-api.js:46:16)
    at run (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file:33:59)
    at Object.<anonymous> (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file:30:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
(node:162485) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:162485) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Software versions

• OS: Ubuntu 20.04.1 LTS
• ArchiveBox version: 0.21
• Python version: Python 3.7.9
• Chrome version: Chromium 84.0.4147.105

[cdvv7788]

@drpfenderson there is a PR that should fix the issue with the index. You can give it a try (it should be merged this week tho): https://github.com/pirate/ArchiveBox/pull/452

About this issue, I will give it a check.

[cdvv7788]

@drpfenderson It works for me. Can you try running it with docker or installing the npm dependencies? (the quickstart has been updated with the instructions)

[drpfenderson]

@cdvv7788 I get the same timeout using docker/docker-compose, though I am really not sure how to run the node_modules commands from inside the docker to see the exact error. If I cd to the directory listed, don't I lose the docker-compose.yml and the path for node_modules? I tried researching the exec/run commands a bit to figure out how to execute the single-file command from within a container, but I can't really grok it.

$ docker-compose run archivebox add https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram

[i] [2020-09-03 15:55:38] ArchiveBox v0.4.21: archivebox add https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram
    > /data

[+] [2020-09-03 15:55:41] Adding 1 links to index (crawl depth=0)...
    > Saved verbatim input to sources/1599148541-import.txt
    > Parsed 1 URLs from input (Plain Text)
    > Found 1 new URLs not already in index

[*] [2020-09-03 15:55:41] Writing 33 links to main index...
    √ /data/index.sqlite3
    √ /data/index.json
    √ /data/index.html

[▶] [2020-09-03 15:55:42] Collecting content for 1 Snapshots in archive...

[+] [2020-09-03 15:55:42] "theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram"
    https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram
    > ./archive/1599148541.818964
      > title
      > favicon
      > wget
      > singlefile
        Extractor failed:
            TimeoutExpired Command '['/node/node_modules/single-file/cli/single-file', '--browser-executable-path=chromium', '--browser-args="["--headless", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "--disable-software-rasterizer", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"', 'https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram', '/data/archive/1599148541.818964/singlefile.html']' timed out after 180 seconds
        Run to see full output:
            cd /data/archive/1599148541.818964;
            /node/node_modules/single-file/cli/single-file --browser-executable-path=chromium "--browser-args="["--headless", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "--disable-software-rasterizer", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"" https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram /data/archive/1599148541.818964/singlefile.html

EDIT: To be clear, the npm dependencies were for sure installed, as far as I can tell. No errors on install using npm install --prefix . 'git+https://github.com/pirate/ArchiveBox.git'. I actually installed both using the prefix for the master branch, as well as your sql_index branch.

[gildas-lormeau]

Author of SingleFile here. This is a bug in SingleFile due to the presence of "," and spaces in the --browser-args switch (e.g. 1440,2000 and Mozilla/5.0 (Windows NT 10.0; Win64; x64)). I'm trying to see how it could be fixed or circumvented. I guess proposing to pass a JSON string was not a good idea.

[gildas-lormeau]

Finally, I was able to fix the issue by formatting the --browser-args switch like this (surrounding quotes included) in your example:

"--browser-args=""[""--headless"", ""--no-sandbox"", ""--disable-gpu"", ""--disable-dev-shm-usage"", ""--disable-software-rasterizer"", ""--window-size=1440,2000"", ""--user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36""]"""

instead of:
"--browser-args="["--headless", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "--disable-software-rasterizer", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]""

[cdvv7788]

Thanks. We definitely need to improve the command so it can be run directly on case of error. I will add that to the pending tasks.

@drpfenderson What is your TIMEOUT setting? Have you modified any setting or changed any environment variable? I cannot reproduce it, not even with docker, so I am suspecting you have something unusual in your local setup.

[drpfenderson]

@cdvv7788 My timeout is set to 180, which I set in the docker-compose.yml file. I've tried with much longer times, like 3600 as I've seen mentioned a couple times, but it didn't change anything. The only other change is SHOW_PROGRESS, but I've also tested it with the default docker-compose file.

I'm wondering if there is some config or library file somewhere on my system from the much older versions of archivebox that is floating around and interfering. Before upgrading to v0.21 or docker, I made sure and scrubbed the .conf files and .local stuff related to it, in my user folder as well as the archive folder itself, but there still must be something somewhere that I'm missing since the problem happens with the brand new archive as well. I'll do some deeper searching and let you know.

[cdvv7788]

@drpfenderson I just created a PR that should escape correctly the argument. With that you should be able to, at least, run the command properly, and come back with the error that singlefile is returning.

[drpfenderson]

Well, I essentially did a find/replace/delete for every archivebox/node/python file on my system that could be related. There were a number of weird places with files due to the various installation methods I've used over the years for this program. Realizing the amount of work to disentangle everything, I spun up a new server, attached the archive to it, wiped all the local config/conf files, ran the docker-compose run archivebox add and it all worked!

Thank you all for your infinite patience with me.