Archivebox: Question: wget problems

[poblabs]

I'm getting this error below on a lot of sites. The archive is failing with the wget command:

failed: Connection refused.
failed: Cannot assign requested address.

If I remove --span-hosts from the wget command, it seems to work somewhat - some stuff is still missing.

Any tips on how to get past this?

[poblabs]

Maybe this is an ipv4 / ipv6 issue? Using -4 seems to work somewhat too but not entirely.

[poblabs]

On the same sites which wget fails, the chromium singlepage, pdf, etc. also fails. I'm a bit lost on why these normall working sites are failing in these 2 tools

[cdvv7788]

Can you try changing the user agent? Those sites may be blocking the requests based on that.

[poblabs]

Nope - no change when I set the chrome user agent, or the wget user agent. Can you try adding https://belchertownweather.com and see if it works for you? I have about 6 sites that fail and that is one of them.

[cdvv7788]

@poblabs I was able to add it without any issues:

This seems to be a network issue. Can you try running it on a VPN?

[poblabs]

Strange. When trying it on localhost to the website (I own the website), it still gives me an error but this time with chromium.

I am running this in docker.

wget appears to work but it's not downloading everything. Some JS files are 404'ing.

Not sure what to make of it.

[+] [2020-10-20 09:21:00] "belchertownweather.com"
    https://belchertownweather.com
    > ./archive/1603200060.693373
      > title
      > favicon
      > wget
      > singlefile
        Extractor failed:
            TimeoutExpired Command '['/node/node_modules/single-file/cli/single-file', '--browser-executable-path=chromium', '--browser-args=[\\"--headless\\", \\"--no-sandbox\\", \\"--disable-gpu\\", \\"--disable-dev-shm-usage\\", \\"--disable-software-rasterizer\\", \\"--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36\\", \\"--window-size=1440,2000\\"]', 'https://belchertownweather.com', '/data/archive/1603200060.693373/singlefile.html']' timed out after 60 seconds
        Run to see full output:
            cd /data/archive/1603200060.693373;
            /node/node_modules/single-file/cli/single-file --browser-executable-path=chromium "--browser-args=[\"--headless\", \"--no-sandbox\", \"--disable-gpu\", \"--disable-dev-shm-usage\", \"--disable-software-rasterizer\", \"--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36\", \"--window-size=1440,2000\"]" https://belchertownweather.com /data/archive/1603200060.693373/singlefile.html

      > pdf
        Extractor failed:
            TimeoutExpired Command '['chromium', '--headless', '--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage', '--disable-software-rasterizer', '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36', '--window-size=1440,2000', '--timeout=60000', '--print-to-pdf', 'https://belchertownweather.com']' timed out after 60 seconds
        Run to see full output:
            cd /data/archive/1603200060.693373;
            chromium --headless --no-sandbox --disable-gpu --disable-dev-shm-usage --disable-software-rasterizer "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36" --window-size=1440,2000 --timeout=60000 --print-to-pdf https://belchertownweather.com

      > screenshot
        Extractor failed:
            TimeoutExpired Command '['chromium', '--headless', '--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage', '--disable-software-rasterizer', '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36', '--window-size=1440,2000', '--timeout=60000', '--screenshot', 'https://belchertownweather.com']' timed out after 60 seconds
        Run to see full output:
            cd /data/archive/1603200060.693373;
            chromium --headless --no-sandbox --disable-gpu --disable-dev-shm-usage --disable-software-rasterizer "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36" --window-size=1440,2000 --timeout=60000 --screenshot https://belchertownweather.com

      > dom
        Extractor failed:
            TimeoutExpired Command '['chromium', '--headless', '--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage', '--disable-software-rasterizer', '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36', '--window-size=1440,2000', '--timeout=60000', '--dump-dom', 'https://belchertownweather.com']' timed out after 60 seconds
        Run to see full output:
            cd /data/archive/1603200060.693373;
            chromium --headless --no-sandbox --disable-gpu --disable-dev-shm-usage --disable-software-rasterizer "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36" --window-size=1440,2000 --timeout=60000 --dump-dom https://belchertownweather.com

      > readability
      > mercury
      > media
      > headers

[√] [2020-10-20 09:25:08] Update of 1 pages complete (4.14 min)
    - 0 links skipped
    - 1 links updated
    - 1 links had errors

[cdvv7788]

What docker image are you using? Can you try building an image with the version on master and try again? (docker build -t archivebox --no-cache).
You can also try the individual commands, to see how they are failing. That may give us more information.

[poblabs]

Same result with master built docker. It seems to be the 1 website of mine giving me problems. I just tried a bunch more and they seem ok. Not sure why mine is special but I don't think this is an archivebox problem. 🤷🏻‍♂️

[cdvv7788]

@poblabs feel free to re-open the issue if you find something that we can help with.