Got: Got request times out

[jdalrymple]

What would you like to discuss?

I'm testing a http request to pull some data from website source. What i can't seem to figure out is why the http get request is hanging. When i perform the same request with curl, it works just fine.

I've tried to narrow down the differences but i figured it would be worth asking if you have any ideas as to how i could discern the problem.

const got = require("got");
const { promisify } = require("util");
const { CookieJar } = require("tough-cookie");

const cookieJar = new CookieJar();
const setCookie = promisify(cookieJar.setCookie.bind(cookieJar));

const ISS =
    "https://www.yellowpages.com.au/search/listings?clue=cars&location=victoria";
const headers = {
    "user-agent":
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36",
    accept: "*/*",
};

setCookie(
    "yellow-guid=3d848e20-2b4d-4c44-ae26-f2dd2744f151",
    "https://www.yellowpages.com.au/search/listings"
)
    .then(() => {
        return got(ISS, { cookieJar, headers });
    })
    .then((output) => {
        console.log(output);
    })
    .catch((e) => {
        console.log(e);
    });

curl --request GET   --url 'https://www.yellowpages.com.au/search/listings?clue=cars&location=victoria'   --header 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'   --cookie yellow-guid=3d848e20-2b4d-4c44-ae26-f2dd2744f15

Thanks for any feedback!

Checklist

• [x] I have read the documentation.

[jbgrunewald]

I'm having similar timeout issues using the shopify-api-node client which uses got for http requests.

got version: 10.1.0
node version: 12

[szmarczak]

~I have tried [email protected] and everything works flawlessly.~ Actually I ran it on RunKit and indeed, there's a bug I think...

[szmarczak]

All versions of Got are affected...

[szmarczak]

Actually it's not a bug in Got. Native https fails too:

const https = require("https");

const ISS =
    "https://www.yellowpages.com.au/search/listings?clue=cars&location=victoria";
const headers = {
    "user-agent":
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36",
    accept: "*/*",
};

https.get(ISS, {
    headers
}).once('response', response => {
    console.log('got response');
});

got response is never logged.

[szmarczak]

The server doesn't send any data when running on RunKit. I think its IP has been banned. Closing as this is not a Got issue.

[jdalrymple]

Bugger. Well thanks for looking into it! Any ideas on how I could further debug? Or what sort of things curl could be doing that the native http would be missing out on?

[szmarczak]

Pass the -v (verbose) flag to curl.

[szmarczak]

Also, it works as expected if you set the http2 option to true.

[jdalrymple]

Didnt think of that! Thank you!!