Colly: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[maldahleh]

I apologize if this is something I am overlooking as I'm new to Go and webscraping in general, though what I am trying to do is scrap this webpage (https://dd.weather.gc.ca/climate/observations/daily/csv/), visit each link there, and create a mapping of basically station IDs to their names, and the month/years which they have available data.

This is what I came up with so far: https://github.com/maldahleh/weatherstats-location-service

I get the following:

Error: Get https://dd.weather.gc.ca/climate/observations/daily/csv/AB/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

I've tried a few things but I can't figure out how to get around that error.

[vosmith]

Hey @maldahleh
I tried navigating to that site manually, it just takes a very long time to load. The default request timeout for colly is 10s. You can override this in your collector with the SetTimeout function.

[maldahleh]

@vosmith Thanks, sorry I missed that and that fixed it.

On a link like https://dd.weather.gc.ca/climate/observations/daily/csv/ON/ I seem to only get links including to and up to 613F606, which is like 70% down the page and not the remainder of them.

Any ideas what would be causing this?

Thanks in advance