Hosts: updating freezes -- should we timeout?

[stolk]

When updating, it freezes for a long time because of malwaredomainlist.com

$ python3 updateHostsFile.py 
Do you want to update all data sources? [Y/n] 
Updating source extensions/fakenews from https://raw.githubusercontent.com/marktron/fakenews/master/fakenews
Updating source extensions/social from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/social-hosts
Updating source extensions/porn/sinfonietta from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/pornography-hosts
Updating source extensions/porn/sinfonietta-snuff from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/snuff-hosts
Updating source extensions/porn/clefspeare13 from https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/0.0.0.0/hosts
Updating source extensions/gambling from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts
Updating source data/yoyo.org from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext&useip=0.0.0.0
Updating source data/mvps.org from http://winhelp2002.mvps.org/hosts.txt
Updating source data/someonewhocares.org from http://someonewhocares.org/hosts/zero/hosts
Updating source data/tyzbit from https://raw.githubusercontent.com/tyzbit/hosts/master/data/tyzbit/hosts
Updating source data/StevenBlack from https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
Updating source data/Badd-Boyz-Hosts from https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Updating source data/add.Dead from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Updating source data/malwaredomainlist.com from https://www.malwaredomainlist.com/hostslist/hosts.txt
Problem getting file:  https://www.malwaredomainlist.com/hostslist/hosts.txt
Error in updating source:  https://www.malwaredomainlist.com/hostslist/hosts.txt
Updating source data/add.2o7Net from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Updating source data/SpotifyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/SpotifyAds/hosts
Updating source data/add.Risk from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Updating source data/add.Spam from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Updating source data/UncheckyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
Updating source data/adaway.org from https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Updating source data/KADhosts from https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Do you want to exclude any domains?

[welcome[bot]]

Hello! Thank you for opening your first issue in this repo. It’s people like you who make these host files better!

[StevenBlack]

Hi Bram @stolk.

Can you try this again? It's not uncommon to have issues reaching some domains sometimes.

[funilrys]

Nice welcoming bot Steven @StevenBlack :+1:

Back to our issue,

What I can note:

• a wget or curl
  
  
  
  • of https://www.malwaredomainlist.com/hostslist/hosts.txt ==> doesn't work
  
  
  • of http://www.malwaredomainlist.com/hostslist/hosts.txt ==> works
  
  
• a browser access
  
  
  
  • to https://www.malwaredomainlist.com/hostslist/hosts.txt ==> works
  
  
  • to http://www.malwaredomainlist.com/hostslist/hosts.txt ==> works
  
  

I don't know which file to change but I currently have time to write a PR if you want.

Have a nice day/night.

[StevenBlack]

@funilrys yes, @probot is pretty cool.

Yes a PR would be awesome!

[StevenBlack]

@funilrys both are working for me now, from central Canada.

[stolk]

Still pauses a minute or so from Vancouver, BC.
But this time it did not report "Problem getting file" so I assume it eventually got it.

[funilrys]

My results were from Germany as I live there but using a Canadian, France or even a Spanish IP there's no problem ...

The problem is somewhere else ... Do we have an implemented timeout @StevenBlack ? :thinking:

[StevenBlack]

@funilrys Nissar I would entertain a PR for timing out requests on fetching sources. One that does not clobber the existing hosts file with an empty one upon timeout....

[rautamiekka]

https://www.malwaredomainlist.com/hostslist/hosts.txt appears to be taking unnecessarily long but browser access from a Finnish 4G LTE IP and a wget access from an American one both work (others untested), with the former taking something half a minute to handshake the TLS and the latter a few extra seconds than usual it should with TLS.

[funilrys]

Not sure about this one @stolk may confirm this but this may also be fixed by #520 as I could download that file and even bigger files without any problem @StevenBlack :thinking:

[funilrys]

Hi @StevenBlack,
This one may be closed due to the lack of feedback after #520 and its child PR.