Hosts: UnicodeDecodeError

[khalyk]

Error in updating source: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Traceback (most recent call last):
File "hosts/updateHostsFile.py", line 1513, in
main()
File "hosts/updateHostsFile.py", line 168, in main
update_all_sources(source_data_filename, settings["hostfilename"])
File "hosts/updateHostsFile.py", line 595, in update_all_sources
update_data = json.load(update_file)
File "/usr/lib/python3.6/json/__init__.py", line 296, in load
return loads(fp.read(),
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 27: ordinal not in range(128)

Dockerfile attached.

docker run -it <image_id> /usr/bin/python3 hosts/updateHostsFile.py --auto --extensions social porn fakenews gambling

Dockerfile.txt

Thanks!!

[welcome[bot]]

Hello! Thank you for opening your first issue in this repo. It’s people like you who make these host files better!

[StevenBlack]

Hi @khalyk what version of Python are you using?

[khalyk]

Hey @StevenBlack this is on Ubuntu and python3 (3.6.5-3).

Thanks!

[StevenBlack]

@khalyk is this comment helpful in your case?

[khalyk]

Thanks @StevenBlack I will add the above to the container environment and give it a try.

[khalyk]

It does log a bunch of error messages, but seems to succeed:

docker run -it -e LC_ALL=C.UTF-8 -e LANG=C.UTF-8 af6a7053aece /usr/bin/python3 hosts/updateHostsFile.py --auto --extensions social porn fakenews gambling

Updating source hosts/extensions/fakenews from https://raw.githubusercontent.com/marktron/fakenews/master/fakenews
Error in updating source: https://raw.githubusercontent.com/marktron/fakenews/master/fakenews
Updating source hosts/extensions/porn/sinfonietta from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/pornography-hosts
Error in updating source: https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/pornography-hosts
Updating source hosts/extensions/porn/clefspeare13 from https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/0.0.0.0/hosts
Error in updating source: https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/0.0.0.0/hosts
Updating source hosts/extensions/porn/sinfonietta-snuff from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/snuff-hosts
Error in updating source: https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/snuff-hosts
Updating source hosts/extensions/social from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/social-hosts
Error in updating source: https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/social-hosts
Updating source hosts/extensions/gambling from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts
Error in updating source: https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts
Updating source hosts/data/add.Dead from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Error in updating source: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Updating source hosts/data/someonewhocares.org from http://someonewhocares.org/hosts/zero/hosts
Error in updating source: http://someonewhocares.org/hosts/zero/hosts
Updating source hosts/data/lightswitch05 from https://raw.githubusercontent.com/lightswitch05/hosts/master/ads-and-tracking-extended.txt
Error in updating source: https://raw.githubusercontent.com/lightswitch05/hosts/master/ads-and-tracking-extended.txt
Updating source hosts/data/yoyo.org from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext&useip=0.0.0.0
Error in updating source: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext&useip=0.0.0.0
Updating source hosts/data/add.Risk from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Error in updating source: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Updating source hosts/data/CoinBlockerList from https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
Error in updating source: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
Updating source hosts/data/malwaredomainlist.com from http://www.malwaredomainlist.com/hostslist/hosts.txt
Error in updating source: http://www.malwaredomainlist.com/hostslist/hosts.txt
Updating source hosts/data/add.2o7Net from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Error in updating source: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Updating source hosts/data/add.Spam from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Error in updating source: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Updating source hosts/data/Badd-Boyz-Hosts from https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Error in updating source: https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Updating source hosts/data/mvps.org from http://winhelp2002.mvps.org/hosts.txt
Error in updating source: http://winhelp2002.mvps.org/hosts.txt
Updating source hosts/data/KADhosts from https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Error in updating source: https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Updating source hosts/data/UncheckyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
Error in updating source: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
Updating source hosts/data/adaway.org from https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Error in updating source: https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Updating source hosts/data/StevenBlack from https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
Error in updating source: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
==>fe00::0 ip6-localnet<==
==>ff00::0 ip6-mcastprefix<==
==>ff02::2 ip6-allrouters<==
==>ff02::3 ip6-allhosts<==
Success! The hosts file has been saved in folder
It contains 71,000 unique entries.

[StevenBlack]

I'm gonna document this.

[funilrys]

This one is weird everything return error but still have 71 000 entries :thinking:

[funilrys]

Wait... @khalyk it's python 3.6.5 !? :scream:

[real-t0mg]

hi,

same error happens in python 2.7.12 in Ubuntu 16.04.4 LTS and in Python 2.7.15 in Slackware 14.2, with C.UTF-8 locales and any other locale you set.

Slackware:

Traceback (most recent call last):
File "updateHostsFile.py", line 1513, in
main()
File "updateHostsFile.py", line 187, in main
merge_file = create_initial_file()
File "updateHostsFile.py", line 640, in create_initial_file
write_data(merge_file, start + curFile.read() + end)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19824: ordinal not in range(128)
Failed to update hosts file

Ubuntu:

Failed to update hosts file
Traceback (most recent call last):
File "updateHostsFile.py", line 1513, in
main()
File "updateHostsFile.py", line 187, in main
merge_file = create_initial_file()
File "updateHostsFile.py", line 640, in create_initial_file
write_data(merge_file, start + curFile.read() + end)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19824: ordinal not in range(128)
Failed to update hosts file

[funilrys]

What about using Python 3 @real-t0mg ?

[iharosi]

Experiencing the same on macOS High Sierra 10.13.6 with python version 2.7.10.
I've installed dependencies with pip.
The exact error message is the following:

Do you want to exclude any domains?
For example, hulu.com video streaming must be able to access its tracking and ad servers in order to play video. [Y/n] n
OK, we'll only exclude domains in the whitelist.
Traceback (most recent call last):
  File "updateHostsFile.py", line 1513, in <module>
    main()
  File "updateHostsFile.py", line 187, in main
    merge_file = create_initial_file()
  File "updateHostsFile.py", line 640, in create_initial_file
    write_data(merge_file, start + curFile.read() + end)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19824: ordinal not in range(128)

[lightswitch05]

Maybe time to drop support for Python 2 in this project? The countdown has already begun, and it will also fix the failing travis-ci tests

[StevenBlack]

Yes, Daniel @lightswitch05, I think it's time to drop Python 2.7 support.

[anudeepND]

@StevenBlack @lightswitch05 I agree

[9ae8sdf76]

For those who are stuck using python 2.7 for any reason, you can add the following two lines (right below the list of imports) to updateHostsFile.py to work around the above error:

reload(sys)
sys.setdefaultencoding('utf8')

[funilrys]

Hello there,

we are dropping Python 2 soon so please report to #729 and #769.

About this issue the comment of Steve @StevenBlack https://github.com/StevenBlack/hosts/issues/700#issuecomment-401974478 is clear and works as tested by many Python 2 users ...

Steve @StevenBlack Maybe it's time to close this and invite the Python 2 users to update their Python version or follow the discussion at #670 if they don't want to ...

[StevenBlack]

Thanks @dohpaz42. I agree, @funilrys.

Closing.

[9ae8sdf76]

@funilrys I understand; I meant that to be for future users to find. Thank you though. :)