Hosts: Extra newlines in generated hosts file around comments

[jameswilson]

When I run python3 updateHostsFile.py then scroll to the bottom of the generated hosts file I see this:

#

# The contents of this file (containing a listing of additional domains in

# 'hosts' file format) are appended to the unified hosts file during the

# update process.



0.0.0.0 someblacklisteddomain.com

Which comes from the blacklist file, however in the blacklist file those lines look like this:

# blacklist
#
# The contents of this file (containing a listing of additional domains in
# 'hosts' file format) are appended to the unified hosts file during the
# update process.

0.0.0.0 someblacklisteddomain.com

Somehow the script is inserting extra newline characters, but only around empty lines (/^\s?\n/ in regex parlance) and lines that start with comments (/\s?#.*/ in regex parlance).

Additionally comments after a valid line are being stripped, so for example.

In blacklist...

0.0.0.0 someblacklisteddomain.com  # some comment here

Becomes...

0.0.0.0 someblacklisteddomain.com

Interestingly, myhosts is not affected by this issue. Inline comments at the end of valid hosts rules are left as-is, and there are no extra newlines between normal comment blocks.

all 3-rd party blocklists and the included custom blacklist are both affected by this. Example:

# Start adaway.org

# AdAway default blocklist

# Blocking mobile ad providers and some analytics providers

# 

[welcome[bot]]

Hello! Thank you for opening your first issue in this repo. It’s people like you who make these host files better!

[StevenBlack]

Thanks James! Yep, this is a bug!

[funilrys]

Hello @jameswilson,

welcome here and thanks for this awesome first issue,

I may know where the problem comes from for the \n after commented line.

Will look into this.

[jameswilson]

I've added @funilrys fork as a remote, checked out the branch issue-777 and confirmed that this does indeed solve the issue.

$ git remote add funilrys [email protected]:funilrys/hosts.git

$ git checkout issue-777 

$ python3 updateHostsFile.py
Do you want to update all data sources? [Y/n] n
OK, we'll stick with what we've got locally.
Do you want to exclude any domains?
For example, hulu.com video streaming must be able to access its tracking and ad servers in order to play video. [Y/n] n
OK, we'll only exclude domains in the whitelist.
==>fe00::0 ip6-localnet<==
==>ff00::0 ip6-mcastprefix<==
==>ff02::2 ip6-allrouters<==
==>ff02::3 ip6-allhosts<==
Success! The hosts file has been saved in folder 
It contains 60,123 unique entries.
Do you want to replace your existing hosts file with the newly generated file? [Y/n] n

$ tail -n 8 hosts

# blacklist
#
# The contents of this file (containing a listing of additional domains in
# 'hosts' file format) are appended to the unified hosts file during the
# update process.

0.0.0.0 someblacklisteddomain.com

Thanks @funilrys ... lucky #777 😆 🤞

[jameswilson]

Thanks guys. 🥇 This is really going to help keep my flavor of OCD in check.