Hosts: Invalid domain

[cleanev]

Invalid domains show when updating pihole using pihole -up at console

[i] Target: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
[✓] Status: Retrieval successful
[i] Received 110984 domains, 8 domains invalid!
Sample of invalid domains:
- ɢoogletranslate.com
- accounts.ɢoogletranslate.com
- ɢoogle.company
- ɢoogle.email
- ɢoogle.life

[lightswitch05]

These are just unicode characters. Pihole cannot read them, so it just ignores them. My understanding is that the next major version of PiHole will be able to read these. In the mean time, they are not causing any trouble and can be safely ignored.

[cleanev]

Thanks Daniel for quick turnaround. I am running latest available beta from pihole v5 and this may need someone’s attention if not already to bake in the code to read these Unicode characters.

Thanks

[lightswitch05]

My comment about v5 supporting it came from this comment: https://github.com/lightswitch05/hosts/issues/145#issue-582111769

Other then the comment - I don't have any more information about it. If its still showing as an error in v5, I would just continue to ignore it. The important part is that when I added these hosts, I also added the punycode versions. Which means these hosts are in the list twice, and so regardless of the format that your blocking tool uses (in this case the pihole) - these hosts should still get blocked.

[cleanev]

Opened a topic with pihole to get pulse of this issue
https://discourse.pi-hole.net/t/invalid-domain-bug/29935

[yubiuser]

Other then the comment - I don't have any more information about it. If its still showing as an error in v5, I would just continue to ignore it.

Yes, pihole still complains about the domains - but as I knew they are duplicated I just ignored the warning.

For longer answer see the discussion on pihole discourse.

[DL6ER]

My take is that Unicode characters are wrong. Browsers like Firefox and Chrome do automatically convert international domain names to Punycode before issuing them to the DNS server. This is the correct behavior following the Internet standards. Check your Query Log, you will see that your clients correctly requested the Punycode (xn--...) form.

Applications like dig are so-called "not-IDNA aware" applications and cannot be used with international domains. The fact that they send Unicode characters without conversion is incorrect.

Just saying: dnsmasq includes an IDN feature which automatically converts all loaded domains into IDN form. This may also show you that this is the "correct" form for a DNS server. We do not use this feature as it prolongs the list loading time from about 0.5 seconds to roughly 120 seconds (measured using a Pi Zero with little less than a million domains). This seems inappropriate given the negligible (if any) advantage it brings.

I put up some more details on the linked Discourse discussion.

[jfb-pihole]

To summarize from the Pi-hole perspective (from the linked Discourse discussion):

1. The domains are invalid (see DL6ER discussion above)
2. Pi-hole does not import the domains
3. Pi-hole correctly reports that the domains were not imported
4. There are no pending changes in Pi-hole code to change this behavior.

[lightswitch05]

Hey great news, I did some testing with uBlock Origin and uBlock Matrix. Both of them will check for the Punycode version in block lists. See https://github.com/lightswitch05/hosts/commit/10c0513cd37b30993cb72196985915ae38a997d3.

After this test, I'm satisfied that only the Punycode versions need to be added my list. I'll go through and remove the entries causing issues.

Sorry to the PiHole team from taking up your time with this!

[lightswitch05]

closed by https://github.com/lightswitch05/hosts/commit/818cd79549688955bd57e7f05d76a6c31bb17c41

Thank you everyone for contributing to this discussion and giving me the motivation to actually test it with browser extensions.

[yubiuser]

We do not use this feature as it prolongs the list loading time from about 0.5 seconds to roughly 120 seconds (measured using a Pi Zero with little less than a million domains).

Thank you for not using this feature.