Filterlists: automatic URL validation

[collinbarrett]

All URLs in the Directory should be periodically validated for the following:

• [ ] Is valid URI syntax
• [ ] Availability (i.e., HTTP 200 vs 404)
• [ ] Prefer direct URL if 301/302 redirected
• [ ] Prefer HTTPS if available
• [ ] Malware (using VirusTotal API or similar)

[collinbarrett]

Partially being done now by the Agent BOT over in #836

[collinbarrett]

blocked by #940

[collinbarrett]

Maybe integrate https://github.com/funilrys/PyFunceble . Great tip from @DandelionSprout

[DandelionSprout]

PyFunceble is slightly difficult to install and keep updated, but it works very well once it has been set up. A simple command like PyFunceble -uf [File containing full URLs] should work pretty well.

[funilrys]

Hi there, maintainer of PyFuncebke out here!
Thanks for the shootout @collinbarrett and @DandelionSprout !

I was looking for discussions about PyFunceble and found this issue. How can I help reduce the difficulties of using, installing, and keeping PyFunceble up-to-date?

Have you tested the docker images?

Let me know if I can help reduce complexity around PyFunceble!

Have a nice day/night.
Stay safe and healthy.
Nissar

[DandelionSprout]

On my end, I had big update problems this spring wherein I had problems updating existing builds, and to get PowerShell and/or Cygwin to recognise the existence of the PyFunceble command. My unqualified guess is that it was the result of me trying multiple methods to install and update PyFunceble on my PC (e.g. PIP and pure Python), leading to parallel builds of it in various folders that conflicted with each other.

However, in early July I decided to try to delete every single file and folder on my PC that had the word PyFunceble in them, and then it went far better to get it to work after a reinstall with the pure Python method.

[funilrys]

Just a notice for you guys :-) Would you be interested in PyFunceble as a web API ? 🤔

I'm starting to think of what to prioritize next and such an idea is my box of ideas. I just need some input about how many people or infrastructure will be interested in that.

[collinbarrett]

@funilrys YES!!!!! That'd be awesome. It'd be incredibly helpful to keep the URLs that are on FilterLists clean and valid. I had thought about trying to get an instance of PyFunceble running on the FilterLists server, but just haven't had the time to learn how it works. If there was a simple Web API we could call, that'd be much easier to integrate.