Filterlists: support creating custom combination lists

[collinbarrett]

In the same vein as Steven Black's project (but for filter lists as well as hosts lists), it could be useful for FilterLists to be able to easily merge and de-duplicate (likely server-side) multiple lists into custom combination lists. Depending on list licensing, FilterLists could even automatically maintain these custom lists so that users could subscribe to them for future updates.

[collinbarrett]

Potentially useful tools:

• Find useless filters
• Redundancy check for Adblock Plus rules
• FOP (also here)

[bershan2]

Probably you knew this already, but for the record: uBlock Origin already has a de-duplication functionality. Lists are de-duplicated upon loading them (update), and the information about duplicates is displayed to the user. Specifically, on uBlock dashboard on the "3rd-party lists" pane and in the logger.

The easiest thing would be to take JavaScript de-duplication function from there.

Also, I think (and might be wrong) that de-duplication should be a standard feature for every network filter.

[collinbarrett]

Thanks, @bershan2 . Yes, I am aware. My vision for this feature would be to allow users to create custom combo lists easily (via simple point and click web interface). If that happened, I would want the resulting lists to be de-dupped rather than relying on the adblock software to handle it. Good note about uBO already having this built in. We could look at cloning it somehow. Thanks for your comments, and if you have any further ideas on this issue or time to work on, please feel free!

[bershan2]

I still do not understand why you want to de-duplicate lists yourself, I have a few points that I will enumerate for simplicity of referencing.

1. Would it be useful at all?
   I still believe that simple remove-exactly-matching-lines de-duplication is a fairly basic feature and every use-worthy (made up word) network filter/blocker should have it already. However, I might be wrong so I'll try to look into it later.
2. Why on your server part 1: "Who pays the bills?"
   I'm still confused why you'd want to compute anything on your server, if it is doable on clients' machines (JavaScript from uBlock Origin alseady runs on client machines). Since this computation benefits only the list subscriber, it is only fair that the list subscriber pays for it (and probably won't notice this computation at all). If you do it on your server, you pay for computational resources. If you get server resources for free, then ask yourself (0) who pays the bills, and (1) is it really a good use of that sponsor's generosity and (3) see next section?
3. Why on your server part 2: Privacy
   As a rule of thumb, ideally, you can only trust (accuracy and privacy of) the computation that you can oversee -- which requires that which means you have control over the computational machine. Less ideally, but more realistically, someone else performs the computation for you and you have to trust that the reward you give them is enough to maintain their honesty to you. Finally, if user does not pay for the service, he/she has greater chance to become the product itself -- that's what lead to ads-sponsored Internet. By the way, I'm not against ads -- the entire industry (both users and content creators) made the "lazy choice" of using ads and tracking in place of actual payments. The point of network filtering is to revert this trend and shift to a better privacy-respecting model. Thus user should rely only on services that he/she pays for. So do you really want to off-load any privacy-related task onto someone out of your and user's control?
4. This might be more resource-intensive then you think.
   Even with caching most common requests you'd have very many possible combinations (that's the whole point of customization). Furthermore, so far your server just serves people static pages* who are looking for more lists (very rare occasion), but lists need frequent automatic updating and update frequency is equal to the minimum over all used lists.
5. Finally, if implemented, would it be used?
   If you were a regular user and saw such a feature on a site, would you use it? I can definitely say that I would not... because of the above points. But would you?..

Note *: or they should be static because they have functionality of static pages, but I'm not familiar with WordPress.

[collinbarrett]

Hi @bershan2 , sorry for the delay. It's been a crazy week.

Thanks for your detailed thoughts. TBH, this feature is still pretty far in the future and may or may not actually happen. There are a fair number of pieces that would need to get implemented first. But, I have made note of your comments, and I really do appreciate your feedback.

[elypter]

@bershan2

it does but even though lists are relatively small they can pile up. imagine you have 10 different versions of a big list like fanboy ultimate then that would be 10*3,5 mb. if you have more of those then this can put a dent in your data volume or ram because some part of the system, wither the filter application or the os cache will keep a copy. it also means that files stay the same for longer because duplicates do not trigger an update. privacy and security is a point that needs to be addressed but it is also important to note that if you subscribe to a big portion of the available lists you are going to connect to dozens, if not hundreds of servers that can all log you access details. about the list integrity: since the site is open source you could create a clone of it and compare the results to see if one of the sites is abusing its central position.

i think this is very doable on both the client and the server. it might be easier to create lists of lists without merging that the client software can handle first like i described in this issue but a server side refinement of lists can be very beneficial because it works on all systems and has the potential to make creating those lists easier for maintainers.

[Atavic]

I makes sense, as the deduplications in uBo make the double entries inactive but the whole lists are still loaded. For me it could be useful for RAM optimization on low-end devices.