Tribler: Redesign of the Search/Channels feature
Use cases.
We strive to provide the same kind of service that is provided by:
- Highly organized repositories of torrents (trackers)
- Scientific articles repositories.
- Music playing services, like Spotify.
- Video playing services, like Youtube.
- Scanned books archives, like Google Books.
- Internet libraries
On fuzzy searches.
We do not try to become a distributed Google. Google is good at finding the knowledge that is relevant to words. It is used when the user _does not know_ what he or she is looking for. Our user _always_ knows what he or she wants. Therefore, we should disable metadata (or torrent contents) search by default, and only search by the torrent name. On the other hand, the user should be able to use metadata to _refine_ the search results. Like, "sort by date of creation" or "sort by tag/channel", etc. This model already works perfectly for all use cases indicated above (except Youtube).
Data structurization.
We must be able to replicate the basic information organization structures that are used by our example use cases:
- Tracker forum : tree-like structure, traversable both up and down. When the user finds a torrent/channel, he or she should be able to see its immediate parent/children nodes, and the "root" of the hierarchy. Thus, nested channels.
- Scientific articles database: a single record in the database should be able to hold an arbitrary amount of metadata fields, and each field must be able to be used as a base for a hierarchy. For example, top-level owner/channel "ScientificArticles" could provide "pseudochannels" for "authors" "journals" "years" "tags", etc. These are essentially selecting the metadata field that would be used to build the structure "just in time". Advanced users would be able to select several criteria at once, to find the intersection of metadata sets. Thus nested channels can fully exploit the features of the underlying relational database.
- Music search: should be fast and simple. This could be only achieved by implementing a distributed cloud-based search, with some data cached at user's device. We can't get new user do download XX Gigabytes search database, especially for mobile usage (that is basically everything for music playing). One search query should take 5-30 seconds, and it's results should include related info (like, the search for a single song should get that song _and_ reveal the link to the whole album). To solve this problem, at some point we could introduce search mining. Fast hosts producing highly relevant search results get credit.
- Libraries and book archives fall somewhere between scientific articles and tracker forum.
It is important to note that a single instance of the system is not required to serve all of these features at once. Instead, it should provide building blocks, or modules, to compose the use cases as necessary.
For an average user a single database could be used to serve every type of content, but, e.g., users interested in building a huge scientific database should be able to set up a separate database instance optimized for scientific articles.
Constraints on design choices
The design should consist of several _completely independent_ parts:
- Networking subsystem
- Data transfer protocol
- Trustworthy gossip protocol
- Storage subsystem
- Database/store
- Cache management / mining algorithm
- Search subsystem
- Distributed search protocol
- Distributed search algorithm
- Data distribution algorithm
- Metadata format
It is very important that we do not make any of these parts depend on specific implementation of one another. We must be able to design the system in such a way, that it would be trivial to exchange one implementation of some part for another one.
User experience requirements:
- Search results should appear faster than the user can process them.
Implementation requirements:
- Scalability (should only sync neighbors, and not the whole network)
ichorid
All 117 comments
Mostly duplicate of #2455
synctext
on 7 May 2018
Frequency distribution of words in all natural languages follow Zipf's law. Speakers generally optimize the term (word, phrase ...) that they assign to the denotate (the physical object/idea ...) for efficiency of communication.
I.e. when I ask someone where can I find "John Doe", the combination of words "John"+"Doe" uniquely identifies the person by that name, at least in the context of the conversation. On the other hand, if I ask where can I find "blue table", this could mean either "_a_ blue table" or "_the_ blue table".
- "_A_ blue table" is a general search query, and the person would probably point me to a furniture store, or try to sell me a table (if he is in the furniture buisness). "_A_ blue table" is a general search query, Google-style.
- "_The_ blue table" is an exact search, and the person would probably be confused if there is no special thing in her mind that is denoted as "the blue table". Otherwise, the person would point me to it directly. "_The_ blue table" is a search for the exact infohash (or some special search token), or search in local context.
So, the best thing about natural languages is that humans use them to name things, like movies, game, music, torrents... This means that when someone does a search query for a thing he already knows _exists_, he addresses it by the terms that identify that thing _uniquely enough_ in the database of human knowledge. And when someone just wants something in some genre - he names it by the name of the genre. Everything in between ("that movie with this guy with cool beard") we leave to Google.
This means we could use the same search mechanism for both exact queries ("Bla-bla-avenger-name 2035") and for genre search ("western 2035").
ichorid
on 3 Jun 2018
FASD: A Fault-tolerant, Adaptive, Scalable, Distributed Search Engine (2003 master thesis) - almost exactly what I came up with. No works by that author afterward, though :disappointed:
Associative search in peer to peer networks: Harnessing latent semantics (2006)
A nice survey on the topic of decentralized search: "Survey of Research towards Robust Peer-to-Peer Networks: Search Methods"
Hmmm... This stuff seems to be beaten to death in the first decade of 00`s. Still, no one truly succeeded.
ichorid
on 3 Jun 2018
ichorid
on 4 Jun 2018
ichorid
on 5 Jun 2018
xoriole
on 8 Jun 2018
ichorid
on 9 Jun 2018
synctext
on 9 Jun 2018
ichorid
on 9 Jun 2018
ichorid
on 9 Jun 2018
synctext
on 9 Jun 2018
ichorid
on 10 Jun 2018
synctext
on 13 Jun 2018
devos50
on 13 Jun 2018
ichorid
on 13 Jun 2018
synctext
on 13 Jun 2018
ichorid
on 14 Jun 2018
synctext
on 14 Jun 2018
ichorid
on 14 Jun 2018
qstokkink
on 15 Jun 2018
synctext
on 15 Jun 2018
ichorid
on 16 Jun 2018
qstokkink
on 17 Jun 2018
ichorid
on 17 Jun 2018
devos50
on 22 Jun 2018
qstokkink
on 22 Jun 2018
ichorid
on 22 Jun 2018
ichorid
on 22 Jun 2018
ichorid
on 28 Jun 2018
devos50
on 28 Jun 2018
ichorid
on 5 Jul 2018
qstokkink
on 5 Jul 2018
ichorid
on 5 Jul 2018
synctext
on 9 Jul 2018
synctext
on 5 Sep 2018
ichorid
on 21 Sep 2018
ichorid
on 24 Oct 2018
ichorid
on 6 Jan 2019
ichorid
on 10 Feb 2019
Dmole
on 31 Mar 2019
ichorid
on 31 Mar 2019
Dmole
on 31 Mar 2019
ichorid
on 31 Mar 2019
Dmole
on 31 Mar 2019
ichorid
on 31 Mar 2019
Dmole
on 31 Mar 2019
synctext
on 31 Mar 2019
vi
on 31 Mar 2019
Dmole
on 31 Mar 2019
vi
on 31 Mar 2019
vi
on 31 Mar 2019
Dmole
on 31 Mar 2019
Dmole
on 31 Mar 2019
synctext
on 1 Apr 2019
ichorid
on 19 Jul 2019
ichorid
on 22 Jul 2019
synctext
on 23 Jul 2019
ichorid
on 23 Jul 2019
ichorid
on 14 Sep 2019
ichorid
on 1 Oct 2019
ichorid
on 5 Oct 2019
ichorid
on 6 Oct 2019
ichorid
on 15 Feb 2020
ichorid
on 16 Feb 2020
ichorid
on 16 Feb 2020
ichorid
on 16 Feb 2020
ichorid
on 16 Feb 2020
qstokkink
on 17 Feb 2020
ichorid
on 20 Feb 2020
ichorid
on 5 Mar 2020
ichorid
on 19 May 2020
ichorid
on 19 May 2020
vi
on 19 May 2020
ichorid
on 19 May 2020
ichorid
on 20 May 2020
Dmole
on 20 May 2020
ichorid
on 20 May 2020
Dmole
on 20 May 2020
ichorid
on 23 May 2020
ichorid
on 23 May 2020
ichorid
on 24 May 2020
synctext
on 26 May 2020
ichorid
on 31 May 2020
synctext
on 3 Jun 2020
ichorid
on 3 Jun 2020
synctext
on 8 Jun 2020
ichorid
on 29 Jun 2020
ichorid
on 29 Jun 2020
synctext
on 1 Jul 2020
synctext
on 14 Jul 2020
synctext
on 23 Jul 2020
ichorid
on 24 Jul 2020
ichorid
on 9 Aug 2020
qstokkink
on 10 Aug 2020
ichorid
on 18 Aug 2020
synctext
on 26 Aug 2020
ichorid
on 26 Aug 2020
synctext
on 26 Aug 2020
ichorid
on 26 Aug 2020
ichorid
on 1 Sep 2020
synctext
on 11 Sep 2020
ichorid
on 11 Sep 2020
ichorid
on 11 Sep 2020
synctext
on 14 Sep 2020
ichorid
on 14 Sep 2020
devos50
on 14 Sep 2020
ichorid
on 14 Sep 2020
ichorid
on 14 Sep 2020
synctext
on 14 Sep 2020
synctext
on 17 Sep 2020
devos50
on 17 Sep 2020
qstokkink
on 17 Sep 2020
ichorid
on 17 Sep 2020
ichorid
on 26 Sep 2020
ichorid
on 26 Sep 2020
ichorid
on 29 Oct 2020
ichorid
on 7 Nov 2020
Related issues
devos50
路
4Comments
ichorid
路
9Comments
devos50
路
7Comments
ichorid
路
8Comments
synctext
路
11Comments