Redisearch: Feature suggestion: FT.SEARCHSTORE - store results in a Redis Sorted Set

[drittich]

It would be convenient to have the search result IDs persisted to a Sorted Set (optionally with score) for further manipulation such as intersection or union with other non-FT sets.

[dvirsky]

Good idea, thanks!

If you're up for implementing it, I'll be happy to guide you. It really shouldn't be that hard.

[drittich]

Another way to come at this would be the ability to access a Redis set from within a scoring function extension, so I could do intersection, etc., within RediSearch. It looks like the Redis context is not available in that function. Is that a design decision/requirement, or something that can be changed?

[dvirsky]

The problem is that accessing redis data structures in a scoring function is extremely slow (given that it needs to happen for every document, a microsecond is a LOT of time), so I wanted to avoid this.
If this set is not big, you can send it as a query payload, for example.
Can you specify your use case in greater detail? Perhaps there's a more robust solution.

[drittich]

Use case: 1,000,000 indexed documents, but user is only scoped to see 1,000 of them. (Scoping logic is complicated enough that it is not practical to stored the metadata needed for scoping in the FT index.) I have a Redis Set for each user with all of the document IDs that they have access to. So, the goal is to get the FT result set, intersect it with the Redis set of IDs scoped to that user, and then return a page of that data. We're currently trying a Lua solution for this. For some users though, they may be scoped to be able to see 900,000 of those 1,000,000 documents.

[dvirsky]

If it's really in the order of 1000, you can use the INKEYS option of the search and provide the limited ids in advance. (There is no limit for the number of keys in INKEYS but it's not practical to pass on a million keys, 1k or 10k are perfectly acceptable, and will even improve performance)

[dvirsky]

@drittich see http://redisearch.io/Commands/#ftsearch

FT.SEARCH {index} {query} [NOCONTENT] [VERBATIM] [NOSTOPWORDS] [WITHSCORES] [WITHPAYLOADS]
  [FILTER {numeric_field} {min} {max}] ...
  [GEOFILTER {geo_field} {lon} {lat} {raius} m|km|mi|ft]
  -----> [INKEYS {num} {key} ... ] <-------
  [INFIELDS {num {field} ... ]
  [SLOP {slop}] [INORDER]
  [LANGUAGE {language}]
  [EXPANDER {expander}]
  [SCORER {scorer}]
  [PAYLOAD {payload}]
  [SORTBY {field} [ASC|DESC]]
  [LIMIT offset num]

[dvirsky]

@drittich did this solve your use case? in any way, your idea is nice, I'll keep this open

[salt4pommes]

I would try to implement this.

Whats the preferred way?

1. FT.SEARCHSTORE {index} {set-id} {query}
   or as additional Flag to FT.SEARCH:
2. FT.SEARCH {index} {query} STORE {set-id}

The last time I've written something in cpp is a long time ago, but I like the project and want to refresh my skills.

[dvirsky]

Hey @salt4pommes. First of all I'd be glad if you tried, however, I tried thinking about it in greater detail and here are a few things to consider:

1. The structure of a search result doesn't map directly to any redis data structure. for later paging - ideally I would store it in a sorted set, and then we have more than cache - we have a prefetching cache.

2. So ideally I would create a new data type for this, which will allow paging and resorting, and be smart about expiration and not only be time based. This is not trivial even if you know the internals of the code. I'd be more than glad to guide you in doing that, but keep in mind it's going to take a bit of work to get right.

3. Two important things coming up are aggregation pipelines (already working in a different branch) and cursors. Cursors basically mean that we can read the results of a query in an iterative way, which is not that far from a cache. So I need to think if we're not duplicating things here.

4. While the open source version includes just a single shard mode of operation, we have a commercial version that scales to multiple servers, making caching a bit different perhaps.

So basically a bit of design is required here to get this right. The simplest idea would be to store just the top N ids from a search in a sorted set, and then we can read them and load the documents, and use it for paging as well as just plain caching.

I'd love to hear your thoughts on all these :)

[salt4pommes]

Hey Dvir!

I think the main idea behind this, is to store the results for later use or prevent duplicated search overhead.
Sure the fastest and simplest way is to store the results in a sorted set,
but I like the idea of the new data type and the possibilities it could offer.

As you said it is more a cache and a cache requires more maintenance.
That means, if we update or delete a document which is contained in the cache,
then we should also purge all caches which contains the document id.

So we need to maintain a ttl and a type of garbage collection when a document gets removed or updated.

I do not think that this feature will conflict with the cursors.
We could use the cached/stored document ids to fast fetch document data without a complex search and
we are also able to aggregate or query with new filters using the document ids from the cache.

Just another idea if we want a self maintaining cache:
When the cache gets deprecated by some changes in the index, we could save the query in the new data type to fetch the new results and update the cache automatically.

[dvirsky]

Alright, let me think about it a bit and I'll draft a detailed design for the cache and invalidation process. Thanks!