Redisearch: Document expiration support - feature request

[dme-development]

It would be nice to have the document in the RediSearch index expire whenever it expires in the Redis database.

A new command FT.EXPIRE could be a possibilty - as discussed here:
https://groups.google.com/forum/#!topic/redisearch/NACn1ISoWtA

[mnunberg]

(From mailing list)

There are three dimensions to deletion:

1) Removing the actual contents of the document
2) Removing the index entries related to the document (which is separate from the document)
3) Omitting deleted documents from search results.

Implementing such a feature, (1) and (3) are fairly simple to do. (1) can be done because documents are just Redis hashes (at the moment). So expiring/deleting them is built into Redis. For (3), we can add an optional ‘expires’ timestamp into the document, and then compare it with the current time when processing results. Documents which are shown to be expired are then simply marked as deleted and then ignored (perhaps collected by gc).

Removing the index entries is done via garbage collection, which will not at the moment work the same way that Redis’ expiration does (but rather, how FT.DEL does). Since your use case is an Auction platform, lazy garbage collection should not be a problem.

[mnunberg]

The way I had in mind to implement this was:

1. Extend FT.ADD command to have an EXPIRE keyword; with a milliseconds timeout
2. Add metadata into the document (would cost about +8bytes per document for this) about expiration
3. Whenever document is accessed and we do a _Deleted() check, if the document is indeed expired, go and flag it as deleted. This is equivalent to an FT.DEL
4. The GC will eventually delete the contents of the document.
5. The RediSearchCtx will need to be extended to have a now field, to avoid having to recheck the current time

Thoughts?

[zlangbert]

We are looking for this feature as well. We have many short-lived documents that we'd like to automatically expire after a period of time.

@mnunberg I like the idea of adding an EXPIRE keyword to FT.ADD. Potentially naive question about the approach you outlined: if you were to index a document that a future search never hit, would the document still be deleted? i.e there may be futures searches against that index, but not ones that would match the expired document

[itamarhaber]

@zlangbert, in that case, the GC in point 4 should catch that.

[dme-development]

How can we understand the word "eventually" in point 4? Will it expire exactly on time? Or eventually, some undefined time after the expected expiration?

[itamarhaber]

Like Redis' expiry, we'll use active and passive approaches. The active approach relies on accessing the document, recognizing that it has expired, and removing it in one fell swoop. The passive approach relies on the GC (Garbage Collector) to remove expired documents as it traverses the index.

In either case, expiry means that the document will not be returned in results. It does not mean that it is actually removed and deallocated from the index.

[ChristopherLenz]

So this request is nearly 2 years old. Is it still planned? Similar to the creator we need this function for short lived documents. We switched with our logic from a normal Redis to RediSearch, because of the genius query logic. Now with this missing basic feature it is a double-edged sword for us to use which feature. TTL or querying.

There is also no workaround to remove expired documents from the query result? Except having our own expiry-field which is a default query param?

[gkorland]

@dme-development @ChristopherLenz @zlangbert please notice in the upcoming RediSearch 2.0 this feature will be automaticlly supported since RediSearch moves to be based on "standard" Redis Hahs