Badger: Add NextSequence method in Badger

[tonyalaribe]

In bleve there is bucket.NextSequence() which is very useful in storing lists. Can someone share any strategies for storing lists in badger? I want to be able to add items with ease.

[peterstace]

Badger doesn't do anything like this natively, but it's possible to build something on top.

The best approach would really depend on the types of operations you wish to perform on the list (this is the same question you would have to ask when designing any data structure).

One approach:

Badger keeps its key/value pairs sorted internally, so you could take advantage of that for fast traversal. Iterating over a sorted range of keys in badger is very fast. The keys could be the indexes into the list, and the value could be what you're storing in the list. For adding a new element in between two existing elements, you just have to generate a new index that is between the two elements. If you wanted to insert a new element between two indexes that are numerically adjacent, you can always just extend the length of the key. E.g. you want to insert between 0x4a and 0x4b, you can make a new key made from two bytes, 0x4a 0x77 which would be sorted in between.

[manishrjain]

NextSequence() only gives you a unique monotonically increasing id from my understanding. How does that correspond to storing lists?

[tonyalaribe]

What is getting me confused, is how to generate autoincrementing keys, so I can have something like:
product:1 ==> {product:{}}
product:2 ==> {product:{}}
product:3 ==> {product:{}}

[manishrjain]

This can be easily done outside of Badger, by using a lease based system to "lease out" ids in batches (of say 10000), writing the max lease to Badger, using an atomic counter in memory to hand out ids, until you reach the max lease, then repeating the process. This would ensure that it is crash resistant.

We could also bake this in as a simple API, for ease of use of users switching away from BoltDB. @deepakjois ?

[svperfecta]

I think this is probably worth it. While it might be easy, I think its one of those things that is out of scope for someone who isn't building systems like this and simply wants to use it.

Also, if they get it wrong, or use something else instead they'll pay a performance penalty. For instance, perhaps a simple approach is to use a UUID for all IDs. Nice, but if they do that they pay for it later when they write.

[manishrjain]

CC: @deepakjois . Let's prioritize this.

[rebootcode]

+1 - This is so much needed

[manishrjain]

Done. #346.