Tendermint: use an order-preserving, scan-friendly database key encoding

Created on 14 Mar 2020  路  3Comments  路  Source: tendermint/tendermint

The BlockStore currently uses this encoding for e.g. block metadata keys:

[]byte(fmt.Sprintf("H:%v", height))

This uses alphabetical ordering instead of numerical ordering, such that e.g. block 10 is ordered between block 1 and block 100, not between 9 and 11. This makes it impossible to do efficient range scans - if we e.g. want to prune all blocks between 0 and 1000000 we have to explicitly test for the existence of each and every one rather than simply scan the keys that actually exist. Since this obviously does not scale, one must resort to e.g. short-circuiting a reverse iteration on the first missing key, which is not robust.

The encoding should instead use the big-endian binary representation of the number, or some other order-preserving encoding.

breaking encoding jank perf

Most helpful comment

I believe this is also for the evidence database as well

All 3 comments

This applies to the state database as well.

I believe this is also for the evidence database as well

SQLite uses a varint encoding that preserves order: https://sqlite.org/src4/doc/trunk/www/varint.wiki

It's unclear if this is the same encoding which is used e.g. by the Go binary package: https://golang.org/pkg/encoding/binary/

But if that package preserves ordering as well, we should use varints.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

ddsvetlov picture ddsvetlov  路  3Comments

ebuchman picture ebuchman  路  3Comments

banishee picture banishee  路  3Comments

melekes picture melekes  路  4Comments

melekes picture melekes  路  3Comments