Xxhash: Use case check for memcached

[dormando]

Hey,

Thanks for xxh3! I apologize if this is a waste of time for you folks but given that documentation isn't due until v1.0.0 I'd like to make sure I'm not about to make a horrific mistake that I won't notice for six months.

I'm replacing the hash algorithm in memcached again. I have an ideal set of requirements and XXH3 seems to tick all boxes:

• 32bit mask for internal hash table lookup (<= 250 byte keys, typically very small. bits are masked to hash table power-of-two size rather than mixed down. currently murmur3_32)
• 32bit or 64bit (if necessary) checksum for the extstore disk system (not for uniqueness; just wrong-data detection) (larger, often > 500 bytes. currently crc32)
• 64bit hash for feeding into jump consistent hash algorithm and multi-probe consistent hash algorithms for some new code. (<= 250 byte keys, typically small)

Also:

• stable outcomes for x86, x86_64, and ARM/aarch64
• with optimizations for all of the above platforms.

Think I saw xxh3 in 2019 or early 2020 and saw it was marked experimental. It looks like the output is marked as stable and the one-shot + streaming API's are stable but not the secret's stuff. Some earlier notes I found wrt poor outcomes seem to be resolved.

This all accurate? Should I have any reservations for baking xxh3 into apparently every major subsystem in my code?

Thanks!

[Cyan4973]

Hi @dormando ,

yes this is a pretty important question, and I believe your assessment of the situation is completely valid.

Just repeating below, just for the sake of making it "official":

XXH3 is considered stable since v0.8.0, released last year.
Its one shot and streaming APIs are also labelled stable, linking to these symbols will work in future versions of the library.

A few features are still considered "experimental", notably around the secret builder feature. They are separated in the header. Their usage is fine for "ephemeral" data, which is both generated and consumed locally, within the same session.
Experimental features will likely change in the future, so these ones should be avoided for any hash value that is stored or transmitted. It is planned to continue working on them and several of them will become stable around v0.9.0

[dormando]

Awesome! Thanks so much for clarifying. It looks like I can make use of withSeed but don't need the secret stuff, at least right now. I do need stable output for jump/mcp consistent hashing so I will be sure to avoid the experimental features as you advise.

So this hash somehow magically solves all my problems. Excellent work :) I already did a quick test integration for the hash table and while it's a wash for tiny keys there is a measurable difference as they get longer.