Xxhash: XXH3 clarification

[centminmod]

I am looking at utilising xxhsum for file backup checksum/verifications but need clarification for enabling XXH3 support. It's bloody fast already compared to md5 checksums. Thanks !

To access experimental prototypes, one need to unlock their declaration using macro XXH_STATIC_LINKING_ONLY

How exactly is that done ? Is it something done at xxHash compile time. An example of the steps in documentation would help clarify things alot :)

cheers

George

[Cyan4973]

Hi @centminmod ,

in the user's source code :

#define XXH_STATIC_LINKING_ONLY
#include "xxhash.h"

For completeness, note that there are also 2 other alternatives :

#define XXH_INLINE_ALL
#include "xxhash.h"

On top of enabling experimental functions, this will inline them (it's implied).
This can be very useful when the use case involves hashing primarily small data (< 100 bytes), __especially if the size can be expressed as a constant__ (which is not always possible).
Inlining is not useful if the use case involves hashing primarily large data (checksumming).

Finally :

#include "xxh3.h"

This is technically equivalent to previous method.
It's only available now. In the future, xxh3.h is expected to be merged within xxhash's main implementation.

[centminmod]

Thanks for the info, still a bit confused as I see this is already in xxhash code ? Where is this placed exactly ?

#define XXH_STATIC_LINKING_ONLY
#include "xxhash.h"

grep -C3 -rin 'XXH_STATIC_LINKING_ONLY' xxhash.c xxhash.h xxhsum.c 
xxhash.c-115-
xxhash.c-116-#include <limits.h>   /* ULLONG_MAX */
xxhash.c-117-
xxhash.c:118:#define XXH_STATIC_LINKING_ONLY
xxhash.c-119-#include "xxhash.h"
xxhash.c-120-
xxhash.c-121-
--
xxhash.h-100- *  It's not useful to compile and link it as a separate object.
xxhash.h-101- */
xxhash.h-102-#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
xxhash.h:103:#  ifndef XXH_STATIC_LINKING_ONLY
xxhash.h:104:#    define XXH_STATIC_LINKING_ONLY
xxhash.h-105-#  endif
xxhash.h-106-#  if defined(__GNUC__)
xxhash.h-107-#    define XXH_PUBLIC_API static __inline __attribute__((unused))
--
xxhash.h-300-
xxhash.h-301-
xxhash.h-302-
xxhash.h:303:#if defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742)
xxhash.h-304-#define XXHASH_H_STATIC_13879238742
xxhash.h-305-/* ================================================================================================
xxhash.h-306-   This section contains declarations which are not guaranteed to remain stable.
--
xxhash.h-590-}
xxhash.h-591-#endif
xxhash.h-592-
xxhash.h:593:#endif /* defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) */
--
xxhsum.c-56-
xxhsum.c-57-#include "xxhash.h"
xxhsum.c-58-
xxhsum.c:59:#define XXH_STATIC_LINKING_ONLY   /* *_state_t */
xxhsum.c-60-#include "xxhash.h"    /* note : intentional double include, for validation.
xxhsum.c-61-                        * this test ensures that xxhash.h can be included in any order. */
xxhsum.c-62-

On Intel Core i7 4790K 4C/8T server with CentOS 7.7 64bit

xxhsum -b
xxhsum 0.7.2 (64-bits x86_64 + AVX2 little endian), GCC 8.3.1 20190311 (Red Hat 8.3.1-3), by Yann Collet 
Sample of 100 KB...        
XXH32                    :     102400 ->   102400 it/s (10000.0 MB/s) 
XXH32 unaligned          :     102400 ->   102400 it/s (10000.0 MB/s) 
XXH64                    :     102400 ->   307200 it/s (30000.0 MB/s) 
XXH64 unaligned          :     102400 ->   307200 it/s (30000.0 MB/s) 
XXH3_64b                 :     102400 ->   360819 it/s (35236.2 MB/s) 
XXH3_64b unaligned       :     102400 ->   357211 it/s (34883.9 MB/s) 
XXH3_64b seeded          :     102400 ->   379261 it/s (37037.2 MB/s) 
XXH3_64b seeded unaligne :     102400 ->   379260 it/s (37037.1 MB/s) 
XXH128                   :     102400 ->   345170 it/s (33708.0 MB/s) 
XXH128 unaligned         :     102400 ->   345170 it/s (33708.0 MB/s) 
XXH128 seeded            :     102400 ->   341335 it/s (33333.5 MB/s) 
XXH128 seeded unaligned  :     102400 ->   341334 it/s (33333.4 MB/s) 

So XXH3 already enabled ?

[Cyan4973]

Yes, XXH3 as a binary generated function is present, and xxhsum does include it.

From a libxxhash perspective, xxhsum is a user program. Any other user program would have to do the same to access XXH3 declarations.

The reason XXH3 prototypes are behind this flag is because it's still experimental,
meaning the values it generates are not yet "stable" and may change in future versions.
So we don't want users to start storing these values and be surprised that it doesn't work any more when compared with a future version. Forcing users to #define XXH_STATIC_LINKING_ONLY in their source code is a way to make them realize and acknowledge the experimental nature of this API.

[centminmod]

I see. Good to know. My intended use of xxhsum is for moment in time checksum verifications for a data migration tool i.e. backup data files, generate checksum, transfer to new server, verify checksum and if ok, restore data and then done. Which is all done within same session. So no long term storage of backup/checks. Always intended to make use of xxHash for this due to it's speed :)

So how does xxhsum on the command line leverage XXH3, i.e. for XXH128 it's via -H2. But for XXH3 ?

Though if I am reading xxhsum -b output correctly from above i7 4790K Haswell and below results, for most of the systems I have tested XXH3 isn't any much faster than XXH128 and even sometimes slower. Right now my data migration script is using xxhsum -H2 by default.

On Intel Xeon E-2276G Coffee Lake 6C/12T

xxhsum -b
xxhsum 0.7.2 (64-bits x86_64 + AVX2 little endian), GCC 8.3.1 20190311 (Red Hat 8.3.1-3), by Yann Collet 
Sample of 100 KB...        
XXH32                    :     102400 ->   102400 it/s (10000.0 MB/s) 
XXH32 unaligned          :     102400 ->    95775 it/s ( 9353.0 MB/s) 
XXH64                    :     102400 ->   307200 it/s (30000.0 MB/s) 
XXH64 unaligned          :     102400 ->   307200 it/s (30000.0 MB/s) 
XXH3_64b                 :     102400 ->   472616 it/s (46153.9 MB/s) 
XXH3_64b unaligned       :     102400 ->   470157 it/s (45913.7 MB/s) 
XXH3_64b seeded          :     102400 ->   529655 it/s (51724.1 MB/s) 
XXH3_64b seeded unaligne :     102400 ->   529657 it/s (51724.3 MB/s) 
XXH128                   :     102400 ->   529655 it/s (51724.1 MB/s) 
XXH128 unaligned         :     102400 ->   529657 it/s (51724.3 MB/s) 
XXH128 seeded            :     102400 ->   529657 it/s (51724.3 MB/s) 
XXH128 seeded unaligned  :     102400 ->   531304 it/s (51885.2 MB/s) 

On AMD Ryzen 3800X 8C/16T

xxhsum -b
xxhsum 0.7.2 (64-bits x86_64 + AVX2 little endian), GCC 8.3.1 20190311 (Red Hat 8.3.1-3), by Yann Collet 
Sample of 100 KB...        
XXH32                    :     102400 ->   102400 it/s (10000.0 MB/s) 
XXH32 unaligned          :     102400 ->    86293 it/s ( 8427.1 MB/s) 
XXH64                    :     102400 ->   172585 it/s (16854.0 MB/s) 
XXH64 unaligned          :     102400 ->   172585 it/s (16854.0 MB/s) 
XXH3_64b                 :     102400 ->   525939 it/s (51361.3 MB/s) 
XXH3_64b unaligned       :     102400 ->   512002 it/s (50000.2 MB/s) 
XXH3_64b seeded          :     102400 ->   529655 it/s (51724.1 MB/s) 
XXH3_64b seeded unaligne :     102400 ->   512002 it/s (50000.2 MB/s) 
XXH128                   :     102400 ->   520680 it/s (50847.6 MB/s) 
XXH128 unaligned         :     102400 ->   513885 it/s (50184.1 MB/s) 
XXH128 seeded            :     102400 ->   520680 it/s (50847.7 MB/s) 
XXH128 seeded unaligned  :     102400 ->   520680 it/s (50847.6 MB/s) 

On AMD Ryzen 3600 6C/12T

xxhsum -b
xxhsum 0.7.2 (64-bits x86_64 + AVX2 little endian), GCC 8.3.1 20190311 (Red Hat 8.3.1-3), by Yann Collet 
Sample of 100 KB...        
XXH32                    :     102400 ->    80844 it/s ( 7894.9 MB/s) 
XXH32 unaligned          :     102400 ->    80844 it/s ( 7894.9 MB/s) 
XXH64                    :     102400 ->   307200 it/s (30000.0 MB/s) 
XXH64 unaligned          :     102400 ->   161686 it/s (15789.6 MB/s) 
XXH3_64b                 :     102400 ->   495484 it/s (48387.1 MB/s) 
XXH3_64b unaligned       :     102400 ->   487621 it/s (47619.2 MB/s) 
XXH3_64b seeded          :     102400 ->   503607 it/s (49180.3 MB/s) 
XXH3_64b seeded unaligne :     102400 ->   487621 it/s (47619.2 MB/s) 
XXH128                   :     102400 ->   503607 it/s (49180.3 MB/s) 
XXH128 unaligned         :     102400 ->   495484 it/s (48387.1 MB/s) 
XXH128 seeded            :     102400 ->   503607 it/s (49180.3 MB/s) 
XXH128 seeded unaligned  :     102400 ->   492545 it/s (48100.1 MB/s) 

[easyaspi314]

That is kinda to be expected. XXH128 is just the 128-bit variant of XXH3. The main loop only differs in that it adds an extra pshufd to cross pollinate better (since otherwise XXH128 would pretty much be 2 independent 64-bit hashes).

And pshufd is ludicrously fast on modern x86 processors

Pseudo-C Algorithm:

for (size_t i = 0; i < ACC_NB; i++) {
    uint64_t input_val = input[i];
    uint64_t secret_val = secret[i];
    if (is_xxh128) {
        acc[i ^ 1] += input_val; // add to paired lane
    } else {
        acc[i] += input_val;
    }
    uint64_t mixed = input_val ^ secret_val;
    acc[i] += (mixed & 0xFFFFFFFF) * (mixed >> 32);
}

[centminmod]

That is kinda to be expected. XXH128 is just the 128-bit variant of XXH3.

Ah thanks @easyaspi314 for the clarification :)

[Cyan4973]

how does xxhsum on the command line leverage XXH3, i.e. for XXH128 it's via -H2. But for XXH3 ?

There is no way. And this is intentional.
XXH3_64bits() can only be accessed through the benchmark module, but its output will not be printed.

The most important reason is to avoid confusion. A 64-bit checksum produced and controlled by xxhsum is meant to be generated using XXH64(). If we start to have 2 different ways to produce checksums of same length, it's a recipe for confusion.

A second more minor reason is that the speed gains achieved by XXH3 are unlikely to matter in a I/O scenario, which is what xxhsum is about, because most (all?) I/O can't keep up with XXH64() already. The benefits of XXH3 are more relevant in presence of in-memory scenarios, such as bloom filters, hash / probe tables, or as part of a pipeline, such as checksum of medium size data already present in cache.
Yes, XXH3 is effectively faster than RAM, and this hardly matters for I/O bounds data flows.

[easyaspi314]

A third reason is that we are still trying to figure out some seed-dependent collisions with the mid-range keys on XXH3_64b. For hash tables, it is fine if you use a good seed, but for checksumming, where we always use the same seed, it is unviable.

[centminmod]

Thanks @Cyan4973 @easyaspi314 for the further clarification and all the hard work !