Xxhash: Add argument for recursive hashing

[vovcacik]

Not a big deal on Unix, but kinda pita on Windows.

Current workaround:

(it's not that complicated, bit chatty)

@echo off
setlocal enabledelayedexpansion

set EXE=xxhsum.exe

:: This batch adds "-r <folder>" argument for recursive hashing,
:: otherwise acts transparent.
if "%~1"=="-r" (
  :: Strip "-r" and folder name from arguments that we are going to pass to xxhsum.exe
  set ARGS=%*
  set ARGS=!ARGS:%~1=!
  set ARGS=!ARGS:%~2=!

  :: Process root dir
  call :process-dir "%~2" "%~2"

  :: Process subdirs
  for /D /r "%~f2" %%d in (*) do (
    :: Dont feed xxhsum.exe with folders that contain no files
    for /F %%_ in ('dir /b /a:-d "%%d" 2^>nul') do (
      :: Process subdir only once (the above for loop will cycle once for each file in %%d)
      if "%%d" neq "!LAST!" (
        set LAST=%%d
        call :process-dir "%%d" "%~2"
      )
    )
  )
) else (
  :: Be transparent
  %EXE% %*
)

goto :eof

:: Usage: process-dir <the-dir-being-processed> <the-dir-from-r-arg>
:process-dir
  :: Get relative path to dir
  set REL=%~f1
  set REL=!REL:%~f2=%~2!
  %EXE% %ARGS% "%REL%\*"
goto :eof

[vovcacik]

Also see #87

[t-mat]

• Plan A: Implement "Read filename(s) from stdin" switch.
  
  
  
  • If we'll have such functionality, we'll be able to use it by
    
    dir /s /b **TARGET-DIR** | xxhsum --read-filenames-from-stdin > C:\my-result
  
  
  • It's not straightforward. But it allows us to cooperate with other shell commands.
  
  
  • We may hit EOF problem of stdin.
  
  
• Plan B: Implement opendir()-ish functionality. For example, https://github.com/cxong/tinydir
  
  
  
  • It's straightforward.
    
    xxhsum --show-hash-recursive **TARGET-DIR** > C:\my-result
  
  
  • Since there's no "perfect" opendir() other than shell, it may cause other problems. e.g Permission, race condition, networking, Unicode/MBCS path name, etc.
  
  
• Plan C: Do not implement anything.
  
  
  
  • Just use
    
    for /R **TARGET-DIR** %f in (*) do ( xxhsum %f >> C:\my-result )
  
  
  • cmd.exe must die :finnadie:
  
  
  • for PowerShell use
    
    Get-ChildItem **TARGET-DIR** -Recurse | where { !$_.PSIsContainer} | ForEach-Object {xxhsum $_.FullName} | Out-File C:\my-result
  
  

Personally, I suppose "Plan C" may satisfy 70% of usage and I believe we should compromise with it.
But "Plan A" also would be nice since it's constructive and easy to implement without platform dependent functionality and/or external dependency.

[data-man]

Plan D (the best :-): add xxHash to RHash. See issue #26.

[t-mat]

@data-man I'm really sorry but I can't read that issue 😢
But RHash looks good.

[data-man]

@t-mat

Sorry, machine translated:

Please add support for xxHash algorithm is a step into the future, available in 202 lines of code. Judging by the tests, runs faster CRC32, which is useful when calculating checksum *.iso, *.mkv, databases, and similar large files. Today it is in HashOnClick, but, alas, for the $$$.

[Cyan4973]

Plan A looks interesting, it's also a fairly new feature, existing utilities like md5sum do not support it as far as I know.
The "without platform dependent functionality" tag feels premature to me : we would need to be sure that dir output can be parsed safely, in a universal way, on all systems.
Alternatively, it could be tagged as working on only a few tested systems, and gracefully refuse to work on others.

As stated in #87, another potential solution could be to support a "recursive mode", typically using -r command, including all files in selected and included directories. It's also not supported by md5sum, but it's a well-known capability of compression tools like gzip, xz, zstd, lz4 etc., and we've got some code to make it work. The existing code is "reasonably portable" : it indeed depends on some OS-related libraries for directory discovery, but works well on a lot of OS, and is automatically disabled on unsupported OS.

[vovcacik]

@t-mat
The for /R **TARGET-DIR** %f in (*) do ( xxhsum %f >> C:\my-result ) is fairly straightforward but awfully slow, thats why I am using for /D /r "%~f2" %%d in (*) in my snippet.

[frenchiveruti]

A variant for WLS (using debian terminal on windows 10):

IFS=$'\n'; for file in *; do ./xxhsum.exe $file >> hashes.txt ;done

Assuming:

1. Terminal is inside the folder with the files to hash.
   
   
   
   1. xxhsum is in same dir as files to hash (that's why ./xxhsum.exe is used)
   
   
   2. Be aware that xxhsum hash will make it into the hashes.txt file.
      
      1.IFS=$'\n' makes the newline the only separator, so you can have whitespaces
   
   

Another method:
Select files to hash, copy all the routes using "shift+right click", paste into a txt like stdin.txt
remove leading and tailed quotations "" using "search and replace" on your favorite text editor.
save file as unix (LF) format. (windows format CR LF gives errors)

Run this command:
bash for file in $(cat stdin.txt); do ./xxhsum.exe $file >> hashes.txt; done
Now the hashes file contains just the files that you wanted to hash.

[sergeevabc]

xxhsum 0.8.0 behaves oddly even without recursive traversal.

$ xxhsum -H128 *.* 
xxhsum: $RECYCLE.BIN: Is a directory                                                                                       
94796d676519ea4c53e10e249f0ef219  2017-03-18 суббота.md                                                             
f91167e2b1937e78a6931748d1df4df3  idea-icon.png                                                                            
xxhsum: Sync: Is a directory                                                                                               
xxhsum: System Volume Information: Is a directory                                                                          
07c9eef2123cff8ed27d1e0a7eaddec1  SYSTEM.LOG1                                                                              
99aa06d3014798d86001c324468d497f  SYSTEM.LOG2                                                                              
df6f507eacafd3219612174a23ac4312  SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TM.blf                                      
18ae6eb42537764911b1248ee9d947a6  SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TMContainer00000000000000000001.regtrans-ms 
0d9f6acee70bd714fd444a4e0c1d4c46  SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TMContainer00000000000000000002.regtrans-ms 
Error: Could not open 'Загрузки': Permission denied.                                                                       

I asked to calculate hashes of files, so why it mentions dirs in the output and why the Cyrillic-named dir is suddenly not accessible?

[t-mat]

I asked to calculate hashes of files, so why it mentions dirs in the output

It's caused by the shell which you're using. In this case, xxhsum does nothing about wildcard. Usually, *nix flavor shell expands wildcard (globbing) such as *.* to filesystem entries including directory.
For example, the following command shows actual expanded arguments.

echo *.*

why the Cyrillic-named dir is suddenly not accessible?

I have no idea. On my terminal, xxhsum for directory which has Cyrillic name shows the following message as intended.

$ xxhsum *
xxhsum: Загрузки: Is a directory

But since error message suggests issue about permission, you can try some method to figure out what is happening

• Use whoami command to make sure you're using your account or not.
• Retrieve some infomation about the directory by ls -ld Загрузки.

[sergeevabc]

@t-mat, I am a Windows 7 administrator with its shell and a few ported UNIX utils, nothing fancy.

$ whoami
...\alexander

$ ls -ld Загрузки
drwxr-xr-x 1 Alexander None 0 мар 19 05:30 Загрузки

Actually, 'denied' issue seems not to be related to permissions as follows

$ mkdir Birds Vögel
$ xxhsum -H128 *.*
...
xxhsum: Birds: Is a directory
Error: Could not open 'Vögel': Permission denied.
                        ^------------------------------------------- the crux, perhaps?