Nim: std/sha1 secureHash/secureHashFile() on big files (1.3gb) wrong hashsum.

[enthus1ast]

it seems that std/sha1 produces wrong results for bigger files (i don't know at which filesize it starts to be wrong)

create test file:

writeFile("/tmp/testfile", "a".repeat(1378007516))

gnu sha1sum:

sha1sum /tmp/testfile
# 96c7efea610361ef950e0540cc12017a0fb08c32  /tmp/testfile

python sha1:

hashlib.sha1( ("a" * 1378007516).encode("utf-8")).hexdigest()
# 96c7efea610361ef950e0540cc12017a0fb08c32

nim sha1:

import std/sha1, os
import strutils

echo secureHash "a".repeat(1378007516)
# 0C53192C09BEC0986DCA91BEF107E4BB88F50E4A

echo secureHashFile "/tmp/testfile"
# 0C53192C09BEC0986DCA91BEF107E4BB88F50E4A

since it's a big file (1.3gb) i recommend building with -d:release ;)

smaller files seems to be correct:

nim:

echo secureHash "a".repeat(100)
# 7F9000257A4918D7072655EA468540CDCBD42E0C

python:

hashlib.sha1( ("a" * 100).encode("utf-8")).hexdigest()
# 7f9000257a4918d7072655ea468540cdcbd42e0c

[LemonBoy]

The old implementation was contributed by @onionhammer and it was a port of the smallsha1 library. I've tried this very same calculation using the C++ version and got the same wrong result so it's evident there's something wrong with the implementation itself (and not with the port).

The new implementation I proposed in the linked PR calculates this checksum just fine (and also seems slightly faster at doing so).