Go-ipfs: buzhash adds trailing zero-sized chunk

[sinkuu]

Version information:

go-ipfs version: 0.5.0-dev-49afb3b5b-dirty
Repo version: 7
System version: amd64/linux
Golang version: go1.14

Description:

Steps to reproduce:

$ yes "foo" | head -n 40000 > test
$ ipfs add test --chunker=buzhash --pin=false
added QmX4kkx8AWgy6GmF9zVtgPVta5GtpEJpPTtatexZ8xrwBs test
 156.25 KiB / 156.25 KiB [============================================================================================================================================================================] 100.00%
$ ipfs dag get QmX4kkx8AWgy6GmF9zVtgPVta5GtpEJpPTtatexZ8xrwBs
{"data":"CAIYgOIJIIDiCSAA","links":[{"Cid":{"/":"QmcNJ4CKYGuya2ndNr9ymV7oQsT4S69SdwmTzaiW7c9vAC"},"Name":"","Size":160014},{"Cid":{"/":"QmaRwA91m9Rdfaq9u3FH1fdMVxw1wFPjKL38czkWMxh3KB"},"Name":"","Size":8}]}
$ ipfs add test --chunker=buzhash --pin=false --raw-leaves
added QmYyFBARF9M2bwfBQ7y5kNZGJVJWmktpx2tapZy6gWoBXC test
 156.25 KiB / 156.25 KiB [============================================================================================================================================================================] 100.00%/
$ ipfs dag get QmYyFBARF9M2bwfBQ7y5kNZGJVJWmktpx2tapZy6gWoBXC
{"data":"CAIYgOIJIIDiCSAA","links":[{"Cid":{"/":"bafkreigqq7maj23te3dkn4j76n7haab7lgmdz5fztubrdgsv44k4vyipsa"},"Name":"","Size":160000},{"Cid":{"/":"bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"},"Name":"","Size":0}]}

Adding a file with --chunker=buzhash --raw-leaves results in two blocks, whose sizes are 160000 and 0 respectively.

EDIT: The trailing block with non-raw-leaves add contains no actual data too, so this is not related to raw-leaves.

[ribasushi]

Definitely a bug. I will try to dig into it before next Monday ( reimplementing the buzzhash convergence in another tool is something I was going to work on tomorrow )

[Stebalien]

@ribasushi status?

[ribasushi]

@Stebalien I haven't dug into this yet: it can't go into 0.5 anyway as it will be changing hashes. Or do we actually want this for 0.5 ( in which case I will speed it up ).

[Stebalien]

Well, (a) this is a bug, (b) this feature is experimental, and (c) it has never been released. So I'd like to fix it before the release and before users start to actually use it.

[ribasushi]

(c) it has never been released

I did not realize this, apologies. Fix will be here by tomorrow.