Js-ipfs: readableStream broken with raw leaves

[parkan]

• Version: v0.30.0
• Platform: any
• Subsystem: files

Type: Bug-ish

Severity: Moderate

Description

The browser-readablestream demo is a very nice way of handling large video files for seeking. Unfortunately, it appears to break when the files are added using --raw-leaves (required for filestore/urlstore) because it attempts to unmarshal the leaf nodes as unixfs protobufs.

(confirmed my suspicion about this with @achingbrain)

Steps to reproduce the error:

add video file with --raw-leaves, attempt to load with readableStream

Note

This is required for the IA-over-IPFS demo at DWeb Summit on August 1, let me know how I can help get it working by then

[alanshaw]

@parkan do you have an error message or stack trace to help us track this one down?

[parkan]

error is:

decode.js:80 Uncaught TypeError: Cannot read property 'length' of undefined
    at Object.decode (decode.js:80)
    at Function.Data.unmarshal (index.js:101)
    at visitor (file.js:88)
    at eval (index.js:41)
    at drain (index.js:19)
    at eval (index.js:45)
    at dag.get (file.js:117)
    at _get (index.js:143)
    at eval (once.js:12)
    at next (waterfall.js:21)

with zdj7Wc9BBA2kar84oo8S6VotYc9PySAnmc8ji6kzKAFjqMxHS (or QmV3voixu7NXC6VChMYC2AmammEj5EBzsPhEKWmavPo47W in v0)

[daviddias]

@alanshaw this happens when we try to do a unixfs.decode in unixfs-engine and the node is just a raw-leaf. //cc @achingbrain

@parkan can you write some interop tests that catch this?

[parkan]

@diasdavid sure, but I won't have any time before DWeb, can we merge a hotfix and get it covered after aug 3rd?

[parkan]

mfs support for raw-leaves got merged: https://github.com/ipfs/js-ipfs-mfs/pull/5 🎉

what remains to take this over the finish line?

[achingbrain]

ipfs/js-ipfs#1454 needs merging to pull in the new version, currently included in v0.31.0

[parkan]

@achingbrain ah nice didn't see that PR somehow, will +1

[alanshaw]

@parkan https://github.com/ipfs/js-ipfs/pull/1454 was merged - are you able to sanity check against master to verify this is now working?

[parkan]

@alanshaw 👏 👏 👏 will check ASAP, probably first thing tomorrow

[alanshaw]

credit where it's due - I pressed the button, @achingbrain did the work 👏 👏 👏

[parkan]

ok unfortunately I am super pulled in another direction today and can't test this comprehensively (with/without trickle, small/big files, across platforms, etc) but overall this seems to be working well!

only problem encountered so far is mitra reporting an "invalid node type" when using readableStream with small (single block) file @ zb2rhhEncXjn7PnqJ16mzfeug1bqWuupQ3PnkhnWLpAaDatiZ added using the default dag adder (same file added w/trickle @ zdj7Wi4b5djhAqLk5VfeFniwKMpZEFXjWwyiNCrip3keANZao works fine)

obviously trying to stream (or trickle add, for that matter) a single block file doesn't make much sense but maybe this highlights a corner case?

[alanshaw]

only problem encountered so far is mitra reporting an "invalid node type" when using readableStream with small (single block) file @ zb2rhhEncXjn7PnqJ16mzfeug1bqWuupQ3PnkhnWLpAaDatiZ added using the default dag adder

@achingbrain do you have some time to take a quick look at this?

[achingbrain]

The node added via the trickle importer (zdj7Wi4b5djhAqLk5VfeFniwKMpZEFXjWwyiNCrip3keANZao) is a UnixFS node with one link to a raw leaf zb2rhhEncXjn7PnqJ16mzfeug1bqWuupQ3PnkhnWLpAaDatiZ:

$ ipfs dag get /ipfs/zdj7Wi4b5djhAqLk5VfeFniwKMpZEFXjWwyiNCrip3keANZao
{"data":"CAIY/h8g/h8=","links":[{"Cid":{"/":"zb2rhhEncXjn7PnqJ16mzfeug1bqWuupQ3PnkhnWLpAaDatiZ"},"Name":"","Size":4094}]}

If you try to play that leaf directly (as above) and as such use the ipfs.files.catReadableStream API, js-ipfs will try to use the exporter from js-unixfs-engine to get the bytes, expecting a UnixFS node and not a raw node, hence the invalid node type error.

[mitra42]

Thanks @achingbrain - these are legacy hashes, from code that changed, where coming from a cache of the metadata somewhere completely different - and I'm now invalidating and rereading when I see them. Looks like nothing wrong with IPFS's handling presuming leaves like that shouldnt be playable.

[parkan]

yep can confirm that adding that file yields the bare leaf node zb2rhhEncXjn7PnqJ16mzfeug1bqWuupQ3PnkhnWLpAaDatiZ -- is that expected?

[mitra42]

I guess so - that zb2 hash comes from the previous version of urlstore

[parkan]

question meant for @achingbrain mostly, adding the file by any means (with --cid-version=1) results in that

[alanshaw]

Closing this now as 0.31 has been released. If there are any further issues please would you kindly open a new issue for it? Thank you ❤️