Go-ipfs: Clarify handling blocks that aren't serialized objects

[ianopolous]

Currently, arbitrary data can be stored and retrieved with block.{put/get}. However, the resulting blocks cannot be pinned, as pinning tries to deserialize an object.

It would seem either blocks should be enforced to always be serialized objects, or pinning should be able to handle non-object blocks.

[jbenet]

I think:

blocks should be enforced to always be serialized objects

[whyrusleeping]

Once CIDv1 lands, you will be able to address raw blocks. That should solve this issue

[robcat]

@whyrusleeping

Once CIDv1 lands, you will be able to address raw blocks. That should solve this issue

I don't see how the issue will solve itself.
Will the blockstore also store separately the <mcodec> mentioned in ipfs/specs#130?

[whyrusleeping]

@robcat the mcodec is embedded in the cid of the block, so yeah, the blockstore will technically be storing the mcodec.

[robcat]

@whyrusleeping
But how exactly?

Example: A CID with _unary_ encoding gets pulled. I'm expecting the blockstore to store it in binary form on my (binary) disk and to be able to retrieve it with the unary CID.

[Kubuxu]

@robcat try echo 'raw data' | ipfs block put -f raw -.

[robcat]

Ok I got it, the block gets named <version><mcodec><mhash> (in base32). This solves this issue.

But in my view the blockstore shouldn't be aware of the CID version and codec, these exist at a higher level (i.e. the object layer, where the deserialization happens).

[Kubuxu]

There is reason for that, I will allow @whyrusleeping to elaborate.

[whyrusleeping]

@robcat Short answer, it could go either way.

Long answer: We could potentially make the blockstore not care about formats and manage to handle things for the most part without them. The reason we opted for storing formats is that it makes everything we store on disk self describing. You can, without any other context, look at blocks and be able to tell what they are. This is really helpful for writing tools like ipfs-see-all that crawl the repo and do analytics on it. Its also somewhat required for blockstore.AllKeysChan to work correctly. (otherwise, it would have to return just generic multihashes and we would be left making guesses about the types of things)

[mib-kd743naq]

Elaborating on https://github.com/ipfs/go-ipfs/issues/2067#issuecomment-262775669 a bit: there must be some sort of flag to allow hash specification. Currently -f raw assumes sha2-256 and that's that...

[whyrusleeping]

@mib-kd743naq mmm, yeah. Great point.

[whyrusleeping]

We can now add 'raw blocks' to ipfs that can be pinned as normal