Go-ipfs: Add batch operation to `ipfs block put`

[ianopolous]

Version information:

go-ipfs version: 0.4.5-rc1-8c07404
Repo version: 5
System version: amd64/linux
Golang version: go1.7.1

Type: Bug

Priority: P2

Description:

The http bulk block.put command seems to only add the first file in the multipart

To reproduce (where file1.txt and file2.txt are two small files):

curl -v -F [email protected] -F [email protected] http://localhost:5001/api/v0/block/put
=> {"Key":"QmRfJWuFg328tVnkDfH1WHGuiuy2FbtZyhyNHQSA2dnh68","Size":4}
* Connection #0 to host localhost left intact

So it's only returning the hash of the first file (and presumably only adding the first), and also not closing the connection.

[whyrusleeping]

The file argument to ipfs block put is not variadic, it only accepts a single input file.

I asked @diasdavid and he thinks this is the correct behaviour.

[daviddias]

Hey, we also sync'ed up again and we discussed adding a flag for 'batch puts', so that existing API work and batching puts because extra feature, how does that sound?

[ianopolous]

Sorry, somehow I got it in my head that this was already how the api worked because it is using multipart.
Adding a batch option sounds great and would hugely facilitate apps running over a high latency (even interplanetary ;-) ) connection.

[ianopolous]

I may be wrong, but I think it wouldn't even require the batch flag. Existing http api implementations can continue as they are, because they are already using multipart, just with a single file. Updated ones can, include more than one file and parse the resulting stream of json objects.

[mib-kd743naq]

@ianopolous there's the related https://github.com/ipfs/go-ipfs/issues/3442: HTTP overhead is quite high in the face of many blocks, so a "cli-streamed" solution seemed more appropriate.

A specific example of what a dataset looks like is at https://ipfs.io/ipfs/QmYbb74fCe7pom18nPRw2rKAPMukza63DfQAdHG53WywbW, containing the entire dataset for https://github.com/ipfs/go-ipfs/issues/3552

[ianopolous]

@diasdavid Having thought more about this and discussed with @whyrusleeping I'm pretty confident this is relatively easy to do without a batch flag (I've also already implemented it in Java for our proxy), and in a backwards compatible way for old http clients.

The key thing is the clients are already using the multipart protocol, but only writing 1 file, so the protocol is the same. The headers for a response may change if there is more than 1 file, but they could (should) remain the same for puts of a single file. This would be bitwise identical with existing behaviour, and as old clients will only ever ask to put 1 file, they will never see any difference.

Currently, the server can't write the first response hash until it's read the first file fully (and there's little point writing the headers before then as the client will have to wait anyway to read the hash), by which point it knows if there is more than 1 file or not and could set the headers accordingly.

[whyrusleeping]

Yeah, we should be able to do a batch block put. lets get this written up (and make sure to have a test for api backwards compatibility)

[Stebalien]

Fixed.