Go-ipfs: Limit blocks files to 256KiB boundaries

[ottuzzi]

Hi,

I'm playing a bit with ipfs and I noted that files inside blocks directory are a bit bigger than 256KB.
I'm on go 1.5.2-win32 and I just built ipfs from github repo.
Files in blocks directory are 262.158 bytes but on disk they use 266.240 bytes: 256KB are exactly 262.144 bytes so for 14 bytes we use a 4KB more.
This is not only a "problem" for used space on disk (about 1.5% more) but it is not optimized for disk reads as we have to read 4KB more for only 14bytes. Probably all modern FS will take care of that but it will be nice if we can constraint us to exactly 262.144 bytes.

Bye
Piero

[Kubuxu]

This is something that should be thought about while implementing IPLD as we will break hashes either way.

CC @mildred @whyrusleeping @jbenet

[mildred]

This is to take in consideration when chunking files to fit in multiple blocks. It doesn't have to do with IPLD directly. But you are right, we can use the IPLD transition to change the chunking algorithm as well.

[Kubuxu]

We should also place limits on creating blocks (sanity check), maybe also on receiving them from the network (only if they are in IPLD format for backward compatibility)

Note: In Windows world 4kB means 4KiB, so the cluster size on windows is: 4 * 2^10.
Then our blocks should also be 256KiB in size, 256 * 2^10 = 262,144

[davidar]

@Kubuxu it usually means 4KiB on most platforms, no?

[Kubuxu]

No, on darvin and Linux kB = 1000 bytes as for metric SI prefixes.

[davidar]

@Kubuxu what I mean is, afaik block sizes on linux are usually 4KiB too (they're just more explicit in distinguishing between KiB and kB than windows is)

[Kubuxu]

Yeah, but as originator of the issue used kB, this indicated for me that he was using Windows, I had to check documentation to make sure that Windows (FAT and NTFS) is for sure using 4KiB cluster sizes (you know Windows, everything is possible).

[ottuzzi]

Hi, I was thinking more at a disk layout level where last generation disks have a 4096bytes "sector" size :)

Bye
Piero

[whyrusleeping]

This should be resolved when moving to ipld as it will allow us to reference raw blocks (without needed the extra framing)

[whyrusleeping]

3307 should fix this

[whyrusleeping]

Using ipfs add --raw-leaves should acheive the goal of disk blocks being limited to exactly 256k