Go-ipfs: fuse reports incorrect disk usage

[bmwiedemann]

Version information:

go-ipfs version: 0.4.19-
Repo version: 7
System version: amd64/linux
Golang version: go1.11.4

Type:

• bug
• enhancement

Description:

stat and du on fuse reports incorrect disk usage.

du /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
1       /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
du --bytes /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
18092   /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
wc -c /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
18092 /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
stat /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
  File: /ipfs/zb2rhfMfthqjiEBqANmRyP1L3kzHxabZrwZH9az8QA2ZKm1oQ
  Size: 18092           Blocks: 1          IO Block: 4096   regular file
Device: 27h/39d Inode: 11723163882849603120  Links: 1

As can be seen, stat reports the number of 256 KB blocks.

The same file in ext4 has

  Size: 18092           Blocks: 40         IO Block: 4096   regular file

stat --format %B reports 'the size in bytes of each block reported by %b' as 512 in both cases.

[Stebalien]

Looking at the documentation, it looks like we should always use 512 as the block size when calculating the number of blocks. I wish I were kidding but: "This field indicates the number of blocks allocated to the file, in 512-byte units."

Basically, "block" is just being used to mean "512 byte chunk".

[Stebalien]

@djdv can you make sure your new implementation doesn't have this issue?

[djdv]

@Stebalien
For sure.
I believe this works already but with the next revision I'll test it explicitly and report back.
We basically are passing through a statfs for the volume, by using relevant syscalls on the ipfs config dir.
(essentially statfs(~/.ipfs))
So the mount volume metrics should match whatever volume it lives on.

For individual records, we use the first specified unixfs blocksize (if any) for a node. It may be worth disregarding this though and using the same (system-expected) blocksize.

The only funny thing I'm doing in regards to metadata at the moment, is specifying a size for directories. As far as I'm aware this isn't defined, so I'm using it to store the entry/child count (for internal use).
If it causes weird issues, I'll drop it back to 0 and store that value elsewhere.

[bmwiedemann]

The size of directories depends on the implementation of the underlying filesystem.
I think, du takes it into account, so it would be good if it is close to reality.

stat .
  File: .
  Size: 12288           Blocks: 24         IO Block: 4096   directory

[djdv]

For directories, we have the option to use the IPFS object size (link+data sizes), but this is the size of the entire encapsulation, not just the unixfs portion of that IPFS object. It's looking like it will be best to return the unixfs size for directories (undefined:0) to not break programs.

[Stebalien]

I believe the _correct_ way to calculate directory size is to sum up all the nodes that make up the directory. However, IMO, this is mostly useless and 0 is probably the best solution (and what ls now returns).