Go-ipfs: ipfs files ls -l doesn't return hash or size when listing files directly

[achingbrain]

Listing a directory shows hashes and sizes:

$ ipfs files ls -l /
data.txt    QmahmHDpSXGKUNxVGre3ZGj9wG9pGwhb691SZdep4gV4XU  142646400

But if you list the file directly, extended information is not returned:

$ ipfs files ls -l /data.txt
data.txt        0

[schomatis]

From what I understand from reading the files API, there is important information (such as the hash and the size, and I'm not sure if the name also) that doesn't belong to the node that represents the file but rather to the directory that contains it, i.e., that has a link (to the file) where that information is actually stored. This means that there is some information that can only be obtained by querying the directory (AFAIK).

[achingbrain]

The name, sure - that's stored on the DAGLink of the containing directory, but the hash and the block sizes of the data nodes are stored as part of the file node and you have to walk the tree to find out what hash /data.txt has at the moment, so all that information should be available.

[schomatis]

Yes, so scratch most of my last comment, the files ls command is not extracting the node to get any information, it's just repeating back the file part of the path passed as argument.

https://github.com/ipfs/go-ipfs/blob/611d572ef23e92af6640f6bde207c3bdc69e48ab/core/commands/files.go#L463-L467

This can be improved, I'll send a PR.

[schomatis]

@achingbrain Could you give https://github.com/ipfs/go-ipfs/pull/5045 a try to see how it feels? Keep in mind that the sizes reported by listing the directory and the file might vary by a small amount due to some metadata (ideally those sizes should be unified).

[achingbrain]

I must be doing something wrong because I still see 0 for the file size and no hash.

I did:

$ cd ~/.gvm/pkgsets/go1.9/global/src/github.com/ipfs/go-ipfs
$ git remote add schomatis https://github.com/schomatis/go-ipfs.git
$ git fetch
$ git checkout fix/ls/dir-size
$ make clean && make install

Sorry, I don't know much about go..

Is it going to print out the node size or the sum of the block sizes?

[schomatis]

Oh, sorry, the PR I pointed you to is not yet merged (nor it will be in the near future until I figure out the size discrepancies), the commands you're listing sound correct but I would recommend you try the official GitHub way which don't require to add my repo with the branch but to fetch it indirectly through the PR number (using the original ipfs remote):

git fetch origin pull/5045/head:fix/cmd/files-ls-file-long
git checkout fix/cmd/files-ls-file-long
make install

Is it going to print out the node size or the sum of the block sizes?

It should print the node size plus the size of the blocks (links) its referencing.

https://github.com/ipfs/go-ipfs/blob/611d572ef23e92af6640f6bde207c3bdc69e48ab/merkledag/node.go#L226-L239

But that's what I want to be certain of before merging the PR.

[achingbrain]

Aha, that's better - thanks.

From what I can see, if something is a directory, it should have size 0, if it's a file, you should extract the UnixFS block sizes from the files' data block - the sum of which will give you the total size of the file without the proto-buf wrapper overhead.

Otherwise ipfs disagrees with itself on what the file sizes are. E.g. with your PR changes:

$ ipfs files ls -l /
data.txt    QmahmHDpSXGKUNxVGre3ZGj9wG9pGwhb691SZdep4gV4XU  142646400
$ ipfs files ls -l /data.txt
data.txt    QmahmHDpSXGKUNxVGre3ZGj9wG9pGwhb691SZdep4gV4XU  142684224

[schomatis]

Yes, I've modified the PR to get the size from the mfs.File object instead of ipld.Node to make it homogeneous. Could you give it one last try please? (you would need to fetch it again from the remote)

[achingbrain]

That's great, both invocations have the same file size now.

[schomatis]

Great, thanks for raising this issue and guiding me through it, it has been very illustrative.

I'm cc'ing @Stebalien to make sure what we have been discussing makes any sense, if he approves I'll unblock the PR and it will be ready to merge.

[Stebalien]

That sounds correct. Sorry, I hadn't realized the size bug had been resolved. That PR should be good to go (modulo the review nit).

[achingbrain]

Closing this in favour of the PR that will resolve it: #5045

[schomatis]

Hey @achingbrain, not a big deal but just to keep up with the convention allow me to keep this issue open until #5045 gets actually merged (when that happens the "Fixes" word in the PR comment will signal GitHub to automatically close this).

[achingbrain]

Ok, sorry about that. Was just trying to not have redundant issues open.

[schomatis]

And, since I'm here, I'm including this issue in the milestone (just a symbolic thing, but this was the issue that motivated me to really learn about the Files API).