Go-ipfs: [BOUNTY] Implement UnixFSv1.5 in go-ipfs

[Stebalien]

This issue has a bounty!

Successfully closing this issue by producing a production-ready, mergeable PR can earn you not only the undying love of the IPFS community — it can net you a financial reward. See the current list of bounty issues and their values here.

The need in brief

Add UnisFSv1.5 metadata support per the spec https://github.com/ipfs/specs/blob/master/UNIXFS.md#data-format, which includes: specifying metadata, respecting existing metadata, and displaying metadata.

This aims to support large data import use cases (ex package managers), who need to retain file metadata, particularly last modified time (mtime), in order to selectively sync only data that has changed. Up until now if you wanted to host a large data set on IPFS, like a package manager's repository, it would be difficult to update - but with file mtime you can only reimport the changed files, making things much more efficient.

Deliverable

• Implement the new mode & mtime fields and obey the new rules around default modes & mtimes as per the metadata spec.
• Plumb this metadata all the way through from ipfs add on one IPFS node to ipfs get on another.
• An end-to-end sharness test that validates that this metadata is preserved.

Requirements

• This feature must be interoperable with js-ipfs's implementation of the UnixFSv1.5 spec.
• The behavior of ipfs add must not change except when the user has requested that metadata be preserved via the appropriate flags (see the implementation details below). Specifically, the final hash produced by ipfs add must not change.
• The final hash of files added with this feature enabled must match the hash produced by javascript.

Guidelines

• Please use the IPFS Go Contributing Guidelines as your north star — adherence to these guidelines are crucial when it comes to awarding a potential bounty!
• Please DO NOT bundle a general code refactor with the completion of this bounty!

[momack2]

For context, here is the js-ipfs unixfsv1.5 implementation done by @achingbrain: https://github.com/ipfs/js-ipfs/pull/2621, and the blog post describing the shipped work: https://blog.ipfs.io/2020-02-13-js-ipfs-0-41/#unixfs-v1-5

If you want to start working on this, ping @Stebalien for a quick walkthrough of where to start and what already exists to build on top!

[Stebalien]

Implementation Notes

Implementing this feature will touch several sub-components of go-ipfs:

• The go-ipfs-files library will need to be updated to:
  
  
  
  • Support modification times and modes.
  
  
  • When reading from and writing to disk.
  
  
  • When reading from the web (WebFile).
  
  
  • When sending/receiving over HTTP using the "multipart" file types.
  
  
  • Expose modification times and modes via the Node interface.
  
  
• Implementations of the Node interface from go-ipfs-files will need to be updated:
  
  
  
  • github.com/ipfs/go-unixfs/file/unixfile.go
  
  
  • github.com/ipfs/go-ipfs-http-client/apifile.go
  
  
• The protobuf definitions in github.com/ipfs/go-unixfs/pb/unixfs.proto will need to be updated and regenerated according to the spec.
• The go-unixfs (de)serialization logic in go-unixfs will also need to be updated to (de)serialize modes/mtimes.
• The go-ipfs file adder (github.com/ipfs/go-ipfs/core/coreunix/add.go) will need to be extended to handle modification times.
• The go-ipfs "add" command (ipfs add) will need two new flags (defaulting to false):
  
  
  
  • --preserve-mtime -- record the file modification time when adding.
  
  
  • --preserve-mode -- record the file mode when adding.
  
  

The new go-ipfs-files Node interface should be:

// Node is a common interface for files, directories and other special files
type Node interface {
    io.Closer

    // Mode returns the file's Mode
    Mode() os.FileMode

    // ModTime returns the files last modification time.
    //
    // If the last modification time is unknown/unspecified, ok is false and
    // the mtime is the Unix epoch (default timestamp).
    ModTime() (mtime time.Time, ok bool)

    // Size returns size of this file (if this file is a directory, total size of
    // all files stored in the tree should be returned). Some implementations may
    // choose not to implement this
    Size() (int64, error)
}

[kalmi]

@Stebalien Hi, @momack2 said to ping you for a walkthrough if one would like to start working on this, but I think it is safe to assume that your last comment is that walkthrough, and I will just go ahead. Thanks!

[achingbrain]

ipfs add also needs the following new options:

• --mode - specify a file mode
• --mtime - specify the seconds component of the mtime
• --mtime-nsecs - specify the nanosecond component of the mtime

And there are two new mfs commands:

• ipfs files touch [path] (update mtime to the current time)
  
  
  
  • --mtime - optionally specify the seconds component of the mtime to update to
  
  
  • --mtime-nsecs - optionally specify the nanosecond component of the mtime to update to
  
  
• ipfs files chmod [mode] [path]

[momack2]

Hey @kalmi! How are things going? Did you get your questions answered / are you unblocked?

[kalmi]

Hi!

Thanks. I'm not blocked, I had some personal issues last week, sorry for the silence. I plan to spend most of the upcoming long weekend on this. So far I have got ipfs get to use perm modes passed from the tarwriter, and I might have overdone the handling of the edge cases that can happen during ipfs get, but probably not. I want it to update modes for files and directories that already exist without doubling the number of syscalls, and that proved to be a bit tricky.

Status (WIP, "done" is very loosely defined at this point, this list is bound to be incomplete):

• [ ] go-unixfs
  
  
  
  • [X] Regenerate unixfs.pb.go
  
  
  • [x] Update go-unixfs serialization logic 
  
  
• [ ] go-ipfs-files
  
  
  
  • [X] Implement ModTime and Mode (default-returning) stub for all Node implementers
  
  
  • [X] Change Node interface
  
  
  • [X] Use ModTime and Mode in tarwriter (used by ipfs get)
  
  
  • [X] serialfile
  
  
  • [x] WebFile
    
    
    
    
    
    • [ ] Check if there is such a thing in js-ipfs
    
    
    
    • [ ] Use headers (What headers?)
    
    
    
  
  
  • [ ] multipart
    
    
    
    
    
    • [ ] Check used headers in js-ipfs
    
    
    
    • [ ] Use headers
    
    
    
  
  
  • [ ] filewriter
    
    
    
    
    
    • [ ] Find where this is actually used
    
    
    
  
  
  • [ ] others?
  
  
• [ ] go-ipfs-http-client (yet to investigate)
• [ ] ipfs get (this mostly touches whyrusleeping/tar-utils in a backward compatible way)
  
  
  
  • [X] mode perms
    
    
    
    
    
    • [X] umask is respected (maybe too much, even for preexisting file, that might be misguided)
    
    
    
    • [X] mode perms are updated for preexisting directories and files
    
    
    
  
  
  • [ ] set non mode perm bits
  
  
  • [ ] set mtime
  
  
  • [ ] sharness
  
  
• [ ] ipfs add
  
  
  
  • [ ] go-ipfs/core/coreunix/add.go updated
  
  
  • [ ] --preserve-mtime implemented
  
  
  • [ ] --preserve-mode implemented
  
  
  • [ ] --mode
  
  
  • [ ] --mtime, --mtime-nsec
  
  
  • [ ] sharness
  
  
• [ ] mfs commands
  
  
  
  • [ ] touch
  
  
  • [ ] chmod
  
  
  • [ ] sharness
  
  
• [ ] Investigate OS time resolution differences
• [ ] Consider abusing NTFS alternate streams to store mode
• [ ] FUSE?

[Stebalien]

I'd drop the last two for now. Everything else looks good to me.

Note: If you're trying to find where different types/functions are being used, I recommend grep.app. E.g., .NewMultiFileReader.

[momack2]

Hey @kalmi! Any status update? Are you blocked?

[kalmi]

@momack2 I am making progress, albeit unfortunately it is slower than I originally expected due to external factors. Now I am at the point where I believe the get side of things to be mostly done, and I am moving on to extending add, and I hope to be able to write back the added files with metadata somewhen in the next few days. I wanted to test aginst jsipfs before this as a sanity check, but I had some bad experiences it with it, and eventually decided to push it to the end. In the meantime, I will use the hashes from its test cases for the go-ipfs tests.

I am making sure that all changes to the libraries are backward compatible. The Node interface change specified by Stebalien is not, but I guess that's reasonable and acceptable. When I have everything working, I will start making PRs in the proper merge order.

Do you have a reasonable timeframe in mind for this that I should be aware of?

[momack2]

Hey @kalmi! It'd be great to include this feature in the next release, go-ipfs 0.6. I think the feature cut for that release is ~2 weeks from Wednesday. Do you think that's a reasonable goal?

Note, the sooner to put up draft PRs for review (and the more modular/well tested they are) the easier it will be to land this quickly. =]

[achingbrain]

I wanted to test aginst jsipfs before this as a sanity check, but I had some bad experiences it with it

Could you open an issue on ipfs/js-ipfs please and we'll see if we can get to the bottom of the problem you had?

[Stebalien]

@kalmi Any status updates?

[justicenode]

@kalmi @Stebalien If no one is going to take care of this issue I'm willing to give some of my free time towards it. I'm programming in my free time and one of my programs kinda depend on this . So I'm willing to take care of this so my software works on any ipfs distribution. IDC about the money i just want to get this working. let me know if i can help or anything

[autonome]

Thanks @justicenode, yes it'd be great if you were able to carry this forward.

With no updates in over 60 days, it's ok to unlock this for other contributors.

@kalmi we really appreciate your effort to take this on, and happy if you're able to jump back in and collaborate!

[justicenode]

@autonome okay, I'm on holidays for like the next two weeks but after that i should have enough time to truly work on this.

Is there already something I can continue working on or do i have to start over? I tried looking for stuff from @kalmi, but i couldn't find anything, but maybe I'm just blind. If there's nothing to continue from, could you give me some pointers as to where i should start? Like what repo i should fork, or which files might be relevant.

Also I haven't done much with go so progress might be a bit slow.

[djdv]

@justicenode

Is there already something I can continue working on or do i have to start over?

I did a portion of this initially, maybe useful to see it.
https://github.com/ipfs/go-unixfs/compare/master...djdv:ufs-1.5

I may have deleted the go-ipfs-files stash from my workspace though...

---

Like what repo i should fork, or which files might be relevant.

Make sure to see this for a todo list and file references:
https://github.com/ipfs/go-ipfs/issues/6920#issuecomment-601001679

[kstuart]

@Stebalien, it doesn't look like this is actively progressing so mostly out of curiosity thought I'd take a look over the weekend and ended up implementing one of the cases, so at this time I can successfully preserve or set custom values for mtime/mode using ipfs add and retrieve correctly with ipfs get for a unix file. I've not attempted implementing other cases as the exercise was just to familiarize myself a bit with the codebase and Go language.

I'm not looking to tread on anyone's toes if this is already being worked on, otherwise I'm happy to take this forward, let me know.

[momack2]

That’s great news! I don’t think @justicenode got a chance to pick this up, so no toes stepped on. A PR for the cases you’ve got working would be lovely (more small PRs are always easier to review). @aschmahmann to help with that 🤗

[aschmahmann]

Thanks @kstuart the implementation notes in this issue are hopefully clear enough to guide you in your implementation.

I'd recommend putting up work in progress PRs early and poking folks (e.g. me) for feedback even at the earlier stages instead of waiting to put up a mostly completed PR. This should make things easier to review and make sure that we're on the same page.

I'd also recommend breaking the different steps/sections up into different commits and/or PRs so that they are easier to review + merge.

[kstuart]

Draft PRs submitted, as I wrote this on the hoof I'll not be adding any further functionality for now but focusing on exploratory testing and authoring tests, probably not till the weekend though.

For anyone wanting to try out the new features you'll need all the related PRs, you should then be able to use the functionality detailed in #7754, testing on Windows would be welcome.

For webfile, as I'm not aware of an existing X- header for communicating file mode I'm initially proposing X-FileMode, this is easily changed.

[achingbrain]

I'm initially proposing X-FileMode

We had to change the js implementation around passing file mode/mtime to the HTTP API - because in the browser you have to use FormData objects to construct requests from multiple Blobs in order to prevent buffering file data into memory. Each Blob you add becomes a multipart part - but you cannot specify per-part headers with the FormData API.

So what we ended up doing was encoding the mode/mtime as a querystring and appending it to the field name because it was essentially the only field we could modify that didn't take user input (like the filename for example), so your request ends up looking something like:

POST /api/v0/add HTTP/1.1 
Host: localhost
Content-Type: multipart/form-data;boundary="boundary" 

--boundary 
Content-Disposition: form-data; name="file-0"; filename="/file-with-no-metadata.txt" 

...file data 
--boundary 
Content-Disposition: form-data; name="file-1?mode=0755"; filename="/file-with-mode.txt" 

...file data
--boundary
Content-Disposition: form-data; name="file-2?mode=0755&mtime=500"; filename="/file-with-mode-and-mtime.txt" 

...file data
--boundary
Content-Disposition: form-data; name="file-3?mode=0755&mtime=500&mtime-nsecs=4982"; filename="/file-with-all-the-metadata.txt" 

...file data
--boundary--

If go-IPFS could implement the same thing it would mean this feature will work across both implementations and in browsers.

[kstuart]

@achingbrain, for multipart I've implemented the querystring on field name, the name itself doesn't include an index as the way parts are constructed doesn't lend itself easily to doing so, in this context though I wouldn't expect that to be an issue, e.g.
Content-Disposition: form-data; name="file?mode=0644&mtime=1604320500&mtime-nsecs=55555"; filename="beep.txt"

For the webfile case e.g. ipfs add http://example.com/somefile.txt the current implementation uses the following response headers when present:

• Last-Modified -- to set the last modified timestamp
• X-FileMode -- (as an octal string) to set the unix file mode

Does the js implementation have a position on this?

[achingbrain]

No, looking for those headers in the response sounds perfectly reasonable.

The X- prefix has been deprecated for a few years but my opinion isn't that strong.

File-Mode might be more conventional than FileMode.

[lidel]

Given that we already return X-Ipfs-Path I'd go with Ipfs-File-Mode (assuming we want to drop X- at some point anyway) just for the sake of consistency and emphasis this metadata comes from IPFS.

[achingbrain]

This isn't for what we return though, it's what we will look for in HTTP responses when we are importing from a URL - expecting third parties to prefix headers with IPFS- just in case someone tries to ipfs add the URL seems unreasonable.

[achingbrain]

@kstuart I've put a repo together that'll allow you to run the interface tests js-IPFS runs but with minimal setup: https://github.com/achingbrain/ipfs-interface-tests

There's a readme with instructions but in brief once you've got everything running you'll just need to remove the skips for all the tests that mention mode, mtime or metadata in test/interface.spec.js

Please let me know if you have problems running it or need any help.

[kstuart]

@achingbrain That's great, and very insightful.

Should I be concerned about the preloadNode issue?

[achingbrain]

Should I be concerned about the preloadNode issue?

If you pull from the repo this should be fixed now, sorry about that