Runtime: FileSystem cross platformness and testability

[davidfowl]

Building cross platform applications that talk to the file system is painful. There are several issues like case sensitivity, slash differences and other bizarre restrictions and differences.

On another front, testability of the file system has always been impossible with .NET. Any application/library that uses the FileSystem uses something like https://www.nuget.org/packages/System.IO.Abstractions or a custom abstraction to allow mockability.

Are there any plans to improve the System.IO API to solve some of these problems?

[MpDzik]

Very good question. I think it's a great idea to abstract the API's in System.IO using interfaces, especially the static classes, such as System.IO.File, System.IO.Directory, System.IO.Path, etc.

[davkean]

As part of new FileSystem work to enable it for WinRT and Win32 - we've got an abstraction internally that we now use. We're not yet confident in it however, so it's unlikely to make it for the first version.

Linux/Mac bring up will give us a little more confidence, so we might be able to make it public in a future release.

[migueldeicaza]

The file system APIs in .NET have many severe limitations and they have historically been very difficult to surface properly on Unix, and in general can not be made very robust today.

If there will be work done in this area, it might be a good time to address those long-standing issues.

Among those, you might want to consider:

• Unix filenames are a stream of bytes, not necessarily something that can be converted to a valid Unicode string. While uncommon, they do happen, in particular in countries where codepages are still commonly used, or when dealing with old archives from easter european countries that contain filenames that do not decode properly.
• Directory iteration primitives that decouple the filename from the file name information (getdents + stat vs FindFirst/FindNext idioms). getdents is fast, stat is slow, specially over network shares.
• Support for symbolic links on the API
• Support the various Unix open modes (O_SYMLINK, O_NOFOLLOW)
• Splitting support for Windows and Unix attributes, similar to Java's NIO FileAttribute and FileAttributeViews.
• Design an ACL system that is Unix friendly, and not tied to Windows.

Java has a few

[Petermarcu]

Great stuff. Thanks Miguel. Lots of stuff to look at and consider here.

[MpDzik]

+1 for exposing symbolic links in the API.

[rojepp]

To add to the list, it would be great to add implicit expansion of '~' on OSX/Linux.

[ashmind]

Just a few some small points:

1. If it is not just a IFileInfo, but a better design -- File.ReadAllText and similar methods should be included (potentially as extension methods). They are extremely convenient, and it would be a pity to lose them due to the abstraction. Also I think there should be ReadAllTextAsync etc.
2. There should definitely be a non-static path type similar to the Uri type -- passing potentially invalid strings around is just asking for trouble. Usage of this type should not incur filesystem access.
3. In addition to symlinks, junction/hardlink information/creation seems like a good idea.

[karelz]

We need API proposal on this one.

[polarapfel]

@migueldeicaza As to your points from January 2015, what's your assessment around these issues in regards to how Mono tackled those? Is the implementation in Mono.Unix/Mono.Unix.Native solid enough to handle these properly?

Thanks!

[karelz]

@davidfowl why did you close it?

[Petermarcu]

Seems it was unclear why this was closed. Was also referenced here https://github.com/dotnet/corefx/issues/21967

See also dotnet/templating#1028 for relevant discussion.

Going to reopen to allow for the discussion to finish.

[danmosemsft]

@JeremyKuhne @pjanotti this issue seems lie a grab bag - testability and other things. Perhaps you could break out more specific issues (if we don't already haev them) and close this?

[JeremyKuhne]

Triage, really nice to have, but a significant task to accomplish.

[carlossanlop]

This issue is too wide. We already have individual issues tracking each one of the requests listed in this comment: https://github.com/dotnet/runtime/issues/13946#issuecomment-69137144

So I'll close this in favor of the individual API proposals and issues requesting improvements.

[rmunn]

@carlossanlop - It would be nice to have a list of the individual issues tracking those requests so I can subscribe to them. Some of them I might be able to search for, but some I don't know what search keywords to use. Since you already know what those issues are, perhaps you could add a comment listing them? Would be appreciated.

[carlossanlop]

• For file casing: https://github.com/dotnet/runtime/issues/14321
• For Unix ACLs: https://github.com/dotnet/runtime/issues/928
• Project tracking FileSystem related issues and PRs: https://github.com/dotnet/runtime/projects/42
• Project tracking Symbolic links/Reparse points/Junctions issues and PRs: https://github.com/dotnet/runtime/projects/41

Please feel free to open new issues for individual feature requests, it's easier to track than a general issue.

cc @Jozkee