Runtime: Introduce MONO_IOMAP-like compatibility for Unix platforms

[flibitijibibo]

Prior to 6.0, Mono had a feature known as IOMAP, which allowed users to enable an extra code path that would check for filesystem incompatibilities commonly found when running Windows .NET applications on Unix platforms (Linux in particular). With the introduction of CoreFX to Mono's System.IO implementation, the feature no longer works and the Mono team deemed the feature simultaneously "removed" and "deprecated", though this _almost_ appears to be in name only as 100% of the original code is still present and the feature is still listed in the README as of writing (plus a little bonus issue that we'll get to at the end of this report).

There are lots of reasons that developers need IOMAP, many of them centered around backward compatibility and preservation, but another important reason to have this is for _current_ applications with Windows-centric content pipelines as well as user-generated content. For example, a game built on Windows may have files referenced as plain text paths in their content blobs, and if the cases don't match, Unix builds would need to rebuild all of the content with case sensitivity fixes every time they make a content change, which could mean going as far as porting your content build system to a platform that has a case-sensitive filesystem, if that's even possible to begin with (keeping in mind that this only works if you don't have issues with data mismatches for cross-platform online interactions, unless you take the Unix content and drag it _back_ to the Windows build). If this all sounds annoying and fragile, that's because it is. This is true for both developers and users who make custom content for a game.

As for an implementation, the feature is not as large as you might think; the extra code paths were contained almost entirely in w32file-unix.c, and as it turns out, this file does almost the exact same thing as pal_io.c in System.Native! Plugging the old code into the new code was surprisingly easy to do:

https://github.com/flibitijibibo/corefx/commit/2902d5856985830db1e23f78ed8b83d4c502a261

Almost 100% of the code is a _direct, unmodified copypaste_ of the original, with the exception of exactly one function that was just a wrapper which served no purpose on non-Win32 platforms. In my local testing, it works great! There are some caveats, however:

• This is obviously a nasty copypaste and should probably be cleaned up to match the CoreFX style. A real implementation of this in upstream CoreFX would function the same, but would not do all the sloppy header inlining I did, nor would it do the weird things Mono is doing in their find_file implementation (use of glib is the obvious one).
• Not every pal_io function has a mono-io-portability equivalent. Implementing these should be trivial, but for this proposal I wanted to emphasize how much was already done with _absolutely no new code written whatsoever_.
• Mono, as of the current master revision, uses both the old MonoIO code and the new Interop.Sys code simultaneously. I originally wanted to propose this as a totally new DOTNET_IOMAP feature, but in testing this it turns out you need the variables to match so that it's as consistent as it was before Mono 6. (So yes, in vanilla Mono it's actually possible to have _partial_ MONO_IOMAP support that happens to work if you get lucky.)

[msftbot[bot]]

Tagging subscribers to this area: @jozkee
Notify danmosemsft if you want to be subscribed.

[danmosemsft]

Is the scenario entirely a developer machine scenario? It seems potentially a bit risky, to switch on this mode and then have your app accidentally delete a file that differed in casing.

@marek-safar @akoeplinger do you have a read on how widely this was used? I see some discussion here https://github.com/mono/mono/issues/9300#issuecomment-406614665

[flibitijibibo]

This feature is currently used in the retail versions of Terraria, Wizorb, A Virus Named TOM, and Unexplored. That’s my catalog of examples based on the FNA game list at least. It is also usable for mod support in other games like Celeste (Everest tries to enforce case sensitivity though), Stardew Valley, and Terraria (tModLoader for example). There are also tools like Rhys which need it to run Windows XNA games on Linux via FNA’s XNA ABI compatibility files.

[danmosemsft]

Any idea whether Unity includes it in their Mono builds? Just since you mention games.

[flibitijibibo]

It appears it is available in their runtime, an example of it being used to fix a plugin: https://forum.unity.com/threads/darkrift-server-plugins-on-linux-machines-errors-and-how-to-fix-it.444328/

EDIT: Seems this is a separate companion application rather than a Unity export, but still... the Google results go on _forever_.

[marek-safar]

@danmosemsft I believe the primary use-case was for non-development scenarios. The use case is to run libraries which have bugs in the casing on filesystems which distinguish between different cases.

[rfht]

Without MONO_IOMAP, it can be impossible to run assemblies generated on Windows on a Unix machine if the developer didn't pay close attention to match cases in filenames and the paths in the assemblies. Of course, it would be nice to have the developers pay attention to it, but it's easy to not even notice the issue.
Without a replacement for MONO_IOMAP=all as an option, cross-platform compatibility will be impaired in my opinion.

It seems potentially a bit risky, to switch on this mode and then have your app accidentally delete a file that differed in casing.

The switch should only be used if necessary with appropriate warnings IMO.

[danmosemsft]

@jkotas has this come up before?

[jkotas]

This is the first time I am hearing about this. It looks like a partial Windows emulator. I do not think it is something we would want to include in the product by default.

it can be impossible to run assemblies generated on Windows on a Unix machine if the developer didn't pay close attention

There are many other reasons why code written for Windows may not work on Unix.

[flibitijibibo]

IOMAP is not something that Windows users and developers will be intimately familiar with, but on other platforms it's essential for everyday use. It's enough to where I would 100% have to make a directly competing repository to ensure that C# development is still possible on Linux should this not be integrated in any form.

By default the feature is not enabled at runtime, and the build is configurable in such a way that IOMAP can be completely disabled and removed from the final binary via --disable-portability. If the default behavior in .NET's official runtime was to disable this, I would be entirely okay with that, I have to self-build everything for MonoKickstart anyhow so one added flag will be no problem for us.

[TheSpydog]

There are many other reasons why code written for Windows may not work on Unix.

Isn't one of the main selling points of .NET that the same assembly can work across multiple platforms? The use case mentioned in this thread (cross platform games using the FNA framework) is a great example of what that can look like.

FNA is entirely built around the idea of single assembly portability -- the same .exe used on Windows also works on macOS and Linux. All platform-specific code is handled by a small set of native libraries. In fact, it's not just limited to desktop platforms; FNA also supports iOS, Xbox One, and Nintendo Switch, all from the exact same C# project with no platform-specific branches.

For situations such as these, where portability has been carefully considered, I see no reason why having an optional environment variable to fix up hardcoded path strings would be harmful in any way. Especially for preservation projects like Rhys, since it's quite possible the source code for these games has been lost to time, making a "proper" fix of the file paths impossible.

[jkotas]

the same assembly can work across multiple platforms?

can != will. You can make the same assembly work accross multiple platforms if you follow the rules for writing cross-platform code, like not assuming a case-insensitive file system.

[TheSpydog]

But _they do work_ already, just with a slightly older version of Mono. Perhaps you missed the last part of my comment, about how this would especially benefit preservation projects. For games that were originally made targeting XNA and Windows only, going back and fixing up the source isn't even possible sometimes. Does that mean they should be abandoned entirely, even if they're already functional on Linux with the use of this environment variable?

[flibitijibibo]

can != can't, if we aren't given the resources to fix a bug then products will just be broken forever and the people who made the choice to take those resources away will be the ones to blame. Again, I would like to make it as clear as possible that this feature is going to get written either way, it just seems impractical to maintain an entire runtime to patch in code _that an official runtime already had a decade ago_.

[cybik]

I believe that what's being said is that the feature which is requested here, would be _a good "opt-in" addition to the framework_ that should also never be active by default. I don't think that anyone here would want this feature active at all times for all .net payloads.

While I do understand the argument that "good programming practices are always the best" (and believe me, I advocate for it), at the same time it is unfortunately wrong to assume that everyone would follow them, or would have either the inclination or the time to do so if their original target (and the platform on which they did their good work) did not have such restrictions.

Also, as @flibitijibibo mentioned, sometimes they can't even get to the code, which directly hampers any efforts to fix the casing issues.

[danmosemsft]

If we wanted to do something here (not suggesting we do) would another possible approach to expose a hook such that a 3rd party library could be discovered that contained the "mangling" code. That still has significant design/build cost but there would be no maintenance of the "mangling".

[flibitijibibo]

I'd be up for evaluating that as an alternative - whatever lets me plug into pal_io.c+pal_time.c will likely be good enough for me (it all gets statically linked at the end anyway).

[jkotas]

a hook such that a 3rd party library could be discovered that contained the "mangling" code.

That is not that different from telling people who really wants this to build their own libSystem.Native.so.

[cybik]

That is not that different from telling people who really wants this to build their own libSystem.Native.so.

Maybe so, but I'd argue that rewriting/bundling a whole libSystem.Native.so just to introduce filepath case auto-mangling/demangling is a smidge overkill.

[danmosemsft]

I'd recommend that you build and share your own as suggested above. This kind of "heuristic" code has a maintenance and testing burden and a hook would require significant design and implementation (eg., to discover the library, to avoid perf issues) and some maintenance as well. It seems that sharing your own binary would work well enough for this scenario.

[0x0ade]

This kind of "heuristic" code has a maintenance and testing burden

Isn't this burden already mostly lifted by the fact that it was maintained and has existed as part of mono? Please correct me if I'm wrong, but as far as I can tell, the point of this issue is to "reintroduce" MONO_IOMAP into the codepath that is now shared between core and mono, right?

[flibitijibibo]

I think we have the resources to test (and write tests, assuming Mono's unit tests aren't sufficient), I'm not so sure about maintaining a patchset repo. MonoKickstart will always be okay since that's for static bundles, but distributions with system runtimes will struggle with this, and telling users to shove an arbitrary file from my public FTP into /usr/lib64/ seems a little... unsavory.

[TheSpydog]

@danmosemsft @jkotas I'm a bit confused what the resolution was here, if there was any. In an attempt to clarify this issue, let me ask the following question:

If a community member (read: likely me and/or @flibitijibibo) fully implemented a MONO_IOMAP-equivalent feature that was...
1) Disabled by default at build time, guarded by a compile flag
2) Disabled by default at runtime, and only enabled via an environment variable
3) Compliant with CoreFX style
4) Compatible with all relevant pal_io functions

...would that be a candidate for inclusion in the product? If not, why? What else would be necessary for the inclusion of this feature?

I totally understand not wanting to increase the maintenance burden of the .NET runtime with niche features, but this is a unique situation in that A) it's of vital importance for preservation projects, B) it was already present in prior versions of the Mono runtime, and C) it doesn't have any feasible workarounds, short of forking the whole runtime.

[jkotas]

I am sorry - I do not see this as a good feature to have in this repo, even if the code is not building by default.

What else would be necessary for the inclusion of this feature?

Convince me or some other maintainer of dotnet/runtime that it is a good feature to have. E.g. Do other runtimes (Java, Go, Rust, ...) have a feature like this?

It is a non-goal for us to keep all 10 or 20 year apps working on latest .NET Core versions without any changes. There are number of features present in .NET Framework and Mono runtimes that are stripped in .NET Core. Even between major .NET Core versions, we do occasionally make breaking changes to move the .NET platform forward to stay relevant that may require adjustments of the user code. To run old code without any changes, it is best to use the old runtime that the code was developed against.

[cybik]

If I recall correctly, and thanks to some work I did for $dayjob last week, with Gradle & Java, it is possible to apply interesting compiled classpath transforms at build time. One could presumably write a File.class intercept of some sort that would be comprehensive enough to replicate an automatic different-case resolution behaviour close to what's being suggested here. [_edit: There might also be some Kotlin shenaniganery making such extensions to FileIO possible._]

But that would apply to source-accessible approaches, which may not be available in the case of preservation-type projects that may have lost access to such sources.

For the sake of argument though: going down the classmod path fully and altering assembled executable Java payloads via such transforms would also mean altering the "executable" (JAR files) pre-run (or creating an alternate executable just before runtime, or using class modification packages with a java "launcher" of sorts), which as a build engineer I am very much not fond of - altering runtimes / build workspaces is a gigantic _/redacted/ no_, as one can't guarantee anything when modifications like that start happening.

Honestly, if I had to say, the interest of the IOMAP feature in discussion is that the mangling/demangling happens at the framework level, and the additional components in the library can focus on their own features. Altering the "origin" executable is literally the antithesis of the objective here, and in some cases could actually have less desirable implications beyond the scope of simple programming/adaptation.