Wasmtime: Offer an API for applications to receive the paths to their config/cache/data/... directories

[soc]

This is inspired by https://github.com/soc/dirs-rs/issues/19.

History has proven time and time again that people can't be bothered or relied on to implement these things correctly if they aren't given a straight-forward API that does everything for them.

So it would be nice if WASI had an API that followed the individual platforms' standards instead of forcing people to poorly reinvent things on their own.

I currently tend to think offering a mid-level abstraction like https://github.com/soc/directories-rs#projectdirs would be preferable to a low-level abstraction like https://github.com/soc/dirs-rs, but I think this would be a matter of consideration and debate.

I think timing is crucial, otherwise this ecosystem just turns into a mess similar to the situation on Linux/macOS/Windows.

[kubkon]

@sunfishcode @pchickey

[sunfishcode]

From a WASI perspective, dir's design of handing out absolute-path PathBuf's makes it not quite a direct fit, since at the WASI level there are no absolute paths. And in general, we wish to avoid exposing platform-specific directory names to applications. However, it does seem like we could bridge this with the preopen mechanism and some remapping.

Here's a possible design sketch:

• Let wasm applications think in terms of XDG paths. Question: Is XDG the best choice here?
• Add new command-line options to Wasmtime such as --cache_dir= and --audio_dir=, similar to --dir= and --mapdir=, but which take a host path and associate with an XDG path to make available to the application.

To be sure, these command-line arguments should eventually also be wasmtime API parameters, so that people can use Wasmtime as a library and use this functionality there too.

Thoughts?

[soc]

@sunfishcode It's possible to think about handing out opaque "path-y" things that don't allow traversing the path to the top, but e. g. allow appending further path elements, but I don't think it's the immediate question.

Let wasm applications think in terms of XDG paths. Question: Is XDG the best choice here?

No, follow the rules of the platform you are running on.

Add new command-line options to Wasmtime such as --cache_dir= and --audio_dir=, similar to --dir= and --mapdir=, but which take a host path and associate with an XDG path to make available to the application.

The issue is the 90% of the people who don't care and can't be bothered, not the 10% who try to be good citizens on the platform they are running on. I think these options will barely get more than 10% of adoption.

[sunfishcode]

Wasm's portability model is to expose a virtual platform. We need to make some rules for this virtual platform. Is XDG a good convention to follow?

People will use these options if there are no other choices :-). WASI is sandboxed, so applications can't reach out and access arbitrary absolute paths.

Another question is whether it's worth separating these known paths (XDG, if we follow that) from paths that users supply explicitly. We could add a scheme: prefix for paths like URIs.

[soc]

Is XDG a good convention to follow?

Can you clarify what you mean with this? It's a good convention to follow on Linux, because that's the rules there, but it would be a poor idea to follow on macOS or Windows.

From my point of view, WASI should provide functions that return cache/config/... dirs (in whatever shape appropriate) and it's the job of the runtime to implement these functions according to the rules of the platform it is running on.

Experience has shown that an API that returns the "generic" cache/config/... directories still results in developers getting the remaining parts wrong (remaining parts: how the organization/application name is turned into a directory).

Having thought a bit more about this, I'm starting to favor the mid-level approach of people specifying their organization/application name, and receiving complete paths to their specific application's cache/config/... directory.

I. e.

projectDirs(qualifier, organization, application).configDir()

instead of

configDir().append(/* some string that is likely to be wrong on all but one platform */)

I think it would be a worthwhile discussion whether qualifier, organization, application arguments should be provided explicitly (like above), or whether they are derived from the application's meta-data somehow. (Which in turn creates the question of how to handle the distinction between libraries and applications.)

[kubkon]

I think what @sunfishcode meant (prove me wrong here please if I twisted anything!) was that nothing is set in stone yet when it comes to offering support for cache/config/... dirs so we're in the position now to agree on a convention that WASI would use and stick with it.

As @sunfishcode mentioned, Wasm's portability model is exposing a virtual, sandboxed platform, hence we could easily follow an XDG there. In other words, a Wasm app run in a virtual guest platform such as wasmtime, would perceive it as XDG-compliant entity, while behind-the-scenes, wasmtime would map the cache/config/... dirs from within the guest platform into some real location on the host. Then, directories-rs and the lower-level dirs-rs would treat WASI as XDG-compliant, while on our side, the Wasm runtime's side, we'd worry about properly mapping those config dirs into some real location on the host. Now, what that would be is a different story, and I like @sunfishcode suggestion here to intially add a command line option --cache, etc. to wasmtime which would create a bridge between some dirs on the host and the cache/config/... dirs in WASI. This way I believe that we preserve the security model of the sandbox.

[soc]

@kubkon Can you expand what you mean with

a virtual, sandboxed platform, hence we could easily follow an XDG there

and

XDG-compliant

in general?

Then, directories-rs and the lower-level dirs-rs would treat WASI as XDG-compliant

What exactly would this entail?

XDG got mentioned repeatedly already and I still don't understand what role it is supposed to play inside that platform: How would a XDG-compliant platform look like, compared to a non-compliant platform or a $something-compliant platform?

[kubkon]

Sure thing! So, to my limited but hopefully expanding understanding, WASI is meant to be a portable sandbox with no concept of a full virtual filesystem. Rather, the only paths present in its environment will be paths which were explicitly mapped by the user. In more abstract terms, WASI is a capability-based sandbox: you can only access whatever you're explicitly provided by the user.

To expand a bit further on that, suppose that upon launching a WASI runtime, I explicitly map the folder /home/me/.config as .config. Then, the apps running in WASI will see .config dir in its relative form only (am I correct here @sunfishcode?) which will point behind-the-scenes to /home/me/.config on my Linux host. However, say I was running WASI runtime on Windows host, then I could equally well map RoamingAppData to .config dir also.

Building on that, as suggested by @sunfishcode, we could now agree on a convention to use which would be a special mapping of sorts, in which config dirs on different hosts (you could run a WASI runtime on Linux, Win, or Mac) would be mapped to the same location in WASI, for instance, .config if following the XDG-like spec. Now, from the perspective of dirs-rs and directories-rs, you'd be given a file handle to the relative .config dir or whatnot, whereas we'd implement the actual mapping to a config location or whatnot in WASI runtime. Or the user of WASI would explicitly specify where the config files reside on the host (e.g., via a --config flag, etc.).

Does this make sense, or did I convolute it even further? :-)

[soc]

Ok, that clears things up. I would not offer access to the "generic" config/cache/... directories, I would only offer access to the application's specific config/cache/... folders.

This is because almost no developer knows the rules of how the application-specific parts need to be constructed, and even if they did, making people use e. g. the Linux/XDG conventions wouldn't allow you to derive the correct paths for the mapping on macOS or Windows.

I'd strongly recommend going the more high-level route, where the developer provides qualifier, organization and application name, and the runtime maps to the correct OS-level directory using the correct rules for each platform.

[kubkon]

Sounds good to me. We could give it a spin and see where this takes us :-)

[soc]

For example, here is how the qualifier, organization and application arguments get translated according to the various platforms' rules:

Linux: https://github.com/soc/directories-rs/blob/master/src/lin.rs#L81
macOS: https://github.com/soc/directories-rs/blob/master/src/mac.rs#L81
Windows: https://github.com/soc/directories-rs/blob/master/src/win.rs#L88

I tried quite a few different API approaches, because I found it a bit onerous to force people to specify parameters which might not even be used on the particular platform they are targeting, but all alternative approaches were worse.

[soc]

@kubkon Cool, let me know if I I can help!

[sunfishcode]

+1 here :-).

Wasmtime could have a command-line parameter (which will eventually be an API parameter, when it's refactored to be fully usable as a library) something like --config="uk.co/Mega Corp/SuperApp" which could (a) create the appropriate host path, and (b) expose it to the wasm program as a preopened directory. This could also be cached or set up in a config file, so users don't need to type the full string all the time.

Looking at the conventions, Linux's are fairly terse; it seems like since we have a choice here, we might as well keep all the path components, like this: /wasi/config/uk.co/Mega Corp/SuperApp. We could also have convenience functions in wasi-sysroot for constructing these paths.

This will still require some porting effort in applications, because these paths aren't the Linux, Windows or Mac paths. But doing it this way will allow us to avoid exposing the concept of a "home" directory, which would weaken the capability model.

[soc]

we might as well keep all the path components, like this: /wasi/config/uk.co/Mega Corp/SuperApp

What would be the benefit of this?

From my point of view, I think it would make more sense to keep the path as opaque as possible to applications, as whatever form of display WASI picks will be wrong on every – except one – platform.

If there absolutely has to be some "string" representation of the path, I'd pick something that looks as invalid as a path could possibly look like, on as many platforms as possible, like :appConfig: or ?<appCache> or similar.

[soc]

I have thought about the question of libraries vs. applications trying to retrieve config/cache/... folders, and would love to hear your thought on it:

Basically, for applications things are straight-forward: provide your qualifier/org name/app name and receive the paths.

But for libraries, there are two options:

• The library wants to use its own config/cache/... directories
• The library wants t use the config/cache/... directories of the applications it is linked into.

If the WASI format required some metadata that specified

• whether their code is supposed to be a library or an application
• the qualifier/org name/app name
  it would allow setting some thing up correctly _by default_, such that the API exposed to developers would not even require passing qualifier/org name/app name when retrieving config/cache/... directories.

This would also allow having functions for libraries that behaved like "give me the config/cache/..." of the application I'm linked into – in fact they could be the same as the functions for applications!

Is there anything planned for the WASI metadata format yet in this regard?