Runtime: System.Environment - Writing environment variables on Mac/Linux

[polarapfel]

Hi,

@stephentoub @jkotas @JeremyKuhne @eerhardt

I've been looking at System.Environment and how to read and write Environment variables on Linux/Mac, using the 1.04 release and 2.0 preview currently available. With the 2.0 preview, the APIs expanded to allowing the use of the EnvironmentVariableTarget enum, which features the Process, User and Machine scope where as prior to this, only the Process scope was implemented as a default.

Unfortunately, this is not helpful at all on Linux/Mac. With the current API in 2.0 preview (and prior API versions), setting environment variables on Linux/Mac do not bind to the underlying shell process, they live within the scope of the dotnet process executing the assembly setting the environment variable(s) and stop to exist once dotnet terminates assembly execution. Using other scopes (e.g. User, Machine) on non-Windows platforms) is not mapped in the implementation. This is fairly useless for persisting information to be used by other processes.

Example:

using System;
using System.Runtime.InteropServices;

class Sample 
{
    protected static string myVarPrefix = "FOOBAR_";
    protected static string myVarA = myVarPrefix + "DEFAULT"; // default process
    protected static string myVarB = myVarPrefix + "CURRENT"; // Current Process
    protected static string myVarC = myVarPrefix + "USER"; // Current User
    protected static string myVarD = myVarPrefix + "MACHINE"; // Local Machine

    protected static string myVarValue = "FOOBAR";

    public static void Main() 
    {
        Console.WriteLine("Setting default process...");
        Environment.SetEnvironmentVariable(myVarA, myVarValue);

        Console.WriteLine("Setting current process...");
        Environment.SetEnvironmentVariable(myVarB, myVarValue, EnvironmentVariableTarget.Process);

        Console.WriteLine("Setting user...");
        Environment.SetEnvironmentVariable(myVarC, myVarValue, EnvironmentVariableTarget.User);

        Console.WriteLine("Setting machine...");
        try
        {
            Environment.SetEnvironmentVariable(myVarD, myVarValue, EnvironmentVariableTarget.Machine);
        }
        catch (System.Exception)
        {
            Console.WriteLine("Failed setting machine...");
        }

        if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux)) System.Diagnostics.Process.Start("/bin/bash", "-c export");

        Console.Write("Press <Enter> to exit... ");
        while (Console.ReadKey(true).Key != ConsoleKey.Enter) {}
    }
}

Only FOOBAR_DEFAULT and FOOBAR_CURRENT will be set on Linux while the app executes and running /bin/bash -c export _after_ this sample terminated shows that both variables are gone and don't exist anymore.

What's the plan here to fix this?

To my knowledge, Mono simply implements reading/writing environment variables on Unix/Linux using the same namespace(s) as in .NET, Mono.Unix.Native/Mono.Posix have nothing added to specifically read and write environment variables. Hence, my expectation would be to have this very basic stuff handled with an identical abstraction in CoreFX that is not platform dependent?

For Linux, there is of course no scope such as "Machine" and "User" would bind to the underlying shell process (and any child process starting after the fact through inheritance).

The current scope of implementation in 2.0 preview should be fixed to make the "User" scope work on Mac/Linux, where it binds to the scope of the underlying shell.

As a workaround to properly set environment variables on Linux/Mac, one could simply query RuntimeInformation.IsOSPlatform(OSPlatform.Linux) and use System.IO to check for the presence of any familiar shell (sh, ksh, and bash, maybe also from other shell families such as csh) in the file system and then write simple wrappers calling to their built-in functions to set environment variables correctly such as

// omitting check for presence of bash binary
if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux)) System.Diagnostics.Process.Start("/bin/bash", "-c export " + variable + "=" + value);

Whatever the API implementation in CoreFX is going to be (for Mac/Linux), it should work and have the same outcome as making a call to the shell built-ins for managing environment variables.

--

Thanks,

Tobias W.

[JeremyKuhne]

Mono doesn't implement User/Machine.

Conceptually I'm OK with trying to expand the support here- it would be cool to get better equivalency. It is, however, out of scope for 2.0.

[jkotas]

System.Diagnostics.Process.Start("/bin/bash", "-c export " + variable + "=" + value);

I do not think that this works (just tried as well to be sure). export will export the variable for the child of the current shell process to use. This line is starting a new bash process that exits immediately. It means that it has no effect. It does not set the variable in any sort of global or persisted environment.

[tmds]

binds to the scope of the underlying shell.

That's not possible. The environment of a process is determined at process creation (by the parent process). After that, only the process itself can change the environment.

[polarapfel]

@tmds @jkotas You're both right. I ignored that calling bash that way will result in its own environment rather than its parent's environment.

What's the solution then if I want to set an environment variable that outlives the execution time of the current assembly?

[jkotas]

I assume you meant to say how to set an environment variable that outlives the execution time of the current process.

There is no reasonable way to do it on Unix. You need to use some other communication mechanism between child and the parent process - files, pipes, etc.

[polarapfel]

@JeremyKuhne Based on my initial misconception on how dotnet could hook into shell builtin functionality to set environment variables in Linux for the parent shell, I think we can scrap that idea.

@jkotas

Maybe it's still a good idea to add an extra paragraph within the remarks section for Mac/Linux to the documentation for setting environment variables? Currently, there is no information in there, explaining how this will work when executed on Linux/Mac.

--

Some guidance on how to end up with similar behavior to using the "User" scope for setting variables in Windows when trying to get to the same behavior on Linux/Mac might help developers writing cross platform apps using dotnet.

On Windows, I can just rely on the User scope for environment variables when I want to persist a value in the environment across executables running in their own process. That'll work fine.

On Linux/Mac, you're right, there's no convenient way, but I played with this idea and it works for my purpose (with caveats that can be taken care of):

• I can only manage the environment from within the actual shell for the shell's environment, child processes (including dotnet executing an assembly) cannot modify the environment.
• I can work around that by using a shell script wrapping the dotnet execution and the assembly I want to execute. This shell script does have access to the environment of its current shell (as it executes in this context), so I can write environment variables in the same context. Calling my dotnet app will always happen through this shell script on Mac/Linux, making this wrapping shell script the place to manage the environment.
• I can either have the dotnet app write values to a file or standard out, grabbing them from there in the wrapping shell script. The shell script doesn't need to terminate once the dotnet app terminates and can still write back values to the environment after the dotnet process terminates. It only needs a place to grab the values from that originate from within the dotnet app.
• I need to make sure command line parameters to the script are passed through to the dotnet app, as well as making sure any exit codes coming from the dotnet app terminating is passed through once the wrapping script terminates.
• If I want to persist variables beyond the current shell, I can even append a script setting these variables to $HOME/.profile (or similar), so any new shell being initiated, will source the variables from there.
• When relying on BASH, I can also use any of the BASH specific environment variables (e.g. such as $BASHPID), if I need to make file names of files storing variable values specific to an identified parent shell PID.

--

Thoughts?

[tmds]

Thoughts?

Throw PNSE for EnvironmentVariableTarget.{User,Machine} on Unix.

[jkotas]

Maybe it's still a good idea to add an extra paragraph within the remarks section for Mac/Linux

Agree. We do have improving discoverability/documentation for platform differences on our backlog.

[danmosemsft]

Throw PNSE for EnvironmentVariableTarget.{User,Machine} on Unix.

Seems to me this we should definitely do. @polarapfel any interest in submitting PR for that? The fix is here https://github.com/dotnet/coreclr/blob/master/src/mscorlib/src/System/Environment.cs#L647 and https://github.com/dotnet/coreclr/blob/master/src/mscorlib/src/System/Environment.cs#L757

and a separate PR for tests here

https://github.com/dotnet/corefx/blob/master/src/System.Runtime.Extensions/tests/System/Environment.GetEnvironmentVariable.cs
https://github.com/dotnet/corefx/blob/master/src/System.Runtime.Extensions/tests/System/Environment.SetEnvironmentVariable.cs
(may be best to split into files just for unix)

[stephentoub]

Seems to me this we should definitely do

Why? Arguments in favor of what we currently have:

• It's what Mono does. Do we know that it's caused problems for Mono customers?
• It allows more code to "just work".

[polarapfel]

@stephentoub I can see your point about Mono code remaining easily portable. But just silently ignoring unsupported environment scopes leads to situations where a developer familiar with how this works on Windows, wrongfully assumes it works the same way on Mac/Linux. For that reason, I actually think a PNSE makes sense in both CoreFX and Mono's .NET implementation.

I think addressing documentation will already help as a first step. I'm happy to look into PR to add PNSE for unsupported scopes.

[felixfbecker]

I recently added macOS support to ps-nvm, which uses [Environment]::SetEnvironmentVariable() to persist a default NodeJS version across shell restarts:

https://github.com/aaronpowell/ps-nvmw/blob/9784e5ada938a7485a6786371ac76b3ed78d37fb/nvm.psm1#L113

The scope argument being present in .netcore and the documentation not mentioning any different behaviour for macOS/Linux, I would have expected it to work on macOS. I don't know what exactly the Windows version does, but I assume it writes the variable to the registry. Therefor I would have expected something equivalent to happen on macOS - e.g. use launchctl setenv to set it, write to /etc/paths or whatever.

[moinoviimir]

I ran into this same issue, and it took me a while to get through to the root cause. Lack of exceptions or relevant docs was a bit of an issue.

I feel a PR is in order, to the code or the docs or both, in order to save the next person their time. @polarapfel have you had the time to work on your PR? If not, I could try to look into that.

[roozbehid-ic]

I just ran into this. took few hours to know what was wrong!

[jkotas]

I have moved this issue to the documentation repo: https://github.com/dotnet/docs/issues/6277