Powershell: Start-Process -UseNewEnvironment provides an environment that is missing crucial standard environment variables while not supporting passing a new environment

[mklement0]

On both Windows and Unix platforms, Start-Process -UseNewEnvironment results in an environment that is missing crucial standard environment variables, making the new environment virtually useless, while not providing a mechanism to define a new environment:

• On Windows, -UseNewEnvironment defines _only_ the variables that are _explicitly_ defined, as displayed in System Properties (sysdm.cpl), with crucial, usually automatically defined variables such as $env:ProgramFiles missing, and $env:USERNAME unexpectedly containing SYSTEM.

  • This even prevents another powershell instance from being started this way - see https://github.com/PowerShell/PowerShell/issues/3545#issuecomment-320699488 and below.

• On Unix, an _empty_ environment is essentially passed (with _no_ environment variables defined at all, except $env:PSModulePath).

  • This even prevents invoking any executable by filename only, given that $env:PATH is undefined - see below.

  • While this behavior is similar to POSIX env -i on Unix platforms, the env utility notably includes a mechanism to provide a _new_ environment via name=value pairs, which Start-Process lacks.

---

Possible solutions:

• Repurpose -UseNewEnvironment to not start with a blank slate / crucial variables missing, but to provide the same environment that a shell would see when directly launched by the system (in other words: any environment-variable modifications made by the calling shell would be ignored).

• Additionally, provide a way to pass a custom environment, applied _on top_ of the current or pristine (-UseNewEnvironment) environment:

  • E.g., a new -Environment <hashtable> / -Environment <Collections.DictionaryEntry[]> parameter could be used.

  • If someone truly wanted an _empty_ environment, they could start with $emptyEnv = (Get-ChildItem env:); $emptyEnv | % { $_.Value = $null } and pass -Environment $emptyEnv.

---

The -Environment feature would allow for an - incomplete - approximation of the convenient ad-hoc, command-scoped environment-variable definition feature that POSIX-like shells such as bash offer, where you can prepend one or more name=value pairs to a command:

# Bash (any POSIX-compatible shell)
# Defines $env:FOO as 'bar', but *only for the some-utility child process*.
FOO=bar some-utility 666

_Update_: #3316 suggests emulating this syntax in PowerShell, which would be the best solution.

The PS approximation would be:

Start-Process -Wait  -Environment @{ FOO = 'bar' } some-utility -Args 666

That said, a crucial limitation is that use of Start-Process makes the external utility operate outside PowerShell's streams, so the only way to provide input / collect output is via the -Redirect* parameters, which requires _auxiliary files_.

Current behavior

Start-Process -UseNewEnvironment -Wait -NoNewWindow pwsh -args '-Command', 'gci env:; whoami'

• On Windows: PowerShell refuses to start - see #3545; note that adding -LoadUserProfile makes no difference.

Internal Windows PowerShell error.  Loading managed Windows PowerShell failed with error 8009001d.

• On Unix: PowerShell starts, lists $env:PSModulePath as the _only_ environment variable - with a seemingly temporary user account's module directory prepended - and the whoami invocation fails, because it cannot be located in the absence of a suitable $env:PATH.

Name                           Value                                                                                                                           
----                           -----                                                                                                                           
PSModulePath                   /tmp/ba38e79f-40c2-440e-ae08-7cf32e0708e1/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/...
whoami : The term 'whoami' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path 
was included, verify that the path is correct and try again.
At line:1 char:21
+ Get-ChildItem env:; whoami
+                     ~~~~~~
    + CategoryInfo          : ObjectNotFound: (whoami:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Environment data

PowerShell Core v6.0.0-beta.5 on macOS 10.12.6
PowerShell Core v6.0.0-beta.5 on Ubuntu 16.04.3 LTS
PowerShell Core v6.0.0-beta.5 on Microsoft Windows 10 Pro (64-bit; v10.0.15063)
Windows PowerShell v5.1.15063.483 on Microsoft Windows 10 Pro (64-bit; v10.0.15063)

[GeeLaw]

On Windows, you should use CreateEnvironmentBlock function to create a complete block of environment variables for a specified user.

There is another issue on UseNewEnvironment. When combined with Credential, it currently starts the subprocess in the new user's context, but with the current user's environment variables, which is surely a bad thing to do. If instead, you turn off UseNewEnvironment and supply Credential, the subprocess will use the default environment block for the specified user, which is likely what the user wants.

[mklement0]

To provide some more insight into what, specifically, is wrong (in addition to the _conceptual_ flaw that @GeeLaw mentions with respect to combining -UseNewEnvironment with -Credential; almost sounds like that combination should be disallowed):

https://github.com/PowerShell/PowerShell/blob/5fcab21111be1551d723bbd3cf2701d79e44fa94/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs#L1867-L1872

On Unix-like platforms, the LoadEnvironmentVariable() calls are effective _no-ops_, because Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Machine) and Environment.GetEnvironmentVariables(EnvironmentVariableTarget.User) quietly(!) return an _empty_ hashtable in Unix, where these targets aren't supported (see https://github.com/dotnet/corefx/issues/32685).

In short, LoadEnvironmentVariable() shouldn't even be called to begin with on non-Windows platforms.

As for _Windows_:

• The original post mentions that crucial built-in variables are missing from the resulting process, which results from the approach of first _clearing_ all variables with

https://github.com/PowerShell/PowerShell/blob/5fcab21111be1551d723bbd3cf2701d79e44fa94/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs#L1869

and then only restoring those variables whose definitions are _registry-based_.
However, there are a number of crucial variables such as %USERPROFILE% that are _not_ registry-based, so they end up missing altogether - see below.

• Also, loading a default $env:PATH value is broken in LoadEnvironmentVariable(), because key access is _case-sensitive_ and the variable name is mistakenly tested for with all-uppercase name PATH:

https://github.com/PowerShell/PowerShell/blob/5fcab21111be1551d723bbd3cf2701d79e44fa94/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs#L2077

---

For the record, here are the built-in, system-defined environment variables that are _not_ registry-based:

ALLUSERSPROFILE
APPDATA
CommonProgramFiles
CommonProgramFiles(x86)
CommonProgramW6432
COMPUTERNAME
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
ProgramData
ProgramFiles
ProgramFiles(x86)
ProgramW6432
PUBLIC
SESSIONNAME
SystemDrive
SystemRoot
USERNAME    # !! There is a *machine-level* definition with value 'SYSTEM', but there is no *user-level* definition - that is added by the system to the process.
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERPROFILE

Command that generated the list above (with the exception of USERNAME)

gci env: | % name | ? { -not [environment]::GetEnvironmentVariable($_, 'Machine') -and -not [environment]::GetEnvironmentVariable($_, 'User')}

[FranklinYu]

I put the "possible solution 2" in #8352. Even if this issue is fixed by implementing solution 1, I believe solution 2 has other use-cases.

Update: that's a dupe of #3316.

[iSazonov]

I'd deprecate UseNewEnvironment (make alias?) and introduce ClearEnvironment
I like new Environment parameter but can we explicitly say that we _add/change_ variable? Maybe -SetEnvironmentVariable?

[SteveL-MSFT]

On GitHub, there's a small amount of usage of -UseNewEnvironment. I believe the intent of this switch is you would start a process from PowerShell as though you just started a new PowerShell process (not from the current PowerShell process). We just have to evaluate if we would break any existing usage (seems unlikely).

As for specifying an environment, I think -Environment @{} is good.

[mklement0]

Agreed re intent, @SteveL-MSFT, so if we wanted to fix this feature (and also make it work on Unix), we could cache a copy of the environment-variable block on PowerShell startup (before reading profiles), and use that.

@iSazonov: I agree that -SetEnvironmentVariable is more descriptive, but it's a bit unwieldy. I think with proper documentation -Environment will do, as Steve says.

-ClearEnvironment wouldn't quite capture the intent of the feature, which mustn't _clear all_ variables - that it currently does so on Unix is what prompted this report.

[iSazonov]

I think really we need three operation:

1. ClearEnvironment [optional hash with new variables to set]
2. RemoveEnvironment
3. SetEnvironment

UseNewEnvironment we could deprecate and hide.

[mklement0]

I don't think there is a need for this complexity:

Re 1 and 2:

You rarely want to start with _no environment variables at all_ - you'll wipe out vital _OS_-defined variables too (which is the current problem on Unix-like platforms).

It is far more common to _build on_ an existing environment, either:

• the current one

• "pristine" one, without any in-session changes, but with the usual OS-defined ones present - that's what -UseNewEnvironment _should do_.

Re 3:

Combining the hashtable -Environment parameter with -UseNewEnvironment then covers all use cases:

As a user, I then:

• decide which environment to start with: the current one (no switch) or a "pristine" one (-UseNewEnvironment)

• then use -Environment to _modify_ that starting environment, by:

  • adding new variables
  • modifying the values of existing variables
  • removing variables

The first 2 modifications are most common, but a single, hashtable-based environment variable also supports _removing_ variables, given that assigning $null or the _empty string_ to environment variables _deletes_ them; e.g., -Environment @{ FOO='bar'; BAR=$null } would _create or update_ env. var FOO and _remove_ (or not define) env. var. BAR

[iSazonov]

I don't think there is a need for this complexity:

Historically environment variables is manipulated by one "Set" operation, even in C# API https://docs.microsoft.com/en-us/dotnet/api/system.environment.setenvironmentvariable?view=netframework-4.8#System_Environment_SetEnvironmentVariable_System_String_System_String_
But a question is can we get PowerShell magic benefits having explicit operations like *-Variables?

CommandType     Name 
-----------     ----
Cmdlet          Clear-Variable
Cmdlet          Get-Variable
Cmdlet          New-Variable
Cmdlet          Remove-Variable
Cmdlet          Set-Variable

Earlier we actively discussed Environment provider/drive. We found this very interesting. Obviously the API will has these operations explicitly. Should we keep consistency with the API in the issue too (also taking in account hosting scenario)?

I do not like changing UseNewEnvironment for 4 reasons: unlikely but breaking change, the name confuses me, not clear why we need the rare scenario and implementation will slow down startup that is very bad. I believe this is not worth the effort until there is a real important business scenario. Moreover, PowerShell actually only changes PSModulePath, and this can be solved in #10300.

Environment.SetEnvironmentVariable Method (System)

Creates, modifies, or deletes an environment variable.

[mklement0]

The *-Variable cmdlets are narrowly focused on _shell_ variables (PowerShell's own, in-session only variables).

However, extending the Environment PS provider to also support managing _persisted_ environment variables, at least on Windows, makes perfect sense to me, and it is what I've proposed here.

I don't see a problem with having that coexist with the -Environment <hashtable> parameter proposed here, or even with the syntax discussed in #3316.

I do not like changing UseNewEnvironment for 4 reasons: unlikely but breaking change

Nothing _meaningful_ can be broken, given the currently useless behavior.

the name confuses me

I think the name is fine, but we could implement an alias, if there's consensus on a better name.

not clear why we need the rare scenario and implementation will slow down startup that is very bad

That is a valid concern - I can't speak to the real-world impact.

I believe this is not worth the effort until there is a real important business scenario.

_Personally_, I won't miss the feature, but I can see how it is desirable in certain situations.

Moreover, PowerShell actually only changes PSModulePath

That's not the point, though. The point is that _you yourself_ or third-party code may have modified the environment in multiple ways that you don't want external processes to see, lest it interfere with their proper operation.

[iSazonov]

That's not the point, though. The point is that you yourself or third-party code may have modified the environment in multiple ways that you don't want external processes to see, lest it interfere with their proper operation.

Interesting, has Windows or Unix an API to directly implement the scenario? (I mean this is too unbelievable scenario)

[mklement0]

Windows has Start-Process -UseNewEnvironment :) - though it is partially broken, as I've pointed out. However, I've always understood the _intent_ of that feature to be what I've proposed.

Unix has the env utility with the -i option (ignore the inherited environment, and only define the variables passed as the other arguments);

On closer examination, however, this appears to be more like a -ClearEnvironment switch, and the only reason it results in a usable shell if you call bash is that bash - unlike PowerShell - apparently sets _default_ values for variables such as $env:PATH.

As a cross-platform shell, however, we'd be looking for standardized behavior, so IF we keep the switch, implementing the proposed behavior makes the most sense to me.

Again, I personally wouldn't miss -UseNewEnvironment, and I can see that it is probably rarely needed overall (as at least historically indicated by @SteveL-MSFT's GitHub search).
I think that's all I have to say on this particular aspect of this debate, and I'll let others decide.

[iSazonov]

It seems UseNewEnvironment doesn't work on Unix. 😕

[mklement0]

@iSazonov: Well, like env -i, I think it radically clears the environment, though that is _accidental_ behavior (the current attempts to read persisted env.-var definitions are quiet no-ops on Unix-like platforms, as explained above).

However, there's more going on, because env -i bash --norc results in a functional shell, whereas the seemingly equivalent Start-Process -UseNewEnvironment bash --norc exhibits very strange behavior with the default terminal apps on macOS 10.14 and and Ubuntu 18.04.

Only with iTerm on macOS could I get Start-Process -UseNewEnvironment bash --norc to work, and then call export to see what env. variables are defined: you'll see that they're only _Bash-defined_ ones (as with env -i bash --norc), implying that an _empty_ environment was passed, and Bash seemingly furthermore declares _shell-only_ variables (you can see these among the output from set) of the same name as vital OS env. variables such as PATH and TERM so as to provide at least a minimal level of functioning.

On both platforms, trying to launch pwsh this way _always_ results in broken behavior, because PowerShell does not anticipate and compensate for the absence of vital environment variables.

[iSazonov]

This code is This code is somewhat confusing (in CoreFX too) 😕

This works in WSL (may be useful for tests):

Start-Process -UseNewEnvironment -ArgumentList "set" bash

[iSazonov]

We need to take in account #6489 Again slow down :-(
If we look Process.ProcessStartInfo.Environment - it force us to copy Dictionary and it again slow down.

This is not so important to start a process but it can be sensitive for startup scenario.

[mklement0]

As for the test command: You're missing -c, and it's better to also suppress loading of .bashrc with --norc; set shows you a _mix_ of shell-only and environment variables, so it's better to use export; thus:

Start-Process  -UseNewEnvironment  -ArgumentList '--norc', '-c', 'export'  bash

If you're experimenting with passing _multiple_ commands to -c, remember the Start-Process bug with respect to spaces (#5576), so embedded quoting is needed:

Start-Process  -UseNewEnvironment  -ArgumentList '--norc', '-c', '"export; set"'  bash

[mklement0]

Also, I don't think we need to worry about #6489 for simply making a copy of the startup environment.

As for the dictionary: are you referring to [environment]::GetEnvironmentVariables()? [System.Diagnostics.Process]::GetCurrentProcess().StartInfo is $null in a PowerShell session.

What [environment]::GetEnvironmentVariables() returns already _is_ a copy of the environment, returned as a case-sensitive [hashtable] (IDictionary) whose later manipulation has no effect on the underlying environment variables.

[iSazonov]

Also, I don't think we need to worry about #6489 for simply making a copy of the startup environment.

Perhaps we can catch a problem if original set contains duplicate variables: we have to set them one-by-one and last will win. Although we use P/Invoke on Windows and this could allow to get around this.

[System.Diagnostics.Process]::GetCurrentProcess().StartInfo is $null in a PowerShell session.

It is not property in Process object. So we can use only [environment]::GetEnvironmentVariables().

Start-Process -UseNewEnvironment -ArgumentList '--norc', '-c', 'export' bash

It does not work for me on WSL. Folow works:
Start-Process -UseNewEnvironment -ArgumentList '--norc', '-c', 'printenv' bash

I hope we can use this on all Unix-s and MacOs.

[mklement0]

Yes, printenv is also preinstalled on macOS (it isn't a POSIX-mandated utility, however), so this should work; I'm baffled that export wouldn't work, though, given that it is a Bash _builtin_ (an internal command).

[mklement0]

I just discovered the cmd.exe-internal start command's /I switch, which was presumably the model for PowerShell's -UseNewEnvironment switch (in intent, not in effect, due to the flawed current implementation); from start /? (lightly edited):

/ I ... The new environment will be the original environment passed
to the cmd.exe and not the current environment.

[iSazonov]

@mklement0 Thanks! I pulled PR - please review.

[iSazonov]

Since we had PowerShell Committee conclusion in https://github.com/PowerShell/PowerShell/pull/10745#issuecomment-542923823 we can continue here to discuss a desired behavior.
From my understanding the conclusion was to implement on all platforms behavior like @rjmholt made for login shell on Unix - with UseNewEnvironment subprocess gets environment like we get with sh -l on Linux and zsh -l on macOS. I don't know can we do the same on Windows.
@SteveL-MSFT @mklement0 Thoughts?

[mklement0]

Thanks, @iSazonov; I think you meant to link to https://github.com/PowerShell/PowerShell/pull/10745#issuecomment-542923823 - given that specific points were made there, let me respond there.

[SteveL-MSFT]

@iSazonov on Windows, the current code clears the env block, then fills it from the system profile (hence username is system) and then the user profile overwriting any that exist in the system. Hence Path is incomplete. It seems the fix here is to merge Path (append user PATH to end of machine PATH). USERNAME should be special cased to the user. The test is to start a new cmd.exe from Windows shell, run set and have the env vars match if using -UseNewEnvironment.