Powershell: pwsh 7 can't parse path in WSL when using Windows integration

[gdoenlen]

I just upgraded to pwsh 7 in my WSL instance and it causes the process to exit when using windows integration of $PATH with an error of: bad variable name: (x86)\Nvidia

It persists even after removing this folder from my windows path (the next error is about a bad variable for Files/Microsoft. This only occurs when starting a fresh WSL instance and doesn't occur when running pwsh from bash.

Changing HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss\{YOUR_GUID}\Flags to 5 will disable the integration and fix this behavior.

Changing the above registry key to 7 (enable integration) and using bash as your shell will also fix this behavior.

Steps to reproduce

1. set your WSL shell to pwsh 7 with: chsh -s /path/to/pwsh
2. Make sure windows integration of path is enabled in the registry: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss\{YOUR_GUID} and the Flags key is set to 7. Setting this key to 5 turns the integration off and pwsh will start fine. Additionally setting this back to 7 and setting bash as your shell will have WSL start fine as well.
3. Restart WSL and the path should fail (depending on your windows path of course).

Expected behavior

Path should be properly parsed.

Actual behavior

Failures related to bad variable names.

Environment data

Name                           Value
----                           -----
PSVersion                      7.0.0
PSEdition                      Core
GitCommitId                    7.0.0
OS                             Linux 4.4.0-18362-Microsoft #476-Microsoft Fri Nov 01 16:53:00 PST 2019
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[iSazonov]

I think it is a problem with Windows paths sharing.
See https://github.com/microsoft/WSL/issues/1640.

@SteveL-MSFT @rjmholt @sdwheeler I believe this is "by-design" and the workaround could be documented.

[gdoenlen]

Yes I know it's from path sharing, hence the registry edits. The work around isn't ideal as you lose access to everything in windows without typing the full path to something (i.e. vscode). Pwsh 6 definitely didn't have the same behavior.

[rjmholt]

It’s a known issue arising from the fact that 7, unlike 6, supports being run as a login shell.

This has shed new light on it though

[iSazonov]

@rjmholt Is it WSL or PowerShell issue? I did not find we parse PATH in the code.

[rjmholt]

Quite possibly a PowerShell issue. exec cares about the path. We use it here:

https://github.com/PowerShell/PowerShell/blob/348ca859475eb0094f97791d0f6c8717577d2935/src/powershell/Program.cs#L325-L330

[iSazonov]

@rjmholt You referenced MacOs code :-).

In any case we don't get explicitly PATH value and don't parse it.

Can you catch the exception in debug build?

[rjmholt]

You referenced MacOs code

The second part is Linux, but my point is about exec.

If exec is failing there will be no exception; it's a native API.

I could be wrong about the provenance of the issue though. We need to see the whole error message (if you could add that please @gdoenlen)

[gdoenlen]

@rjmholt The part after export will change depending on what it encounters in the path first.

: 6: export: Files/Microsoft: bad variable name

[process exited with code 2] <-- this is probably from windows terminal

[gdoenlen]

Does powershell leave a log anywhere? I could check that as well.

[rjmholt]

Does powershell leave a log anywhere

I think it can be configured to, but I don't think at the level we're looking for; it's designed to log commands rather than its internals. But all of this happens so early anyway that we avoid all possible work to reduce startup time (since we have to start up twice).

[iSazonov]

So we see bash error for export. I guess it is possible to repo without PowerShell.

[gdoenlen]

Just for reference I believe /bin/sh points to dash and not bash. My posix shell isn't so great but maybe it's running something from /etc/profile or /etc/profile.d when you use it as a login shell? Though it is odd that bash and zsh as my login shell don't do the same.

[rjmholt]

Just for reference I believe /bin/sh points to dash and not bash. My posix shell isn't so great but maybe it's running something from /etc/profile or /etc/profile.d when you use it as a login shell?

Interesting. We specifically intend to work with dash, but running as login deliberately runs either /etc/profile or ~/.profile. Those are supposed to be /bin/sh compatible according to POSIX

[gdoenlen]

I assume they are posix compliant as setting bash or zsh with chsh -s /path/to/{zsh, bash} runs correctly with the integration on. I'm not sure what is being done different in those situations. I did see a wsl-integration.sh file in /etc/profil.d, but again my shell isn't that great so not sure what's going on there.

[gdoenlen]

So I've reproduced this with setting my shell to dash. chsh -s /bin/dash. The output is: -dash: 6: export: Files/Microsoft: bad variable name but the process doesn't exit like when pwsh tries to start. It seems like you might want to explicitly try for bash on linux if exists and then fall back to /bin/sh.

[rjmholt]

It seems like you might want to explicitly try for bash on linux if exists and then fall back to /bin/sh.

Yeah I think that sounds sensible. We did our best to keep it generic in order to support dash since some distros ship without bash, but it seems the world depends on bashisms.

[iSazonov]

If it is dash issue why does not request its fix?

[gdoenlen]

In my opinion, I would prefer it to be fixed here. Getting a fix into dash and then getting that fix packaged into distros will take time and I'm not even sure if that fix would be back ported by canonical/Microsoft. The WSL kernel is 4.4 after all. And after all that, /bin/sh could point to tcsh, ksh, or some other unknown shell that might be required to be fixed and back ported as well. Lastly, I don't think it's a ton of work to do a: if (File.Exists("/bin/bash") { ... } and be able to have this working right now.

[iSazonov]

Only thing we need from /bin/sh - PATH env variable creation. If dash can not do this on WSL I don't see a point to complicate PowerShell code.

[mklement0]

Just to add some background:

It sounds like there may be a command such as export PATH=foo:$PATH somewhere, right?

For this command to be POSIX-compliant (if the value of $PATH contains spaces), it would have to be written as export PATH="foo:$PATH" (double-quoting of the RHS), and Bashisms indeed do not belong in /etc/profile or ~/.profile

dash is the most strictly POSIX-compliant among the major POSIX-like shells out there (a bare-bones shell built for speed to speed up booting, with very few features added to the POSIX-mandated ones).

[gdoenlen]

It sounds like there may be a command such as export PATH=foo:$PATH somewhere, right?

Maybe, but if this is the case it's something written by Canonical or the WSL team that comes with the installation of WSL.

From ~/.profile:

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

[iSazonov]

@gdoenlen Could you please review all your system files and find where dash set PATH as @mklement0 mentioned? It'd great to discover where the error comes from.

[mklement0]

In my pristine WSL installation of Ubuntu 18.04.1 LTS I see no evidence of non-POSIX compliant code in /etc/profile or ~/.profile, and running sh -l doesn't cause a problem.

If add export FOO=/bar/baz:$PATH to ~/.profile, running sh -l surfaces the problem (with Windows path integration turned on).

Also note that when the system boots, it invariably processes /etc/profile (but not ~/.profile) with /bin/sh, i.e., dash.

When pwsh re-invokes itself via /bin/sh -l, the user-specific ~/.profile comes into the mix.

What puzzles me is that WSL invokes the user's default shell as a _login_ shell to begin with - which the default terminal emulator in a standalone Ubuntu installation does _not_ do (verify with echo $0 with bash as the default shell, for instance: on WSL, you'll see -bash, indicating a login shell; in a standalone Ubuntu with the default terminal emulator, you'll see bash).

[gdoenlen]

n Could you please review all your system files and find where dash set PATH as @mklement0 mentioned? It'd great to discover where the error comes from.

I'll try tonight.

In my pristine WSL installation of Ubuntu 18.04.1 LTS I see no evidence of non-POSIX compliant code in /etc/profile or ~/.profile, and running sh -l doesn't cause a problem.

I'm not sure where it would come from then. I'm not a basher so I haven't customized any of it. Have you checked /etc/profile.d ?

[mklement0]

Good catch, @gdoenlen:

While running /bin/sh -l interactively also processes the scripts in /etc/profile.d/, the offending command is inside a _conditional_ that happened not to be true on my system, so the problem didn't surface:

From /etc/profile.d/apps-bin-path.sh:

# Expand $PATH to include the directory where snappy applications go.
snap_bin_path="/snap/bin"
if [ -n "${PATH##*${snap_bin_path}}" -a -n "${PATH##*${snap_bin_path}:*}" ]; then
    export PATH=$PATH:${snap_bin_path}
fi

That is:

• This script is _not robust_ due to the lack of "..." around the RHS and shouldn't have been written that way, knowing that all POSIX-compliant shells have to be able to execute it properly.

  • This is especially unfortunate, because only a few lines later the robust syntax _is_ used:
    export XDG_DATA_DIRS="${XDG_DATA_DIRS}:${snap_xdg_path}"

• As for the impact:

  • The problem usually doesn't surface in stand-alone Ubuntu installations, because having directory paths _with spaces_ in PATH is atypical.

  • However, if you did add such entries, the /bin/sh (dash) instance that processes /etc/profile _on system startup_ (and from which all user processes inherit their environment) would report a nonzero exit code that causes a blocking GUI error message to pop up during startup.

  • Additionally, even on Windows (with Windows path integration enabled) the problem only surfaces if directory /snap/bin isn't already in listed in PATH

  • In a standalone, full Ubuntu (virtual) machine, /snap/bin _is_ already in the PATH by the time a user's default shell is run, given that snap (package manager Snappy) comes preinstalled on Ubuntu 16.04+, and the variable is set when /bin/sh processes /etc/profile _during system startup_. It is for that reason that Ubuntu terminal emulators do _not_ invoke user shells as _login_ shells - it simply isn't necessary.

  • That isn't the case on WSL, however, which is not a full Ubuntu VM, but a compatibility layer on top of the Windows kernel, so there's no boot-time /bin/sh process that processes /etc/profile; to make up for that WSL _does_ invoke user shells as _login_ shells.

---

Thus, using bash indeed sounds like the right pragmatic workaround.

This is unfortunate, because for long-term stability, speed (though the impact is probably negligible), and implementation simplicity, /bin/sh -l is the right choice.

[gdoenlen]

Thank you @mklement0 ! It seems like shellcheck fails to catch this as well.

/etc/profile.d $ shellcheck -s sh apps-bin-path.sh

In apps-bin-path.sh line 5:
if [ -n "${PATH##*${snap_bin_path}}" -a -n "${PATH##*${snap_bin_path}:*}" ]; then
                                     ^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.


In apps-bin-path.sh line 19:
if [ -n "${XDG_DATA_DIRS##*${snap_xdg_path}}" -a -n "${XDG_DATA_DIRS##*${snap_xdg_path}:*}" ]; then
                                              ^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.

[mklement0]

Good point, @gdoenlen - a request to add such a check has already been made (by the author of shellcheck himself): https://github.com/koalaman/shellcheck/issues/951

[mklement0]

I've updated the previous comment to paint what I hope is the full picture, but in a nutshell:

• The root cause is the poorly written /etc/profile.d/apps-bin-path.sh script that ships with Ubuntu.
• The problem surfaces predictably in WSL (with Windows path integration turned on, which is true by default) with pwsh as the user's default shell, given that WSL of necessity always invokes user shells as a _login_ shell to ensure /etc/profile processing - unlike in actual Linux (virtual) machines, where that isn't necessary.
  
  
  
  • The problem can hypothetically also occur in actual Linux machines, albeit with a different symptom: if you add a directory with spaces to PATH (e.g. via /etc/enviroment), a blocking error message will pop up during system startup.