Powershell: Special folders and $env differ on Linux

[jherby2k]

Steps to reproduce

On Windows:
$env:LocalAppData returns C:\users\appdata\local
$([System.Environment]::GetFolderPath(28)) returns C:\users\appdata\local

On Linux:
$env:LocalAppData returns nothing
$([System.Environment]::GetFolderPath(28)) returns /home//.local/share/

Expected behavior

PowerShell's $env should return the same special folder paths returned by .NET Core's Environment.GetFolderPath() method.

Actual behavior

See above

Environment data

Name Value
---- -----
PSVersion 6.0.4
PSEdition Core
GitCommitId v6.0.4
OS Linux 4.15.0-1019-azure #19~16.04.1-Ubuntu SM...
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

[iSazonov]

Such env variables is absent on Unix. Although for more predictable behavior of scripts between platforms we could implement the default values.

[jherby2k]

I definitely get that “special folders” and “environment variables” are slightly different concepts, but I think it might make sense to set some variables within PowerShell that don’t otherwise exist in the environment, for cross-platform compatibility. In my case this was a failing build script on Linux that required the workaround.

[mklement0]

@jherby2k: I think a better solution to this problem is to provide a special namespace via a new provider that mirrors [Environment]::GetFolderPath() (you don't want to create Windows-only _environment variables_ on Unix); using your example:

[Environment]::GetFolderPath('LocalApplicationData')

would become:

$sf:LocalApplicationData

See #6966 for details.

[iSazonov]

The new namespace does not provide backward compatibility.

[mklement0]

@iSazonov:

True, but I doubt that any existing scripts that were written for just Windows would magically just work if only those environment variables were present on Unix.

If there's really a need for backward compatibility, the WindowsCompatibility module is probably the right place for such shims.

[iSazonov]

@mklement0 So shouldn't the new namespace address the issue too?

[iSazonov]

Also how WindowsCompatibility module can help run windows script on Unix?

[mklement0]

Re module: my bad, I guess I engaged in wishful thinking regarding the purpose of that module - it only facilitates integrating with Windows PowerShell.

However, the point is that I don't think that functionality that projects a Windows world view onto Unix is a sensible cross-platform strategy - such functionality should be _opt-in_, if needed at all.

So shouldn't the new namespace address the issue too?

Address what issue?

[iSazonov]

Address what issue?

I meant the Issue we are discussing here - windows script is not work on Unix due to the lack of environment variables. I believe we could emulate the environment variables internally.

Although there is one more problem - perhaps there is a conflict with .Net Core special folders on Unix.

[mklement0]

believe we could emulate the environment variables internally.

That's precisely what I suggest we avoid (to put it bluntly: hacks that pretend that Unix is Windows).

CoreFx already provides platform _platforms abstractions_ with its [Environment]::GetFolderPath() method; the right thing to do is to surface its functionality in a PowerShell-idiomatic way - which is the proposed $sf: namespace.

What conflicts?

[iSazonov]

What conflicts?

On Unix special folders is mapped here
https://github.com/dotnet/corefx/blob/b384b309061c050a31dcf2b8f377f5da244fcf7b/src/System.Runtime.Extensions/src/System/Environment.Unix.cs#L139

• it can be unexpected behavior for PowerShell scripts. We need review the mapping.

[mklement0]

I've tried to summarize the cross-platform behavior here: https://github.com/PowerShell/PowerShell/issues/6966#issuecomment-393882898

And there is indeed a conflict between $HOME and [Environment]::GetFolderPath('MyDocuments'):

• you get $HOME _itself_ on Unix
• you get $HOME\Documents on Windows

[iSazonov]

So a conclusion here is that:

1. Portable script should use [Environment]::GetFolderPath with special folders names.
2. User could define Windows env variables on Unix as needed. It is a feature for a compatibility module.
3. We don't want fix the issue in the engine by emulating Windows env variables on Unix.
4. We could open an issue in PowerShell script analyzer to detect such problems in scripts (perhaps it already exists).

[jherby2k]

Just because [Environment]::GetFolderPath is clunky and requires looking up the enum, as a user i'd prefer something like Get-SpecialFolder with auto completion.

[iSazonov]

@jherby2k Please look #6966 - perhaps there is a proposed solution (Get-Location).

[mklement0]

Great summary, thanks, @iSazonov.

Yes, [Environment]::GetFolderPath() is clunky and there's no tab completion.

#6966 evolved over time: it started out with suggesting a -SpecialFolder parameter for Get-Location / Set-Location, but it now _only_ proposes the sf: drive (provider), which automatically gives us the $sf: namespace, analogous to the Env: drive / $env: pair.

The advantage of this approach is that you get support in _any_ context (not tied to a particular cmdlet).

For instance, you'd type be able to type $sf:us<tab> and it would expand to $sf:UserProfile - whether as a command argument, an expression operand, or inside an expandable string.

Note: sf stands for special folder; the name is negotiable, but it should be succinct; kf for known folder is another option

[SteveL-MSFT]

The discrepancy of MyDocuments returning $HOME on Unix rather than $HOME\Documents seems like a bug in corefx.

It seems like a SF: drive is a good solution to this.

[mklement0]

Glad to hear it, @SteveL-MSFT.

The MyDocuments behavior is unfortunate, but I wouldn't call it a bug. Presumably, the decision was based on the fact that Unix-like platforms, unlike Windows, have no restriction on placing files and folders _directly_ in the user's home folder.
However, in the age of macOS and friendly Linux distros such as Ubuntu, which come with predefined folders similar to Windows, $HOME\Documents arguably makes more sense.

It's certainly worth considering making a - documented - exception in PowerShell to eliminate this discrepancy when we surface the functionality.

[dragonwolf83]

If not all special folders are available cross-platform, the SF: drive could make it easier to separate them from OS specific folders by using a subdirectory. For example, SF:\Win\WindowsOnlyFolder. This makes it clear that the folder won't exist on Linux or macOS

[iSazonov]

@dragonwolf83 Seem your suggestion make sense only if we have _conflicts_ between of platforms. In any case we want that a script works on all platforms without modifications.

[mklement0]

It's definitely worth tagging the individual entries of the sf: drive with what platforms they're defined on (where they have nonempty paths as their values).

To that end, I suggest attaching a .SupportedOS property to each item, which could be an [enum] type as follows:

[Flags()]
enum SupportedOs {
  Windows = 0x1
  Linux = 0x2
  macOS = 0x4
}

Not sure if that makes for the most convenient way to query the information. however.

[jherby2k]

Isn’t it just as easy to see which ones are empty? If you need to know how they’re going to resolve on a platform other than the one you’re currently on, I think just referring to documentation would suffice.

[mklement0]

@jherby2k: While we could get away with just _documenting_ the platform-specific behavior, my vote is for continuing PowerShell's rich tradition of _programmatic_ discovery, i.e., reflection.

Therefore, running Get-ChildItem sf: could yield something like the following on macOS:

Name                   Path                             SupportedOs
----                   ----                             -----------
AdminTools                                                  Windows
ApplicationData        /Users/jdoe/.config                      All
CDBurning                                                   Windows
CommonAdminTools                                            Windows
CommonApplicationData  /usr/share                               All
CommonDesktopDirectory                                      Windows
CommonDocuments                                             Windows
CommonMusic                                                 Windows
CommonOemLinks                                              Windows
CommonPictures                                              Windows
CommonProgramFiles                                          Windows
CommonProgramFilesX86                                       Windows
CommonPrograms                                              Windows
CommonStartMenu                                             Windows
CommonStartup                                               Windows
CommonTemplates                                             Windows
CommonVideos                                                Windows
Cookies                                                     Windows
Desktop                /Users/jdoe/Desktop                      All
DesktopDirectory       /Users/jdoe/Desktop                      All
Favorites              /Users/jdoe/Library/Favorites Windows, macOS
Fonts                  /Users/jdoe/Library/Fonts     Windows, macOS
History                                                     Windows
InternetCache          /Users/jdoe/Library/Caches    Windows, macOS
LocalApplicationData   /Users/jdoe/.local/share                 All
LocalizedResources                                          Windows
MyComputer                                                  Windows
MyDocuments            /Users/jdoe                              All
MyMusic                /Users/jdoe/Music                        All
MyPictures             /Users/jdoe/Pictures                     All
MyVideos                                             Windows, Linux
NetworkShortcuts                                            Windows
PrinterShortcuts                                            Windows
ProgramFiles           /Applications                 Windows, macOS
ProgramFilesX86                                             Windows
Programs                                                    Windows
Recent                                                      Windows
Resources                                                   Windows
SendTo                                                      Windows
StartMenu                                                   Windows
Startup                                                     Windows
System                 /System                       Windows, macOS
SystemX86                                                   Windows
Templates                                            Windows, Linux
UserProfile            /Users/jdoe                              All
Windows                                                     Windows

---

In fact, the above was obtained with the following code, based on the findings in https://github.com/PowerShell/PowerShell/issues/6966#issuecomment-393882898, which is all that is needed (though it must be kept in sync with .NET Core manually; that said, changes will likely be few and far between):

[Flags()]
enum SupportedOs {
  All = 0x7 # !! Sum of all the flags below - BE SURE To UPDATE THIS IF NEW FLAGS ARE ADDED
  Windows = 0x1
  Linux = 0x2
  macOS = 0x4
}

$specialFolders = [ordered] @{
  'AdminTools' = [pscustomobject] @{ Name = 'AdminTools'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'ApplicationData' = [pscustomobject] @{ Name = 'ApplicationData'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'CDBurning' = [pscustomobject] @{ Name = 'CDBurning'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonAdminTools' = [pscustomobject] @{ Name = 'CommonAdminTools'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonApplicationData' = [pscustomobject] @{ Name = 'CommonApplicationData'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'CommonDesktopDirectory' = [pscustomobject] @{ Name = 'CommonDesktopDirectory'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonDocuments' = [pscustomobject] @{ Name = 'CommonDocuments'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonMusic' = [pscustomobject] @{ Name = 'CommonMusic'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonOemLinks' = [pscustomobject] @{ Name = 'CommonOemLinks'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonPictures' = [pscustomobject] @{ Name = 'CommonPictures'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonProgramFiles' = [pscustomobject] @{ Name = 'CommonProgramFiles'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonProgramFilesX86' = [pscustomobject] @{ Name = 'CommonProgramFilesX86'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonPrograms' = [pscustomobject] @{ Name = 'CommonPrograms'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonStartMenu' = [pscustomobject] @{ Name = 'CommonStartMenu'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonStartup' = [pscustomobject] @{ Name = 'CommonStartup'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonTemplates' = [pscustomobject] @{ Name = 'CommonTemplates'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'CommonVideos' = [pscustomobject] @{ Name = 'CommonVideos'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Cookies' = [pscustomobject] @{ Name = 'Cookies'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Desktop' = [pscustomobject] @{ Name = 'Desktop'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'DesktopDirectory' = [pscustomobject] @{ Name = 'DesktopDirectory'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'Favorites' = [pscustomobject] @{ Name = 'Favorites'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS' }
  'Fonts' = [pscustomobject] @{ Name = 'Fonts'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS' }
  'History' = [pscustomobject] @{ Name = 'History'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'InternetCache' = [pscustomobject] @{ Name = 'InternetCache'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS' }
  'LocalApplicationData' = [pscustomobject] @{ Name = 'LocalApplicationData'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'LocalizedResources' = [pscustomobject] @{ Name = 'LocalizedResources'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'MyComputer' = [pscustomobject] @{ Name = 'MyComputer'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'MyDocuments' = [pscustomobject] @{ Name = 'MyDocuments'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'MyMusic' = [pscustomobject] @{ Name = 'MyMusic'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'MyPictures' = [pscustomobject] @{ Name = 'MyPictures'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'MyVideos' = [pscustomobject] @{ Name = 'MyVideos'; Path = $null; SupportedOs = [SupportedOs] 'Windows, Linux' }
  'NetworkShortcuts' = [pscustomobject] @{ Name = 'NetworkShortcuts'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'PrinterShortcuts' = [pscustomobject] @{ Name = 'PrinterShortcuts'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'ProgramFiles' = [pscustomobject] @{ Name = 'ProgramFiles'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS' }
  'ProgramFilesX86' = [pscustomobject] @{ Name = 'ProgramFilesX86'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Programs' = [pscustomobject] @{ Name = 'Programs'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Recent' = [pscustomobject] @{ Name = 'Recent'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Resources' = [pscustomobject] @{ Name = 'Resources'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'SendTo' = [pscustomobject] @{ Name = 'SendTo'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'StartMenu' = [pscustomobject] @{ Name = 'StartMenu'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Startup' = [pscustomobject] @{ Name = 'Startup'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'System' = [pscustomobject] @{ Name = 'System'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS' }
  'SystemX86' = [pscustomobject] @{ Name = 'SystemX86'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
  'Templates' = [pscustomobject] @{ Name = 'Templates'; Path = $null; SupportedOs = [SupportedOs] 'Windows, Linux' }
  'UserProfile' = [pscustomobject] @{ Name = 'UserProfile'; Path = $null; SupportedOs = [SupportedOs] 'Windows, macOS, Linux' }
  'Windows' = [pscustomobject] @{ Name = 'Windows'; Path = $null; SupportedOs = [SupportedOs] 'Windows' }
}

$specialFolders.GetEnumerator() | % { 
  $_.Value.Path = [Environment]::GetFolderPath($_.Value.Name); $_.Value 
}

[mklement0]

As a stopgap, I've published advanced function Get-SpecialFolder in this Gist, which

• by default only lists those folders that are defined for the OS at hand
• supports tab-completion of the special folder names

Here's the concise form of the CLI help (Get-SpecialFolder -?):

SYNTAX
    Get-SpecialFolder [-All] [<CommonParameters>]

    Get-SpecialFolder [-Name] <String[]> [<CommonParameters>]


DESCRIPTION
    Gets items representing special folders (directories), i.e.,
    folders whose purpose is predefined by the operating system.

    In a string context, each such item expands to the full, literal path it
    represents.

    If no name is given, all special folders known to the current OS
    are listed. Use -All to include those that are special on other platforms.