Powershell: Update CoreCLR to the latest packages

[vors]

On @PowerShell/powershell-maintainers meeting we decided to update all coreclr packages regularly once per release.
We should add back dotnet preview feed.

[daxian-dbw]

Changes to System.Runtime.Loader.AssemblyLoadContext breaks PowerShellAssemblyLoadContext and cause powershell to crash. So the work on this issue is blocked.

The change in AssemblyLoadContext is "an assembly loaded in the default context will not be visible by a different load context, and that means requesting for the same assembly (exactly the same full name) from another load context will get that assembly loaded again."

In open powershell, powershell.exe is actually a shim that initializes the powershell load context, and tell the powershell load context the real entry method. The PS load context initialization code will create a singleton of itself and assign it to a static member “PowerShellAssemblyLoadContext.Instance”. Then the PS load context will load the actual assembly where the real entry method lives and call into the real entry method. So after that, all implicit assembly loading request will be handled by the PS load context. The PS load context singleton is used by powershell a lot, and it’s supposed to be referenced by using the static member “PowerShellAssemblyLoadContext.Instance”. This stopped working after I switched to the latest .NET Core packages. “PowerShellAssemblyLoadContext.Instance” turns out to be null when the actual powershell code starts to run.

I believe this is because the change described above. Using “PowerShellAssemblyLoadContext.Instance” in actual powershell code will trigger an implicit load of the PS load context assembly, and due to the change, the PS load context singleton loaded by the default load context will load the same PS load context assembly again, and thus the static member in that newly loaded PS load context assembly will certainly be null.

I'm having a conversation with CoreCLR team on this, and will update the issue when get more information.

[andschwa]

There are also changes to the HttpClient used by Invoke-WebRequest; it looks like System.Net.WebClient was brought back, and so I'm not sure what happened to the current System.Net.Http APIs @Francisco-Gamino used.

[daxian-dbw]

Updating with the latest .NET Core preview packages causes some regressions:

1. Latest System.Runtime.Loader.AssemblyLoadContext causes a regression in PSAssemblyLoadContext. The regression has been fixed on PS side.
2. With .NET Core 1.0.0 RTM, dotnet.exe doesn't verify the existence of assemblies that are built from the local projects when check the TPA deps assemblies, and thus we are able to remove IL assemblies after crossgen. However, the latest dotnet.exe checks the existence of locally built assemblies, so IL assemblies cannot be removed anymore.
3. The new dotnet.exe loads IL assemblies in precedence, instead of NI assemblies. The symptom is that you will find PowerShellAssemblyLoadContext.dll is loaded by the default load context when both the IL and NI assemblies exist. But with RTM packages, PowerShellAssemblyLoadContext.ni.dll gets loaded by the default load context under the same circumstance. This doesn't affect us too much now because PowerShellAssemblyLoadContext handles the loading of other PS assemblies and it prefers NI over IL assemblies. However, when the legacy APIs are back and it's time to retire PSALC, our crossgen story will be broken if the default context prefer IL over NI. I will start a conversation with .NET Core team on this.
4. Latest Microsoft.Win32.Registry will now raise PlatformNotSupportedException when you call any static members of it on UNIX. In PS code, we call some of its static properties when initializing some types (e.g. System.Management.Automation.Utils), and that caused PS to crash in UNIX. This is fixed.
5. When working on #4, I found both Registry and Certificate providers are shown up in UNIX PS. They don't work in UNIX at all, and should not be exposed (until we have a UNIX implementation of Certificate provider someday). They are now excluded from compilation for UNIX PS.
6. [System.Text.Encoding]::Default is available via reflection at runtime (exposed by System.Private.CoreLib), but it's not exposed in the reference contract of System.Text.Encoding, and thus we cannot use it in PS Code. This mismatch causes a redirection test to fail. The test is fixed to skip Encoding.Default for now. I will check with .NET Core team to clarify this mismatch.
7. The exclusion of 'Certificate' provider causes some failures on UNIX PS, because some tests are using Cert:\ to test the error behavior when not using a FileSystemProvider drive. The tests are fixed.

[daxian-dbw]

@andschwa I didn't observe failures in Invoke-WebRequest and Invoke-RestMethod. I think we should be fine, the HttpClient and HttpMessageHandler are the way to go. Maybe WebClient is brought back just for compatibility reason.

[daxian-dbw]

[For #2 and #3, I talked to CoreCLR team about this and here is the detailed explanation]
Now by comparing the “Property TRUSTED_PLATFORM_ASSEMBLIES” in trace messages, I understand that in RTM, the assemblies produced by local projects are NOT listed in the TPA at all, and the default load context actually depends on APP_PATHS and APP_NI_PATHS to find them. And yes, that indicates the default load context actually prefer NI assemblies over IL assemblies when it can find/see both.

With the latest dotnet.exe, assemblies produced by local projects are now in the TPA list, and APP_APTHS and APP_NI_PATHS are no longer passed in. Therefore, the default load context can only see the IL assemblies, and thus only load the IL assemblies.

_another option would be to only drop you NI image but with the same name as the IL image_

This alternative is suggested by CoreCLR team and has been taken care of by PR #2484.

[daxian-dbw]

[For #6, I checked with CoreFx team]

Encoding.Default will not be brought in the reference contract in Nov, probably as part of our 1.2 release in spring. In general it is isn’t a good idea to use Encoding.Default as it is not the right think for most cases. I would highly suggest you pick the correct encoding for the scenario.

So we just need to keep our current implementation of the default encoding.

[daxian-dbw]

Resolved via PR #2472