Powershell: Tests related to `-SslProtocol` in web cmdlets are failing on macOS daily runs

[daxian-dbw]

Same failures from 2 macOS daily builds. And we got the similar failures for Invoke-RestMethod -SslProtocol.

    [-] Verifies Invoke-WebRequest -SslProtocol Tls11 works on Tls11 145ms
      SSL connect error
      at line: 1527 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1
    [-] Verifies Invoke-WebRequest -SslProtocol Tls12 works on Tls12 125ms
      SSL connect error
      at line: 1527 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1
+++
    [-] Verifies Invoke-WebRequest -SslProtocol -SslProtocol Tls11 fails on a Tls only connection 213ms
      Expected string length 83 but was 13. Strings differ at index 0.
      Expected: {WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand}
      But was:  {No Exception!}
      -----------^
      at line: 75 in /Users/travis/build/PowerShell/PowerShell/test/tools/Modules/HelpersCommon/HelpersCommon.psm1
      75:             $_.FullyQualifiedErrorId | Should Be $FullyQualifiedErrorId | Out-Null
    [-] Verifies Invoke-WebRequest -SslProtocol -SslProtocol Tls12 fails on a Tls only connection 181ms
      Expected string length 83 but was 13. Strings differ at index 0.
      Expected: {WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand}
      But was:  {No Exception!}
      -----------^
      at line: 75 in /Users/travis/build/PowerShell/PowerShell/test/tools/Modules/HelpersCommon/HelpersCommon.psm1
      75:             $_.FullyQualifiedErrorId | Should Be $FullyQualifiedErrorId | Out-Null

[markekraus]

I noticed that the openssl version is different between the last passing daily and the next failing daily.

1.0.2l (failed): https://travis-ci.org/PowerShell/PowerShell/jobs/309351359#L142
1.0.2j (passed): https://travis-ci.org/PowerShell/PowerShell/jobs/308273280#L174

That seems to coincide with the changes in the default macos image on Travis CI https://blog.travis-ci.com/2017-11-21-xcode8-3-default-image-announce

This is the kind of weird behavior that happens on a systems with the non-opnessl libcurl.

brew has 1.0.2m available. I have that on my macOS laptop. I will test master and see if this repros

[daxian-dbw]

@markekraus Thanks for looking into it!

[markekraus]

I can't repro this with openssl 1.0.2l or 1.0.2m. I noticed they also added [email protected] to the image. I installed this and and tested and still cant' repro this. I'm on macOS 10.13.1 and another change they made was the move to 10.12.6. So i can't duplicate this environment entirely.

[daxian-dbw]

I'm able to reproduce the failure:

> $env:PATH = "/Users/psbuildacct/PowerShell/test/tools/WebListener/bin/:" + $env:PATH
> Import-Module /Users/psbuildacct/PowerShell/test/tools/Modules/HttpListener 
> Import-Module /Users/psbuildacct/PowerShell/test/tools/Modules/WebListener
> $response = Start-HttpListener -Port 8080
> $WebListener = Start-WebListener
> $Uri = Get-WebListenerUrl -Test 'Get' -Https -SslProtocol Tls11 
> Invoke-WebRequest -Uri $Uri -SslProtocol Tls11 -SkipCertificateCheck  

Invoke-WebRequest : SSL connect error
At line:1 char:1
+ Invoke-WebRequest -Uri $Uri -SslProtocol Tls11 -SkipCertificateCheck
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (Method: GET, Re...rShell/6.0.0
}:HttpRequestMessage) [Invoke-WebRequest], HttpRequestException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

This happens on macOS 10.12.6 with openssl 1.0.2m installed.
Here are the exception call stack and error messages:

PS:19> $er.Exception.Message                                                                                                                         
An error occurred while sending the request.
PS:20> $er.Exception.StackTrace                                                                                                                      
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncUnbuffered>d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(HttpClient client, HttpRequestMessage request, Boolean stripAuthorization) in /Users/psbuildacct/PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/WebRequestPSCmdlet.CoreClr.cs:line 457
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord() in /Users/psbuildacct/PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/WebRequestPSCmdlet.CoreClr.cs:line 550

PS:21> $er.Exception.InnerException.Message                                                                                                          
SSL connect error
PS:22> $er.Exception.InnerException.StackTrace                                                                                                       
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)

I have no idea about the root cause. For now I will disable -SslPrototol tests.

[daxian-dbw]

One more thing I want to bring up is that our macOS CI actually uses openssl 1.0.2m. The bootstrap script runs brew install curl --with-openssl, which installs the 1.0.2m. See the CI screenshot below:

[markekraus]

yea I missed where it was installing 1.0.2m

The question for me is whether this is WebListener not getting the right TLS applied, or the web cmdlets (HttpClient) not selecting the right tls setting.. or both. I don't have a good way to test that except maybe verify with curl that limiting those protocols to those ports works against web listener. I could not find any public web services that had Tls 1.0, 1.1, or 1.2 only.

I have been trying several things on travis CI, but it takes an eternity for macOS builds to start and finish. can you do a brew list --versions in your environment?

I'm waiting on a build to start that runs brew remove [email protected] before boot strapping. I can't reproduce an issue with it installed on my system with it installed... but maybe it's an issue for 10.12.6?

[daxian-dbw]

but it takes an eternity for macOS builds to start and finish

Yes, it's really a pain to verify anything with the Travis CI mac builder.

can you do a brew list --versions in your environment

Sure, here is the output:

$ brew list --versions
cmake 3.6.2
curl 7.54.0
hub 2.2.9
makedepend 1.0.5
openssl 1.0.2j 1.0.2k 1.0.2m
pkg-config 0.29.1_2 0.29.2

but maybe it's an issue for 10.12.6?

That is a possibility -- I never ran those tests on the macOS I'm using now.

[iSazonov]

If a consumer has a slightly different version of the library, PowerShell will not work. Can we inplace such sensitive libraries in PowerShell Core folder?

[markekraus]

@iSazonov This does not appear to be related to the libcurl library or any other. As far as I can tell, this is an issue between CoreFX and macOS 10.12.6. The same library versions and build process works on 10.12.5 and 10.13.1, but not 10.12.6.

It could also be that the issue is on the WebListener side. there could be something in the 10.12.6 environment that is causing kestrel to bind the wrong protocols to the ports.

[iSazonov]

issue between CoreFX and macOS 10.12.6

I guess that the Issue is that CoreFX uses libcurl. We inplace CoreFX libraries in PowerShell Core folder. I guess we could put there a libcurl of needed version.

[markekraus]

@iSazonov Again, the libcurl is not the issue here. the exact same libcurl library version works fine on macOS 10.12.5 and macOS 10.13.1.

I would caution against packaging libcurl with the project. My experience is that it is one of the least portable Linux libraries.

We already have a step in the build process that ensure the right version is installed. i mistakenly thought other wise earlier in digging into this issue.

[iSazonov]

@markekraus Thanks for clarify!
Can you test Kestrel by browser locally on MacOS? (On Windows in IE we can select TLS versions.)

[markekraus]

@iSazonov the problem I have is that I cannot reproduce this in my mac environment. @daxian-dbw was able to on his 10.12.6 system.

But yes, you can test WebListener in a browser

Import-Module ./build.psm1
Publish-PSTestTools
$WebListener = Start-WebListener

Then use the Get-WebListenerUrl to get the various URLs.

The problem is that this all needs to be don on an environment that is like the one in Travis CI.

Before @daxian-dbw opened #5605 I was trying to do some troubleshooting in Travis CI, but it started taking 4-5 hours to start the macOS builds. 😭

Best way to test would be to use openssl s_client against the WebListener on you can see what TLS version is used in the output.

echo -e "GET / HTTP/1.1\nHost: localhost\n\n" | openssl s_client -connect localhost:8084 -ign_eof 2>&1 | grep 'TLSv' | grep 'Protocol'
echo -e "GET / HTTP/1.1\nHost: localhost\n\n" | openssl s_client -connect localhost:8085 -ign_eof 2>&1 | grep 'TLSv' | grep 'Protocol'
echo -e "GET / HTTP/1.1\nHost: localhost\n\n" | openssl s_client -connect localhost:8086 -ign_eof 2>&1 | grep 'TLSv' | grep 'Protocol'

should be

    Protocol  : TLSv1.2
    Protocol  : TLSv1.1
    Protocol  : TLSv1

[markekraus]

This may be related to libcurl after all.

I have done a ton of testing on my mac and I have discovered that no matter what brew version of curl/libcurl I have installed and no matter how I try to trick pwsh into using the correct lubcurl, it keeps using the system supplied one in /usr/lib/, which uses LiberSSL.

I was testing if the newer version of libcurl and dotnet sdk resolved the inability to use certificate authentication on macOS and it spits this at me:

The handler does not support client authentication certificates with this combination of libcurl (7.54.0) and its SSL backend ("LibreSSL/2.0.20").

Which is odd considering I have curl 7.57.0 (with OpenSSL) installed by brew.

curl --version

```none
curl 7.57.0 (x86_64-apple-darwin17.2.0) libcurl/7.57.0 OpenSSL/1.0.2m zlib/1.2.11
Release-Date: 2017-11-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

You can reproduce the error with:

```powershell
Import-Module ./build.psm1
Publish-PSTestTools
$Url = Get-WebListenerUrl -Test Cert
$certificate = Get-WebListenerClientCertificate 
$result = Invoke-WebRequest -Uri $url -Certificate $certificate -SkipCertificateCheck

There hasn't been any way that I have found to see what version of libcurl is using without making it throw an error or some kind.

I'm used to compilers having away to include the correct library when a system has multiple, but I have no clue how do to that with dotnet. I haved tried setting LD_LIBRARY_PATH to /usr/local/opt/curl/lib, I'v tried setting LDFLAGS to -L/usr/local/opt/curl/lib, i have tred copying the correct libcurl files to the same directory as pwsh.... no matter what i do it sill choses the system libcurl files.

[iSazonov]

Interesting comment.

Update: The other comments are also useful.

[markekraus]

@iSazonov thanks. I'm not sure that plays into this, but at least that pointed me in a direction where I can test this. https://github.com/markekraus/PowerShell/blob/TestMacosFails/test/powershell/Libcurl.Tests.ps1

This is the build job (still pending) https://travis-ci.org/markekraus/PowerShell/builds/311234209

[markekraus]

Yep, The reason why this is failing is because it is using the wrong libcurl:

https://travis-ci.org/markekraus/PowerShell/builds/311234209#L2884

LIBCURl=====================================================
LIBCURL GetSslVersionDescription(): SecureTransport
LIBCURL GetSupportedFeatures(): CURL_VERSION_IPV6, CURL_VERSION_SSL, CURL_VERSION_LIBZ, CURL_VERSION_NTLM, CURL_VERSION_ASYNCHDNS, CURL_VERSION_SPNEGO, CURL_VERSION_LARGEFILE, CURL_VERSION_NTLM_WB, CURL_VERSION_GSSAPI, CURL_VERSION_KERBEROS5, CURL_VERSION_UNIX_SOCKETS
LIBCURL GetSupportsHttp2Multiplexing(): False
LIBCURL GetVersionDescription(): 7.54.0
LIBCURl=====================================================

[iSazonov]

@markekraus Great work! Could you get a path to the curl? Can reflection help?

[iSazonov]

Maybe related #4029

[markekraus]

@iSazonov from what I can tell, .NET calls the System.Net.Http.Native assembly (link) and not directly to libcurl. That assembly either contains links to the system libcurl or has that libcurl baked into it. I'm not sure what voodoo magic it is doing in that assembly. I'm not sure how you generate that assembly other than to obtain it through nuget. This part of the problem is out of my depth as I am very very unfamiliar with the dotent build process. We need someone who knows what they're doing in that knowledge area to take a look.

re: #4029 I suspected that at first there was some macOS/libcurl/CoreFX issue at play, but I went digging into the code and was not sure it is even using libcurl for any of that. Looks to me like they are missing the brew version of openssl.

[iSazonov]

It seems both Issues related with wrong library version (curl and OpenSSL/LibreSSL).
So questions is:

• Where is the root of the problem? How CoreFX looks for the libraries?
• How can we make it stable on all platforms and distributions?

@daxian-dbw @SteveL-MSFT Can you ping CoreFX team?

[daxian-dbw]

@markekraus @iSazonov I remember seeing a PR in CoreFx about removing OpenSSL dependencies on macOS, and digged into the CoreFx PRs, this is what I found: Remove OpenSSL dependency on macOS (dev/apple_crypto -> master).

Quoted from the PR description:

only the types whose name includes "OpenSsl" (e.g. RSAOpenSsl) will use OpenSSL on macOS. Instead, everything is provided by Security.framework.

[markekraus]

If I understand that PR correctly, previously they were directly using OpenSSL in the project for a bunch of operations and removed that. I don't think this changes anything regarding the issue at hand. You should still be able to plug in the libcurl of your choice as HttpClient is just wrapping it. If 'libcurl' is using OpenSSL then you get the supported features such as Certificate authentication and not-broken HttpClientHandler.SslProtocols.

What is happening instead is that "System.Net.Http.Native" is only using the system provided libcurl. on 10.13.1 that is libcurl+LibreSSL, on 10.12.6 that is libcurl+SecureTransport. We have a step in the build process to install curl with openssl, but instead of it actually using that libcurl, it is favoring the system libcurl.

The brew formula says it all:

==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.bash_profile

For compilers to find this software you may need to set:
    LDFLAGS:  -L/usr/local/opt/curl/lib
    CPPFLAGS: -I/usr/local/opt/curl/include
For pkg-config to find this software you may need to set:
    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig

This at least explains why a change in the OS version by Travis CI would affect the web cmdlets tests: new OS version means a different libcurl version and possibly even a different crypto provider (in the case of 10.12-to-10.13).

That means that the features available and working are beholden to the underlying OS unless we have some way to specify or include the libcurl that is called to ensure it is the one installed from brew.

[daxian-dbw]

@markekraus Thanks for figuring out the root cause!

We have a step in the build process to install curl with openssl, but instead of it actually using that libcurl, it is favoring the system libcurl.

The step in our bootstrapping script that installs curl and openssl is actually the leftover from old days when .NET Core had a hard dependency on openssl. The PR name "Remove OpenSSL dependency on macOS" seems to suggest that we don't need this step anymore 😄

You should still be able to plug in the libcurl of your choice as HttpClient is just wrapping it.

@markekraus Can you please open an issue in CoreFx to check with the CoreCLR team whether it's supported for users to plug in the libcurl of their choices for HttpClient to use?

[markekraus]

I have one more thing to try (setting the DYLD_LIBRARY_PATH env var before starting and compiling) and if that doesn't work I will open an issue.

[markekraus]

also:

seems to suggest that we don't need this step anymore

Except that I think it is still being used by other parts of PowerShell, like omi https://github.com/PowerShell/psl-omi-provider/blob/master/src/CMakeLists.txt 😃

[markekraus]

🎉 YES!! 🎉
Adding export DYLD_LIBRARY_PATH=/usr/local/opt/curl/lib:${DYLD_LIBRARY_PATH}; to the .travis.yml made pwsh pickup the correct libcurl. Now I just need to test that it doesn't break anything.
https://travis-ci.org/markekraus/PowerShell/builds/311469327#L2912