Powershell: On CentOS Powershell uses the system libcurl that does not support custom SSL certificate validation

[atanasa]

This is similar to #2211 but for CentOS. This is probably still an issue with dotnet but maybe powershell can patch it in a similar way as the OS X patch. I haven't found a workaround yet.

Steps to reproduce

Run:

$handler = new-object "System.Net.Http.HttpClientHandler"
$handler.ServerCertificateCustomValidationCallback = { $true }
$client = new-object "System.Net.Http.HttpClient" -Arg @($handler)
$client.GetStringAsync("https://google.com").GetAwaiter().GetResult()

Expected behavior

An error telling you that there is no runtime context on the thread. Which means the execution reached the validation script block.

Actual behavior

Error:

Exception calling "GetResult" with "0" argument(s): "The libcurl library in 
use (7.29.0) and its SSL backend ("NSS/3.19.1 Basic ECC") do not support 
custom handling of certificates. A libcurl built with OpenSSL is required."
At line:1 char:1
+ $client.GetStringAsync("https://google.com").GetAwaiter().GetResult()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : PlatformNotSupportedException

Environment data

> $PSVersionTable
Name                           Value                                           
----                           -----                                           
PSVersion                      6.0.0-alpha                                     
PSEdition                      Core                                            
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                         
BuildVersion                   3.0.0.0                                         
GitCommitId                    v6.0.0-alpha.11                                 
CLRVersion                                                                     
WSManStackVersion              3.0                                             
PSRemotingProtocolVersion      2.3                                             
SerializationVersion           1.1.0.1

[joeyaiello]

Do you know if CentOS ships--inbox or via yum--a libcurl built with libssl?

[atanasa]

I couldn't find one. One of my users found one here:
http://ftp.riken.jp/Linux/cern/centos/7/cern/x86_64/repoview/letter_l.group.html
but that does not look official.

The workaround is to download the libcurl-openssl package from there, then install it with yum localinstall and run

export LD_LIBRARY_PATH=/opt/shibboleth/lib64/:$LD_LIBRARY_PATH

to redirect powershell to discover the new dynamic library.

[atanasa]

Exporting LD_LIBRARY_PATH might redirect other executables (e.g. python) to use the new dynamic library. That might be breaking some of them. To limit the scope of the export, one option is to create a bash script that would act as a shortcut for launching powershell.
Example shortcut script:

#!/bin/bash
export LD_LIBRARY_PATH=/opt/shibboleth/lib64/:$LD_LIBRARY_PATH
powershell

I tried setting the env variable inside powershell instead of bash, but for some reason that didn't affect the dynamic library resolution.

[joeyaiello]

Just for reference, this looks like the upstream CoreFX bug: dotnet/corefx/issues/10146

[vmkdaily]

Hi all,
Patiently waiting for this to get fixed. I'm hoping to run Powershell and PowerCLI on CentOS (without downloading bits from Japan). Thanks for your support!

[ryangurazov]

Hi, please provide an update on the issue.
Very much needed and your effort is very much appreciated.
Thank you.

[getSurreal]

I installed libcurl-openssl from rpm.pbone.net and overwrote the existing libcurl and was able to make a connection with VMware's PowerCLI Core.

mv /opt/shibboleth/lib64/libcurl.so.4.3.0 /usr/lib64

[TravisEz13]

Looks like we are going to face the same issue on OpenSUSE (at least 13.2)

[joeyaiello]

Upstream issue has been moved to dotnet/wcf/issues/1792.

@ryangurazov @getSurreal we're really blocked without the fix to that issue.

[joeyaiello]

This is dependent on wcf, so it's not a beta1 blocker, but given the current milestone that it's assigned to over there, we're hoping it gets fixed in the beta1 milestone.

[daxian-dbw]

HttpClientHandler.DangerousAcceptAnyServerCertificateValidator is added to .NET Core 2.0 to support the scenario of having ServerCertificateCustomValidationCallback always returns true. See the issue https://github.com/dotnet/corefx/issues/19709 and the corresponding PR https://github.com/dotnet/corefx/pull/19941

The -SkipCertificateCheck parameter of Invoke-WebRequest and Invoke-RestMethod will be fixed to work on OSX and CentOS by PR #3887. This issue is essentially a duplicate of #3648.

[daxian-dbw]

Verified that this is fixed by PR #3887

PS /> cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)
PS />
PS /> Invoke-WebRequest -Uri 'https://expired.badssl.com'
Invoke-WebRequest : Peer certificate cannot be authenticated with given CA certificates
At line:1 char:1
+ Invoke-WebRequest -Uri 'https://expired.badssl.com'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Method: GET, Re...rShell/6.0.0
}:HttpRequestMessage) [Invoke-WebRequest], HttpRequestException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
PS />
PS /> Invoke-WebRequest -Uri 'https://expired.badssl.com' -SkipCertificateCheck

StatusCode        : 200
StatusDescription : OK
Content           : <!DOCTYPE html>
                    <html>
                    <head>
                      <meta name="viewport" content="width=device-width, initial-scale=1">
                      <link rel="shortcut icon" href="/icons/favicon-red.ico"/>
                      <link rel="apple-touch-icon" href="/i...
RawContent        : HTTP/1.1 200 OK
                    Server: nginx/1.10.0
                    Date: Wed, 14 Jun 2017 22:10:41 GMT
                    Connection: keep-alive
                    ETag: "5939cdd8-1d5"
                    Cache-Control: no-store
                    Accept-Ranges: bytes

                    <!DOCTYPE html>
                    <html>
                    <head>
                      <meta...
Forms             :
Headers           : {[Server, System.String[]], [Date, System.String[]], [Connection, System.String[]],
                    [ETag, System.String[]]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        :
RawContentLength  : 469
RelationLink      : {}

[daxian-dbw]

We are moving back to .NET Core 2.0.0-preview1 via PR #4026 due to regressions in the latest .NET Core 2.0.0-preview2 (dotnet/corefx#21095). So re-open this bug.

[daxian-dbw]

We moved to .NET Core 2.0.0-preview3 via PR #4144. I have verified that this issue is fixed.

PS /> cat /etc/centos-release                                                                               CentOS Linux release 7.3.1611 (Core)
PS />
PS /> Invoke-WebRequest -Uri 'https://expired.badssl.com'                                                   Invoke-WebRequest : Peer certificate cannot be authenticated with given CA certificates
At line:1 char:1
+ Invoke-WebRequest -Uri 'https://expired.badssl.com'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Method: GET, Re...rShell/6.0.0
}:HttpRequestMessage) [Invoke-WebRequest], HttpRequestException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequest
   Command
PS /> Invoke-WebRequest -Uri 'https://expired.badssl.com' -SkipCertificateCheck                             

StatusCode        : 200
StatusDescription : OK
Content           : <!DOCTYPE html>
                    <html>
                    <head>
                      <meta name="viewport" content="width=device-width, initial-scale=1">
                      <link rel="shortcut icon" href="/icons/favicon-red.ico"/>
                      <link rel="apple-touch-icon" href="/i...
RawContent        : HTTP/1.1 200 OK
                    Server: nginx/1.10.3
                    Date: Fri, 30 Jun 2017 22:29:43 GMT
                    Connection: keep-alive
                    ETag: "595583e8-1d5"
                    Cache-Control: no-store
                    Accept-Ranges: bytes

                    <!DOCTYPE html>
                    <html>
                    <head>
                      <meta...
Forms             :
Headers           : {[Server, System.String[]], [Date, System.String[]], [Connection, System.String[]],
                    [ETag, System.String[]]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        :
RawContentLength  : 469
RelationLink      : {}

[ljckennedy]

Just installed the latest as reference here:
https://github.com/PowerShell/PowerShell/blob/master/docs/installation/linux.md#centos-7

and hit this issue passing a client cert:
Invoke-webrequest -Method $method -Uri $protocol$server$resource -Header $header -certificate $certs -UseBasicParsing -Verbose

Is this a different issue?

NB: No issue on ubuntu..

[markekraus]

@ljckennedy it is the same issue. cert configurations of CentOS and macOS use something other than libcurl-ssl and as a result the underlying CoreFX cannot perform certificate based authentication.

[ljckennedy]

Ok. I assumed marked as 'Closed' meant it was fixed.

[anderssonjohan]

Is this only fixed when using Invoke-WebRequest or should the hack using the callback of the ServicePointManager still work?

I'm trying to use the old hack because of code that uses System.Net.WebClient.DownloadString.
I have just updated everything (powershell and curl) using homebrew and still get the error:

System.Net.WebException: An exception occurred during a WebClient request. ---> System.PlatformNotSuppor
                        tedException: The handler does not support custom handling of certificates with this combination of libc
                        url (7.57.0) and its SSL backend ("OpenSSL/1.0.2n").
                           at System.Net.HttpWebRequest.GetResponse()
                           at System.Net.WebClient.GetWebResponse(WebRequest request)
                           at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
                           --- End of inner exception stack trace ---
                           at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
                           at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
                           at System.Net.WebClient.DownloadString(Uri address)

PS > $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.0-rc
PSEdition                      Core
GitCommitId                    v6.0.0-rc
OS                             Darwin 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~...
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[markekraus]

ServicePointManager is not sup[ported by Invoke-WebRequest and Invoke-RestMethod on PowerShell Core as they now use the HttpClient API which does not honor any settings in ServicePointManager.

On macOS Certificate Authentication and Certificate Validation is currently not supported by any API due to limitations between Apple's Security.Framework and CoreFX (.NET Core). This is being tracked on https://github.com/PowerShell/PowerShell/issues/4650

[markekraus]

I should note that -SkipCertificateCheck does work on macOS and is the only available option.

[liger1978]

Still getting this on CentOS 7 when usingInvoke-RestMethod -Certificate $cert...

Error:

Invoke-RestMethod : The handler does not support client authentication certificates with this combination of libcurl (7.29.0) and its SSL backend ("NSS/3.34")

[liger1978]

Forget it, installing powershell-preview (6.1.0~preview.4-1.rhel.7) fixes this.