Runtime: HttpClientHandler.ServerCertificateCustomValidationCallback blows up on OSX

[niemyjski]

I just updated my Project to RTM and did testing on Windows and everything worked. I then went to test on OSX 10.11.5 (latest stable everything, dotnet, openssl) and ran into issues very quickly..

I'm getting The libcurl library in use (7.43.0) and its SSL backend (\"SecureTransport\") do not support custom handling of certificates. A libcurl built with OpenSSL is required. in stable latest everything on macOS...

"   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)\n   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()\n   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\n   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\n   at Exceptionless.Submission.DefaultSubmissionClient.GetSettings(ExceptionlessConfiguration config, Int32 version, IJsonSerializer serializer) in /Users/blake/Code/Exceptionless.Net/src/Exceptionless/Submission/DefaultSubmissionClient.cs:line 89" [string]

5]  
blake@:~/Code/Exceptionless.Net$ brew info openssl
openssl: stable 1.0.2h (bottled) [keg-only]
SSL/TLS cryptography library

[2:16]  
lake@:~/Code/Exceptionless.Net$ brew install openssl
Warning: openssl-1.0.2h_1 already installed
blake@:~/Code/Exceptionless.Net$ brew link --force openssl
Warning: Already linked: /usr/local/Cellar/openssl/1.0.2h_1
To relink: brew unlink openssl && brew link openssl
blake@:~/Code/Exceptionless.Net$

I narrowed it down to defining a custom validation callback breaks the http client.. I can't even do a GET.

var handler = new HttpClientHandler { UseDefaultCredentials = true };
handler.ServerCertificateCustomValidationCallback = delegate { return true; };
var client = new HttpClient(handler, true);
client.GetAsync(url).GetAwaiter().GetResult();

All the code can be found here: http://github.com/exceptionless/Exceptionless.Net

[niemyjski]

Possibly references:

• https://github.com/dotnet/corefx/issues/7812
• https://github.com/dotnet/wcf/issues/1295

[davidsh]

cc: @stephentoub

[stephentoub]

Yes, to use several of the HttpClient features related to certificates, like ServerCertificateCustomValidationCallback, you need to be using a version of libcurl built against OpenSSL; the version you're using is the one that comes with macOS, built against the SecureTransport backend. We do not have the ability to plug in to that backend, hence we throw an exception to indicate that the callback won't be used.

[niemyjski]

@stephentoub I don't think this acceptable. I couldn't make any http request because of it and I'm using the install instructions on the main website which many many others are going to use (that are on osx). Could we get a check to see if it's supported like other handler features? Any request without validation callback is better than nothing.. I get why you are throwing it.. but most peoples http client code is going to be try catched and they are not going to see this...

[stephentoub]

I'm using the install instructions on the main website

We can look to augment the instructions / release notes with related details.
cc: @leecow

Any request without validation callback is better than nothing

I'm not understanding this part. If you want to avoid the error, you can avoid hooking up the callback. And if you've specified a callback, not invoking it could be a significant security hole, as we'd end up using the default validation of a security certificate rather than the validation the developer has explicitly requested via the callback.

but most peoples http client code is going to be try catched and they are not going to see this

I don't understand this part either. Can you elaborate?

Could we get a check to see if it's supported like other handler features?

You mean a capability property, like IsCustomServerCertificateValidationSupported? That's a reasonable request for the future. Thanks.

cc: @ericeil, @bartonjs

[niemyjski]

Yeah, I get all of your points and agree when it comes to security.

if (handler.SupportsServerCertificateValidationCallback) {}

[joshfree]

@stephentoub @bartonjs feel free to submit a PR to update the release notes / docs to make this more clear.

[atanasa]

How is this supposed to work on OSX? Install some package on OSX? Or there is no way to make it work on OSX?

[niemyjski]

I don't think there is any way for this to work on OSX currently (at least I haven't found a way)

[stephentoub]

I don't think there is any way for this to work on OSX currently (at least I haven't found a way)

I'm sure there are other ways, but I did:

brew install curl --with-openssl
brew link --force curl

and then with that library being used:

export DYLD_LIBRARY_PATH=/usr/local/lib/

I can successfully use these features...

stoubmac:~ stoub$ cd testapp
stoubmac:testapp stoub$ cat Program.cs 
using System;
using System.Net.Http;

namespace ConsoleApplication
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var handler = new HttpClientHandler { CheckCertificateRevocationList = true };
            var http = new HttpClient(handler);
            Console.WriteLine(http.GetAsync("https://www.bing.com").GetAwaiter().GetResult());
        }
    }
}
stoubmac:testapp stoub$ dotnet run
Project testapp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.

Unhandled Exception: System.PlatformNotSupportedException: The libcurl library in use (7.43.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.
   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)
   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()
   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ConsoleApplication.Program.Main(String[] args)
stoubmac:testapp stoub$ export DYLD_LIBRARY_PATH=/usr/local/lib/
stoubmac:testapp stoub$ dotnet run
Project testapp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Cache-Control: max-age=0, private
  Vary: Accept-Encoding
  Server: Microsoft-IIS/8.5
  P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
  Set-Cookie: SRCHD=AF=NOFORM; domain=.bing.com; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
  Set-Cookie: SRCHUID=V=2&GUID=05A7870E43E942F68BAC35BB8A4C52C6; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
  Set-Cookie: SRCHUSR=DOB=20160721; domain=.bing.com; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
  Set-Cookie: _SS=SID=0A99E65098CC6D950303EF09991F6C8C; domain=.bing.com; path=/
  Set-Cookie: _EDGE_S=F=1&SID=0A99E65098CC6D950303EF09991F6C8C; path=/; httponly; domain=bing.com
  Set-Cookie: _EDGE_V=1; path=/; httponly; expires=Sat, 21-Jul-2018 23:49:04 GMT; domain=bing.com
  Set-Cookie: MUID=3441EE6881166AB927F3E73180C56B38; path=/; expires=Sat, 21-Jul-2018 23:49:04 GMT; domain=bing.com
  Set-Cookie: MUIDB=3441EE6881166AB927F3E73180C56B38; path=/; httponly; expires=Sat, 21-Jul-2018 23:49:04 GMT
  X-MSEdge-Ref: Ref A: 8C58C90CF2764DCC8D6E5F4EBDDA771C Ref B: 3F2B603999D10209F1CECA268112300C Ref C: Thu Jul 21 16:49:04 2016 PST
  Date: Thu, 21 Jul 2016 23:49:04 GMT
  Content-Type: text/html; charset=utf-8
}
stoubmac:testapp stoub$

[alanrenouf]

Fantastic, the export DYLD_LIBRARY_PATH=/usr/local/lib/ was what i was missing! Thanks

[jchannon]

What's the fix for CentOS?

[bartonjs]

@jchannon CentOS 6 had a curl-openssl package which would (have) work(ed). The problem is that the default libcurl package on CentOS uses Mozilla NSS as the TLS provider, and we don't have the ability to interpret their certificate pointers.

Normal CentOS curl:

$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3

Normal Ubuntu curl:

$ curl --version
curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3

Note the Ubuntu one in this example is OpenSSL/1.0.1f, whereas the CentOS one uses NSS/3.19.1 Basic ECC.

Since curl doesn't have their own abstraction of certificates we would have to have support for each of the (currently 8) possible providers. Right now OpenSSL is the only one.

Since there's not a CentOS package for that anymore that I can find offhand; that might mean you have to build from source. Or maybe there's just something I'm missing in the CentOS package tree.

[jchannon]

@bartonjs we have it working on CentOS now, what we did was rm -f /usr/bin/curl, rm -f rm /lib64/libcurl*, yum groupinstall "Development @Tools" then downloaded latest curl tarball http://curl.haxx.se/download/curl-7.50.1.tar.gz run ./configure, make, make install

[jchannon]

However trying to debug my app on OSX I followed the above instructions with

brew install curl --with-openssl
brew link --force curl
export DYLD_LIBRARY_PATH=/usr/local/lib/

Now when I run dotnet build I see this:

Unhandled Exception: System.TypeInitializationException: The type initializer for 'Crypto' threw an exception. ---> System.TypeInitializationException: The type initializer for 'CryptoInitializer' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'System.Security.Cryptography.Native': The specified module could not be found.
 (Exception from HRESULT: 0x8007007E)
   at Interop.CryptoInitializer.EnsureOpenSslInitialized()
   at Interop.CryptoInitializer..cctor()
   --- End of inner exception stack trace ---
   at Interop.Crypto..cctor()
   --- End of inner exception stack trace ---
   at Interop.Crypto.GetRandomBytes(Byte* buf, Int32 num)
   at System.IO.Path.GetCryptoRandomBytes(Byte* bytes, Int32 byteCount)
   at System.IO.Path.GetRandomFileName()
   at Microsoft.DotNet.InternalAbstractions.TemporaryDirectory..ctor()
   at Microsoft.Extensions.EnvironmentAbstractions.DirectoryWrapper.CreateTemporaryDirectory()
   at Microsoft.DotNet.Configurer.NuGetPackagesArchiver..ctor()
   at Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse(INuGetCacheSentinel nugetCacheSentinel)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)
[1]    9392 abort      dotnet build

I am less than amused 😠

Also these instructions are the complete opposite of what is shown on www.dot.net

brew update
brew install openssl
ln -s /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib /usr/local/lib/
ln -s /usr/local/opt/openssl/lib/libssl.1.0.0.dylib /usr/local/lib/

[Yantrio]

https://github.com/dotnet/cli/issues/3964 see this issue @jchannon

[jchannon]

Thanks, the fix buried in there was to run sudo install_name_tool -add_rpath /usr/local/opt/openssl/lib /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0/System.Security.Cryptography.Native.dylib

Quite frankly this is shit show and MS need to sort this out.

[jchannon]

Trying to debug a simple http request with osx now results in this error message

"One or more errors occurred. (The libcurl library in use (7.43.0) and its SSL backend (\"SecureTransport\") do not support custom handling of certificates. A libcurl built with OpenSSL is required.)"

The code is:

            var handler = new HttpClientHandler { CookieContainer = new CookieContainer() };
            handler.AllowAutoRedirect = false;
            handler.ServerCertificateCustomValidationCallback = (msg, cert, chain, errors) => true;
            var httpClient = new HttpClient(handler);
            var resp = httpClient.PostAsync(url, new StringContent(JsonConvert.SerializeObject(loginDTO), Encoding.UTF8, "application/json")).Result;

Yet when I run curl --version from terminal I get this

curl 7.50.1 (x86_64-apple-darwin15.6.0) libcurl/7.50.1 OpenSSL/1.0.2h zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

The solution to this when debugging is to set the DYLD_LIBRARY_PATH environment variable. To do this in VSCode you add this to launch.json

 "env": {
          "DYLD_LIBRARY_PATH":"/usr/local/lib/"
        }

[evermeire]

I agree, it would be great if someone could figure out an easy solution. MacOS 10.11.x (El Capitan) makes it harder to change the OpenSSL version that ships with OS. I am hoping that 10.12 (Sierra) has a version of OpenSSL that plays nicer with ASP.NET.

/usr/bin/openssl
OpenSSL 0.9.8zh 14 Jan 2016

on MacOS 10.11 (El Capitan) Haven't had a chance to install 10.12 beta yet.

[evermeire]

Also, I deploy my Asp.Net core web app to an Azure instance and than I see the error message of

The specified CGI application encountered an error and the server terminated the process.

[hlogmans]

With Core 1.1, I run into this issue again on Mac Sierra. Reinstalled with the instructions (OpenSSL 1.0.2j) and the symlinks. The dotnet new command runs fine, but when I use the httpclient:

(The libcurl library in use (7.51.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.) ---> System.PlatformNotSupportedException: The libcurl library in use (7.51.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.

I used the instructions in other reports to use a openssl-build of curl, and the command line reports doing so. But the dotnet project seems to reference the system version?

.NET Command Line Tools (1.0.0-preview2-1-003177)

Product Information:
 Version:            1.0.0-preview2-1-003177
 Commit SHA-1 hash:  a2df9c2576

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.12-x64

libcurl.dylib -> ../Cellar/curl/7.52.1/lib/libcurl.dylib

When using DYLD_LIBRARY_PATH things mess up quickly (methods not found etc), and the rpath commands do not seem to help at all.

As I understood, things changed in core 1.1 to both support native Apple and OpenSSL. But how do I choose between them?

[bartonjs]

@hlogmans The 1.1 release started using Security.framework for hashing, HMAC, and symmetric encryption. Asymmetric cryptography, certificates, and TLS are all still performed by OpenSSL.

otool -L System.Net.Http.Native.dylib should tell you how that library linked against curl (@rpath or a direct path reference), which should aid in tracking down why it isn't loading the build of the library you expect.

[hlogmans]

System.Net.Http.Native.dylib:
@rpath/System.Net.Http.Native.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

/usr/lib/libcurl.4.dylib:
/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57740.30.50)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1348.28.0)
/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

So that seems quite logical that it targets the wrong one. Next step: how to fix it form me, and how to fix it for everyone... (and thanks for your support @bartonjs)

Update:
sudo install_name_tool -change /usr/lib/libcurl.4.dylib /usr/local/opt/curl/lib/libcurl.4.dylib System.Net.Http.Native.dylib

[Priya91]

@karelz The release docs needs to be updated as mentioned in this comment, can you please reroute to the right person.

cc @mairaw

[Priya91]

@niemyjski Apart from the documentation update, can you create another issue with an api proposal for the API you've mentioned in this comment. The API proposal process information is here

[mairaw]

Is it about the installation instructions on the dot.net site @Priya91?

[Priya91]

@mairaw Yes..

[mairaw]

Ok thanks for confirming @Priya91. @leecow @kendrahavens, can one of you take a look at this?

[leecow]

Thanks @mairaw, I'll look into the install instruction updates.

[deepumi]

I am having the same issue with CentOS 7-x64 Azure. Do I have to create a new issue?

[bartonjs]

@deepumi Custom handling is only supported on Linux when libcurl uses openssl as its TLS backend. Feel free to open a CentOS-specific issue, though.

[Priya91]

@deepumi Centos default curl installation comes with NSS as crypto backend. Please recompile curl with openssl and then it should work. You can follow the instructions here

[deepumi]

@Priya91 Thanks! I have to work with our Network team on this.

[tlbdk]

In most cases what we are trying to do with ServerCertificateCustomValidationCallback is to disable server side certificate validation or overwrite the root certificate handling with custom root certificates, both something that is support on libcurl with SecureTransport.

https://github.com/curl/curl/blob/33cfcfd9f0378625d3bddbd2c8ac5aad4b646f26/lib/vtls/darwinssl.c#L1465

Would it be possible to extend the interface of HttpClient to support this more directly fx.

httpHandler.ServerCertificateValidation = false;
httpHandler.ServerCertificateRootCaFile = "/.../ca.pem";

Both options would be super easy to implement on platforms that support ServerCertificateCustomValidationCallback as this is very few lines of code and it would make life a lot easier on platforms where certificate validation is harder to overwrite.

[Priya91]

@tlbdk I was of the impression that custom validation callbacks are not supported on OSX with SecureTransport. cc @bartonjs

[tlbdk]

@priya91 they are not when using the default system libcurl, but disabling server certificate validation and using a custom root ca list is, libcurl exposes API for this.

[bartonjs]

Completely disabling validation is something that we could support, if there were API to indicate that was the caller choice. Similarly, if we were to have an API which allowed the specification of a filename to use as a replacement root we could pass that through. (I certainly wouldn't expect us to write a temporary file on a caller's behalf)

What would be generally more useful is if https://github.com/curl/curl/issues/685 were implemented, so we could hook the handshake-complete callback and provide the same level of support that we have with libcurl+openssl on Linux (or, equivalently "the same level of support we have for directly using SslStream").

While I agree that it's unfortunate that custom handshake validation is not possible with HttpClientHandler on macOS, I'm not sure that the level of specific API that we'd need to plumb the options to libcurl make a whole lot of sense.

[tlbdk]

@bartonjs It would enable me to make my client certificate test work on all platforms without having to install certicates. And even if curl gets general support for this it will take a while before it will be shipped, so any way to do this now would be very much appropriate.

[niemyjski]

Is there a plan to address this in 2.0? This is a pretty big issue and it's been on going for about a year now.

[karelz]

It is currently tracked as documentation bug. Which we should address in 2.0.

I see several different asks on this thread. Some about MacOS, some about Linux.
@niemyjski can you please explicitly describe what it is you would like to see? What are the customer
scenarios? Is it "just" for easier testing? Which OS - Mac vs. Linux vs. both? Are there (potential) workaround? (like libcurl changes)

In general, regarding new APIs in 2.0 - it is kind of late -- see details in https://github.com/dotnet/corefx/issues/17619#issuecomment-296872132. While it is not unimaginable, it would have to have very strong case (high demand, no workarounds, low risk, etc.) - basically a DCR.
Also note that networking v-team has lots of critical bugs to fix for 2.0. We are a bit stretched already in that area. Just finding someone available and capable of doing it (with high quality and expertise) would be a challenge at this moment.

[niemyjski]

It just needs to work, we have this code in our exceptionless nuget package which gets installed into thousands of apps. If you are running on linux or osx and we have this code in place you don't get any warning it just doesn't work (no http requests will be made). We can't relie on install documentation when this goes out to customers who then deploy on customers machines. We just need it to work.

[karelz]

@niemyjski it would help if you could answer all my questions - I am trying to help here, but we (or maybe just I) need summary of all relevant info to be able to make any decisions. The discussion above is a bit confusing to me :(.

Here's my attempt:

1. On Linux - if you libcurl backed with OpenSSL, it works. If you have the libcurl backed with non-OpenSSL provider (e.g. NSS), you will get an exception.
   
   
   
   • Is that correct summary?
   
   
   • Is it ok behavior, or do you believe this is a bad behavior? (If bad, please reiterate why)
   
   
2. On Mac - we do not rely on OpenSSL anymore in 2.0 (per customer feedback), and libcurl does not support the feature on Mac crypto (https://github.com/curl/curl/issues/685), so we are blocked.
   
   
   
   • In theory we could do very expensive hacky workarounds, but we do not believe it is the right investment in 2.0. Beefing up libcurl still seems like the best approach.
   
   
   • Is that correct summary?
   
   
   • Is it ok outcome, or do you think we are wrong?
   
   

[niemyjski]

1. I haven't looked to much into the linux front that is @jchannon
2. I haven't looked into 2.0's implementation but it just never worked. It would be great if libcurl was beefed up but I think the best and immediate approach would be to add a property that we could check like if (handler.SupportsServerCertificateValidationCallback) {}. There are already other supports* properties and this makes sense. It allows us to define it if it's supported otherwise we don't.

[karelz]

@niemyjski that sounds reasonable. Can you please file a new issue to track adding SupportsServerCertificateValidationCallback? We should discuss it separately - I would love to see the use cases, discussion about workarounds and how common the scenario is. Please follow sample in API review process.

Note that even if we add it to .NET Core 2.0, it won't be available in .NET Standard 2.0.

[tlbdk]

@karelz Would be happy to do a pull request if we could extend the interface, I don't think this would take a lot of time to implement, fx for the "httpHandler.ServerCertificateValidation = false;" option, the code is already doing this when ServerCertificateValidationCallback is supported as it disables internal validation in libcurl. Getting it to work on other platforms where ServerCertificateValidationCallback is support is also simple as it just setting a delegate that returns true.

Is a bit more work and might be more controversial for the "httpHandler.ServerCertificateRootCaFile = "/.../ca.pem";" option as you would need to do actual certificate chain validation and also the certificates formats might differ from platform to platform.

So you would accept a pull request adding "httpHandler.ServerCertificateValidation = false;"?

[jchannon]

For CentOS we still do the same as mentioned here - https://github.com/dotnet/corefx/issues/9728#issuecomment-239403404

For OSX we have to do the same as mentioned here - https://github.com/dotnet/corefx/issues/9728#issuecomment-239413686

It's a mess and needs sorting ASAP

[niemyjski]

@tlbdk sounds great, any chance you could open an issue to track this :). I thought we had created one before, but I guess it was just mentioned in this issue.

[karelz]

@tlbdk no guarantee if/when we take it - let's have the discussion in separate issue.

@jchannon "needs sorting ASAP" - what do you propose?

[jchannon]

not having to jump through ridiculous hoops.

if i set the callback to true it should not validate the certs.

the issues seem to be with the underlying libraries however but i have no
idea how you solve that but from a user perspective the api provided doesnt
work

On 25 April 2017 at 22:59, Karel Zikmund [e-mail] wrote:

@tlbdk https://github.com/tlbdk no guarantee if/when we take it - let's
have the discussion in separate issue.

@jchannon https://github.com/jchannon "needs sorting ASAP" - what do
you propose?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/9728#issuecomment-297177693, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGapliAHYPwwxHdYrfzZ06AHZiYsUpFks5rzmzDgaJpZM4JAdiR
.

[karelz]

@jchannon we can solve it by documentation. Or by making sure the exceptions are self-describing. Or by adding the API on the compat list.
Do you have preference?

[daxian-dbw]

I got the same problem when using HttpClientHandler.ServerCertificateCustomValidationCallback with netcoreapp2.0 on osx.10.12-x64:

The handler does not support custom handling of certificates with this combination of 
libcurl (7.49.1) and its SSL backend ("SecureTransport")

I tried the workaround I saw from this thread: sudo install_name_tool -change /usr/lib/libcurl.4.dylib /usr/local/opt/curl/lib/libcurl.4.dylib ./System.Net.Http.Native.dylib
And otool -L does show the rpath of System.Net.Http.Native.dylib points to /usr/local/opt/curl/lib/libcurl.4.dylib

otool -L ./System.Net.Http.Native.dylib 
./System.Net.Http.Native.dylib:
    @rpath/System.Net.Http.Native.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/curl/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

However, then I got a different error complaining the combination of libcurl and OpenSSL/1.0.2k

The handler does not support custom handling of certificates with this combination of 
libcurl (7.54.0) and its SSL backend ("OpenSSL/1.0.2k")

Could someone please help point the correct workaround to make HttpClientHandler.ServerCertificateCustomValidationCallback work on OS X with netcoreapp2.0?

[jchannon]

i'm not sure what you'd add to the docs, there seems varying hacks to get it to work. My colleagues seem to have done various other things to get it to work on their system but we all run the same OS.

[daxian-dbw]

On Mac - we do not rely on OpenSSL anymore in 2.0 (per customer feedback), and libcurl does not support the feature on Mac crypto (curl/curl#685), so we are blocked.

@karelz I'm sorry to reiterate on this. Does this mean HttpClientHandler.ServerCertificateCustomValidationCallbackcurrently is not supported in netcoreapp2.0 on Mac at all?
I have tried the workarounds mentioned in https://github.com/dotnet/corefx/issues/9728#issuecomment-239413686 and https://github.com/dotnet/corefx/issues/9728#issuecomment-297179979, and none of them work for me. Is there a workaround that allows ServerCertificateCustomValidationCallback to work in netcoreapp2.0 on Mac?

[karelz]

@daxian-dbw correct, it is my understanding it does not work on Mac at all and that there is no workaround due to the bug in curl.

[Priya91]

We have another bug tracking the documenation of the supported httpclient features with underlying libcurl+backend ssl install. Closing this as dupe of dotnet/corefx#10146

[e23z]

When I try to use export DYLD_LIBRARY_PATH="/usr/local/lib/" from the solutions above I get the following error:

Failed to load /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/libcoreclr.dylib, error: dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/libcoreclr.dylib, 1): Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /usr/local/lib//libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
Failed to bind to CoreCLR at '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/libcoreclr.dylib'

:(
Don't know what else to do, should I just give up of using HttpClientHandler.ServerCertificateCustomValidationCallback on a macOS Sierra environment?

[jchannon]

Try this http://stackoverflow.com/questions/17643509/conflict-between-dynamic-linking-priority-in-osx/35070568#35070568

[bartonjs]

It might be possible to bring the libcurl+openssl1.0 workaround back for 2.0. https://github.com/dotnet/corefx/issues/19718 tracks the work. (Client authentication still won't work on macOS with HttpClient, because the bridge only reliably goes one way)

[karelz]

If you hit this and need the workaround, please go ASAP and vote for the issue dotnet/corefx#19718 which @bartonjs created. Thanks!

[AnandInstro]

What's the fix for Fedora26?

[jrmacks]

FYI getting this on my mac using dotnet core 2.0.2.

my mac version:
ProductName: Mac OS X ProductVersion: 10.13.1 BuildVersion: 17B48
My original curl version:
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

I updated my curl via homebrew and ran through the sudo install_name_tool change to System.Net.Http.Native.dylib and now my app recognizes the new curl version but still get the error.

The handler does not support custom handling of certificates with this combination of libcurl (7.56.1) and its SSL backend ("OpenSSL/1.0.2m").

As i'm reading through this thread it appears that the issue is with a bug in curl and there is _no_ workaround?

[bartonjs]

@jrmacks In .NET Core 2.0 the certificates library and TLS library used on macOS are from Security.framework, there's no longer support on macOS for custom callbacks via libcurl+openssl. And, unfortunately, libcurl+darwinssl (SecureTransport) doesn't support callbacks.

[jrmacks]

Thanks @bartonjs for the quick reply. Are there any plans to have parity on mac with this feature? Based on your response I'm assuming it's not possible with the current architecture.

[niemyjski]

Any updates on this issue when running on macOS?

[karelz]

@niemyjski it would be best to not continue discussion on closed issues - we do not monitor them in general.
For this particular case, I am not sure what you're asking for -- there have been several spin offs, none is tracked by this issue anymore.

[niemyjski]

They shouldn't be closed if they are still an active issue ;) There are issues when you have this handler registered on macOS...

[stephentoub]

@niemyjski, what is it you're concerned is still active? With the default libcurl on macOS, ServerCertificateCustomValidationCallback is simply not supported with arbitrary delegates. However, the issue you originally opened was using a delegate that always returned true, and that has been fully addressed via the DangerousAcceptAnyServerCertificateValidator member that was added in .NET Core 2.0. This also won't be an issue when using the new managed handler implementation that's on a path to ship in 2.1. If you continue to have concerns, please open a different issue.

[jchannon]

What's the 2.1 change you mention? Any docs etc?

On 3 January 2018 at 13:19, Stephen Toub [e-mail] wrote:

@niemyjski https://github.com/niemyjski, what is it you're concerned is
still active? With the default libcurl on macOS,
ServerCertificateCustomValidationCallback is simply not supported with
arbitrary delegates. However, the issue you originally opened was using a
delegate that always returned true, and that has been fully addressed via
the DangerousAcceptAnyServerCertificateValidator member that was added in
.NET Core 2.0. This also won't be an issue when using the new managed
handler implementation that's on a path to ship in 2.1. If you continue to
have concerns, please open a different issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/9728#issuecomment-355009780, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGapujo8hBdAJPStLWaGhmUQiVtyQSHks5tG35bgaJpZM4JAdiR
.

[stephentoub]

What's the 2.1 change you mention? Any docs etc?

It's not done yet.
https://github.com/dotnet/corefx/tree/master/src/System.Net.Http/src/System/Net/Http/Managed
Search issues for "ManagedHandler".

[MonikaMalik123]

Hi Guys
System - Linux 7.3
Run -Time - dotnet core 2.0.0

I am running dotnet Core 2.0 application on this container (registry.access.redhat.com/dotnet/dotnet-20-rhel7:latest) in linux 7.3, but when login to application and enter password gives me this error - "One or more errors occurred. (The handler does not support custom handling of certificates with this combination of libcurl (7.29.0) and its SSL backend ("NSS/3.28.3").)"
FYI - This application works fine on Windows.

Please suggest me any resolution for this issue? Much Appreciated if anyone can help me with this

Thanks

[ahaid]

I'm having the same issue on Mac and RHEL 7.4

[bartonjs]

@omajid Is there an optional package that people need on RHEL to get libcurl backed by OpenSSL? I thought it was marked as a dependency for the dotnet package(s)?

[evertmulder]

Using 2.1.300-preview1-008174 on MacOSX the issue still occurs

Unhandled Exception: System.PlatformNotSupportedException: The handler does not support custom handling of certificates with this combination of libcurl (7.53.1) and its SSL backend ("OpenSSL/1.0.2n").
   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)
   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()
   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)

@stephentoub https://github.com/dotnet/corefx/tree/master/src/System.Net.Http/src/System/Net/Http/Managed

looks like it os renamed to

https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/

I was hoping in the 2.1 release....

[wfurt]

It is in in 2.1 preview1 @evertmulder. But you need to enable it either via app context or via environmental variable.
export DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=true should do the trick.
From the trace, you still use default curl handler.

[evertmulder]

@wfurt is it? Executing this (or exporting the env variable) still leads to a stack trace including the CurlHandler...

DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=1 dotnet run

Unhandled Exception: System.PlatformNotSupportedException: The handler does not support custom handling of certificates with this combination of libcurl (7.53.1) and its SSL backend ("OpenSSL/1.0.2n").
   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)
   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()
   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__62.MoveNext()
   at cf.Program.<Main>d__0.MoveNext() in /Users/emulder/dev/core/tmp/cf/Program.cs:line 36
   at cf.Program.<Main>(String[] args)

But changing my test program to use DangerousAcceptAnyServerCertificateValidator, could help until something like https://github.com/dotnet/corefx/issues/18972 is fixed. (custom root certificate authorities)

c# var handler = new HttpClientHandler(); handler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator; var _httpClient = new HttpClient(handler);

[wfurt]

Sorry about that @evertmulder. It changed after preview1 cut. Please follow notes from https://blogs.msdn.microsoft.com/dotnet/2018/02/27/announcing-net-core-2-1-preview-1/

[wfurt]

Did you get that working @evertmulder? (just curious)

[evertmulder]

@wfurt No, I will try again when the SocketsHttpHandler is available in a preview release.

[thangchung]

@wfurt I run .NET Core stack on Docker for Windows with Kubernetes (Linux container on Windows 10) and got the following exception. I tried to use ServerCertificateCustomValidationCallback, but no luck. I used .NET Core SDK 2.1.301. Could anyone help to clarify?

Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:443/idp/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:443/idp/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: Address not available ---> System.Net.Sockets.SocketException: Address not available
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.

[stephentoub]

@thangchung, I'm not sure what that error has to do with this issue. The key part of that message is "System.Net.Sockets.SocketException: Address not available", which means it couldn't resolve localhost. Sounds like you may need to update the configuration of your image, and/or use a different address.