Runtime: HttpClient with mutual TLS on macOS High Sierra - does not support client authentication (Libcurl/LibreSSL problems)

[asjafjell]

I have the following error in my netstandard2 class library:

The handler does not support client authentication certificates with this combination of libcurl (7.54.0) and its SSL backend ("LibreSSL/2.0.20")

I installed the newest version of Curl with OpenSSL through Homebrew and:

which curl gives /usr/local/bin/, and
curl --version gives curl 7.57.0 (x86_64-apple-darwin17.2.0) libcurl/7.57.0 OpenSSL/1.0.2n zlib/1.2.11.

This does not change the version of Curl being run through my class library.

I have now been reading and testing the possible solutions in dotnet/runtime#17723, dotnet/runtime#23640 and ended up with dotnet/runtime#21679 stating that custom certificate handling will be fixed in netcore 2.1. Do I have to wait till this fix propagates into netstandard?

Is my problem just related to me not being able to change the version of libcurl being executed, or do I have a dead end?

Pardon if the solution is obvious, I have tried for almost a day with outdated and more recent answers of the related posts.

[davidsh]

cc: @wfurt @stephentoub @karelz

[wfurt]

I can take a look @asjafjell If you have more specific steps or simple test app, that would be great.
You may need to export DYLD_LIBRARY_PATH=/usr/local/lib to pick up updated libcurl version.

[asjafjell]

@wfurt I have exported DYLD_LIBRARY_PATH, but it makes no difference.

I will try to make a simplification of the code as soon as possible. In the meantime, you can test the original code. It is a public repo digipost/signature-api-client-dotnet (branch: net-standard-2-build), and a test that uses HttpClient is this one

Note that a certificate path and password has to be added to a secrets file at _~/.microsoft/usersecrets/User-Secret-ID/secrets.json_:

{
  "Certificate:Path:Absolute": "/Path/To/Certificate.p12/",
  "Certificate:Password": "ThePassword"
}

Thanks in advance.

[gbrhaz]

I also have this problem.

I exported DYLD_LIBRARY_PATH according to https://spin.atomicobject.com/2017/09/28/net-core-osx-libcurl-openssl/ because I was getting the following:

Failed to load ?, error: dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.dylib, 1): Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /usr/local/lib//libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
Failed to bind to CoreCLR at '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.dylib'

So even using OpenSSL with curl I get:

System.PlatformNotSupportedException: The handler does not support client authentication certificates with this combination of libcurl (7.57.0) and its SSL backend ("OpenSSL/1.0.2n").

I was under the impression .NET Core 2.0 was going to fix having to use curl with OpenSSL in favour of SecureTransport?

Method

• Getting X509Certificate2 certificate from store (StoreLocation.CurrentUser, StoreName.My)
• Add certificate to HttpClient; HttpClientHandler.ClientCertificateOptions = ClientCertificateOption.Manual is the only one.

Edit

If I don't run any updates to curl it does actually look like it finds the certificate in the Keychain; however, I still get the same kind of error:

System.PlatformNotSupportedException: The handler does not support client authentication certificates with this combination of libcurl (7.54.0) and its SSL backend ("SecureTransport").

[asjafjell]

I have now replaced the default OpenSSL implementation of macOS as described in this Medium article, and I now get the error message:

The handler does not support client authentication certificates with this combination 
of libcurl (7.57.0) and its SSL backend ("OpenSSL/1.0.2n")

I don't know if this makes me any closer to any solution, but this is now the newest version of Curl being run, and it is the version I installed throug Brew with the with-openssl flag.

[wfurt]

simple repro would be good @asjafjell The hit with DYLD_LIBRARY_PATH as meant to help with loading proper curl. However that will prefer everything from that directory over over system default and it may cause the libJPEG.dylib loading error. To avoid that you can try something like = /usr/local/Cellar/curl/7.57.0/lib @gbrhaz

As notes from dotnet/runtime#21679 state, there may be additional work needed to get that working on OSX.
Also note, that you should use API to import key pair instead of manipulating store directly - see notes in dotnet/runtime#23074

[gbrhaz]

@wfurt Yes, I believe I'm doing everything that post suggests.

This is essentially what's going on:

var thumbprint = "...";
var certificateStore = new X509Store(StoreName.My, StoreLocation.CurrentUser);
certificateStore.Open(OpenFlags.ReadOnly);
var cert = certificateStore.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, false)[0];
httpClientHandler.ClientCertificates.Add(cert);

I can check the certificate and it has a private key, is the correct one, all that kind of stuff, but still get the previous error.

Just to clarify: the .pfx was imported into my login keychain. I looked at the documentation here:

https://github.com/dotnet/corefx/blob/master/Documentation/architecture/cross-platform-cryptography.md

Which says "On macOS the CurrentUser\My store is the user's default keychain (login.keychain, by default). The LocalMachine\My store is System.keychain". I have tried importing the .pfx into both login and system keychains and still no luck. Still get:

System.PlatformNotSupportedException: The handler does not support client authentication certificates with this combination of libcurl (7.54.0) and its SSL backend ("SecureTransport").
   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)
   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()
   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)

[wfurt]

Can you wrap that to runable app and point me at right server @gbrhaz? It would save my troubles trying to set up a repro. From the exception it seems like you properly overcome all the other hurtles and there is some platform to work to be done.

[h3smith]

If you guys are just doing dev locally and trying to build out a stack on macOS - our work around was simply to build a nginx Docker to act as a reverse proxy. The nginx container would do the client auth for us and the .NET Core solution just connects to the container.

This is not an optimal solution, but avoided us having to do dev on Windows (We are deploying in Ubuntu with Docker, so in the end we can use client authentication when we get out of local machine development).

Here is the nginx config if anyone is interested. simply add the cert and key (remove password) and voila done.

worker_processes 1;
events { worker_connections 1024; }
http {
    sendfile on;
    server {
        listen 9191;
        error_page 404 =200 /;
        location / {
            # Force the proxy
            proxy_pass https://endpointurl.com;

            proxy_ssl_certificate         /usr/local/custom_certs/api.endpointurl.com.cert.pem;
            proxy_ssl_certificate_key     /usr/local/custom_certs/api.endpointurl.com.key;

            # Pass the IP address            
            proxy_set_header  X-Real-IP  $remote_addr;
            proxy_set_header Host $host:$server_port;
            proxy_set_header X-Host $host:$server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        }
    }
}

[gbrhaz]

@wfurt Unfortunately I don't think it'd be possible to send you a precise project for this. Limitations on my side. In the meantime, I have done some digging around and found this.

Are you able to shed any light on this? It looks as if _any_ certificate that is added as a client certificate is denied and a platform not supported exception is thrown? I thought .net core 2.0 allowed X509 from Keychain so I'm a bit confused by this - or maybe it's only supported for server certificates, not client?

[karelz]

As mentioned above - this should be addressed in .NET Core 2.1 which by default uses SocketsHttpHandler.
2.1 is in RC with go-live license.

Closing

[kaabr]

@karelz, unfortunately the same error occurs while using version 2.1.300-rc1-008673 on CentOS 7 (The handler does not support client authentication certificates with this combination of libcurl (7.29.0) and its SSL backend ("NSS/3.34")).

[karelz]

That is highly suspicious. Please check if you're really running against 2.1. Unless you set explicit opt-in (env.var / AppContext), you won't get libcurl into your process.

[Sannj]

I am running version 2.1.301 and I get
One or more errors occurred. (The handler does not support client authentication certificates with this combination of libcurl (7.54.0) and its SSL backend ("SecureTransport").)
when I am trying to run it on MacOS. The code that I have is:

            var handler = new HttpClientHandler();
            handler.ClientCertificateOptions = ClientCertificateOption.Manual;
            handler.SslProtocols = SslProtocols.Tls12;
            handler.ClientCertificates.Add(new X509Certificate2("/Users/sanjanabadam/Documents/dotnetcore/api/abc.crt"));
            var newClient = new HttpClient(handler);

            using (var response = newClient.PostAsync(url, content).Result)
            {
                if (response.StatusCode != HttpStatusCode.OK)
                {
                    return false;
                }
                using (var result = response.EnsureSuccessStatusCode())
                {
                    return true;
                }
            } 

Is this the right place to ask?

[wfurt]

do you actually target netcore2.1 in your project?

[Sannj]

@wfurt Checked the csproj files and found that it was using <TargetFramework>netcoreapp2.0</TargetFramework> instead of <TargetFramework>netcoreapp2.1</TargetFramework> and changing it fixed it.. thank you for pointing me in the right direction :)

[wfurt]

I'm glad it worked for you. this is not always intuitive.

[asjafjell]

I am currently targeting netstandard2.0. Can I make use of the SocketsHttpHandlerthen?