Runtime: HttpClient with a client certificate SSL Connection Error

[KyleGobel]

I'm attempting to connect to a kubernetes api within the cluster with C#. I can do this with the Python and Go libraries, but would like to do it with C#. I would imagine what I'm doing will become more common as kubernetes grows in popularity and C# becomes more popular in that world.

I'm getting some somewhat generic errors and I'm sorta stumped on how to debug this further and could use some pointers if anyone is able.

The following curl commands works.

curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes/api/v1/services

Here is my C# attempt at trying to replicate the above

private HttpClient GetClient()
{
    const string certPath = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt";
    const string tokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token";
    const string baseAddress = "https://kubernetes/";
    var handler = new HttpClientHandler
    {
        ClientCertificateOptions = ClientCertificateOption.Manual,
        SslProtocols = SslProtocols.Tls12
    };
    handler.ClientCertificates.Add(
       new X509Certificate2(certPath));

    var token = File.ReadAllText(tokenPath);
    var httpClient = new HttpClient(handler)
    {
        BaseAddress = baseAddress,
        DefaultRequestHeaders =
        {
            {"Authorization", $"Bearer {token}"}
        }
    };
    return httpClient;
}

public void Test()
{
    var client = GetClient();
    var result = client.GetStringAsync("api/v1/services").GetAwaiter().GetResult();
    Console.WriteLine(result);
}

This is running in the microsoft/aspnetcore:1.1 docker image.

The exception that's being thrown isn't all that helpful to me

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: SSL connect error
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<GetContentAsync>d__32`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

I've seen a couple issues about things (possibly?) similar https://github.com/dotnet/corefx/issues/12871, and then https://github.com/dotnet/corefx/issues/12962; But I don't really know enough about SSL or this kubernetes certificate i'm using to know if I'm having the same issue or if I'm just configuring the HttpClient wrong.

The certificate at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt looks something like this (I don't know what kind of certificate this is, but maybe somebody else does)

-----BEGIN CERTIFICATE-----
MIIC2DCCAcCgAwIBAgIRAKKWzHRtGNDE41/XVtJZre0wDQYJKoZIhvcNAQELBQAw
<some more letters>
DqzylX68MAVdg+LF
-----END CERTIFICATE-----

I"ve been stuck on this for a couple days now trying random variations of the above and trying to look at what the Go Client library does to get this to work (as far as I can tell..nothing really special). Hoping someone here can push me in the right direction to get this either working or figured out why it's not working.

Thanks

[davidsh]

The certificate that you're adding to HttpClientHandler.ClientCertificates only contains the public key. The SSL connection can't happen without access to the private key.

Using 'curl', specifying the public key on the command line makes curl look up the private key in the machine configuration.

There are 2 ways that a private key is specified in HttpClient. You can read the entire client certificate from the system which will include both public and private key portions in an X509Certificate2 object. Then add that to the HttpClientHandler.ClientCertificates.

The other way is to only add the public key portion (which is what you did). But then HttpClient is supposed to look up the private key in the system if it is missing from the X509Certificate2 object.

This behavior of looking up the private key portion (if missing from the certificate) works on .NET Framework (Windows). It currently does NOT work on .NET Core (Windows) since that was not implemented. This is a current feature gap of .NET Core vs. .NET Framework.

I suspect that the private key lookup is also NOT implemented in the .NET Core Linux implementation. And that is why you are getting an error.

cc: @stephentoub @bartonjs

[KyleGobel]

Thanks, I really appreciate the quick response time.

I'm wondering if this is something simple I can write a little work around for myself?

I'm thinking something along the lines of:

• Look up the private key in the system myself (not really sure where to look)
• Somehow cobble the private key with the public key into a new file (not really sure how this would need to be formatted, one after the other?)
• Use my new public/private key file with the X509Certificate2 object

Am I vastly over simplifying what I would need to do here? I will attempt to do some more research on my own and figure out what I can do, but any guidance you'll might have would be appreciated.

[bartonjs]

The --cacert path to curl changes the trust rules from "trust any system root CA" to "trust this cert (bundle?) as the only root CA(s)". It doesn't have anything to do with client certificates.

The equivalent would be something like

```C#
private HttpClient GetClient()
{
const string certPath = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt";
const string tokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token";
const string baseAddress = "https://kubernetes/";
var handler = new HttpClientHandler
{
//ClientCertificateOptions = ClientCertificateOption.Manual,
SslProtocols = SslProtocols.Tls12
};
//handler.ClientCertificates.Add(
// new X509Certificate2(certPath));

handler.ServerCertificateCustomValidationCallback = (request, cert, chain, errors) =>
{
    const SslPolicyErrors unforgivableErrors =
        SslPolicyErrors.RemoteCertificateNotAvailable |
        SslPolicyErrors.RemoteCertificateNameMismatch;

    if ((errors & unforgivableErrors) != 0)
    {
        return false;
    }

    X509Certificate2 remoteRoot = chain.ChainElements[chain.ChainElements.Count - 1].Certificate;
    return new X509Certificate2(certPath).RawData.SequenceEqual(remoteRoot.RawData);
};

var token = File.ReadAllText(tokenPath);
var httpClient = new HttpClient(handler)
{
    BaseAddress = baseAddress,
    DefaultRequestHeaders =
    {
        {"Authorization", $"Bearer {token}"}
    }
};
return httpClient;

}
```

Though you could certainly save your pinning root certificate to avoid constantly reading and parsing the file.

[mmisztal1980]

Hi Guys, I was just struggling with the same thing, however I took a different route:

In my deployment.yaml file, I've added a sidecar container, running kubectl proxy:

- name: poc-kubectl-sidecar
        image: lachlanevenson/k8s-kubectl
        command:
        - kubectl
        - "proxy"
        - "--port=8000"
        ports:
        - containerPort: 8000

Next, I've attempted to reach my desired API endpoint using localhost:8000 over regular HTTP and it worked as expected.

GET http://localhost:8000/api/v1/namespaces/default/endpoints/poc-kubernetes-service-svc yielded:

{
    "kind": "Endpoints",
    "apiVersion": "v1",
    "metadata": {
        "name": "poc-kubernetes-service-svc",
        "namespace": "default",
        "selfLink": "/api/v1/namespaces/default/endpoints/poc-kubernetes-service-svc",
        "uid": "a5cd2b64-80ff-11e7-8bba-00155d087f28",
        "resourceVersion": "2317",
        "creationTimestamp": "2017-08-14T14:48:34Z",
        "labels": {
            "accessibility": "external",
            "app": "poc-kubernetes-service",
            "kind": "api"
        }
    },
    "subsets": [
        {
            "addresses": [
                {
                    "ip": "172.17.0.4",
                    "nodeName": "minikube",
                    "targetRef": {
                        "kind": "Pod",
                        "namespace": "default",
                        "name": "poc-kubernetes-service-deployment-135725730-r4zsc",
                        "uid": "a01694a7-810a-11e7-8e30-00155d087f28",
                        "resourceVersion": "2315"
                    }
                },
                {
                    "ip": "172.17.0.5",
                    "nodeName": "minikube",
                    "targetRef": {
                        "kind": "Pod",
                        "namespace": "default",
                        "name": "poc-kubernetes-service-deployment-135725730-z8dqb",
                        "uid": "692353df-810a-11e7-8e30-00155d087f28",
                        "resourceVersion": "2170"
                    }
                },
                {
                    "ip": "172.17.0.6",
                    "nodeName": "minikube",
                    "targetRef": {
                        "kind": "Pod",
                        "namespace": "default",
                        "name": "poc-kubernetes-service-deployment-135725730-gcbjf",
                        "uid": "690d5754-810a-11e7-8e30-00155d087f28",
                        "resourceVersion": "2179"
                    }
                },
                {
                    "ip": "172.17.0.7",
                    "nodeName": "minikube",
                    "targetRef": {
                        "kind": "Pod",
                        "namespace": "default",
                        "name": "poc-kubernetes-service-deployment-135725730-mnht8",
                        "uid": "99373ce4-810a-11e7-8e30-00155d087f28",
                        "resourceVersion": "2222"
                    }
                },
                {
                    "ip": "172.17.0.8",
                    "nodeName": "minikube",
                    "targetRef": {
                        "kind": "Pod",
                        "namespace": "default",
                        "name": "poc-kubernetes-service-deployment-135725730-fvnf4",
                        "uid": "9b416b92-810a-11e7-8e30-00155d087f28",
                        "resourceVersion": "2260"
                    }
                }
            ],
            "ports": [
                {
                    "port": 80,
                    "protocol": "TCP"
                }
            ]
        }
    ]
}

I've consulted the Kubernetes documentation, and in here this appears to be the recommended method of communicating with the K8S api from inside the pod.

[karelz]

Let's track test failures separately from this issue - deleting the comments & updating labels.

[dasjestyr]

Was this implemented in 2.0? We're actually here scratching our heads with the same problem but on microsoft/aspnetcore:2.0 -- the cert is in /etc/ssl/certs and the key is in /etc/ssl/private but it's not being found by the code. It appears to find the cert just fine, but no private key.

[karelz]

Nothing was fixed in 2.0.
Is there a standalone from-scratch minimal repro someone can share?
Is it specific to the official docker images, or can it be reproduced on any Linux box?

[dasjestyr]

We'll put one together and also give it a try on an ubuntu vm probably by tomorrow.

[Ugenx]

As promised: https://github.com/Ugenx/netcore-certtest

Reproduced on Ubuntu 17.10

[karelz]

Great! Looks pretty small, thank you!
@wfurt can you please take a look when you get a chance?

[wfurt]

I can take a look @karelz. I can run the code @Ugenx provided and I can see that the key is not loaded.
I'm not sure if that exactly same problem as originally reported but it is a start.

[wfurt]

The problem is that you are importing key and certificate in two separate steps @Ugenx . With that, it would be really difficult to figure out what key belong to what certificate (if any). That is reason why Apache or example above has explicit pointer to certificate AND key.
The /etc/ssl/private is really just a convenience directory and not real cryptographic storage.

However you can import them to certificate store in single step, and the store will maintain the relation. (same way as if you do import on windows)

``` c#
@@ -7,10 +7,21 @@ namespace netcore.certtest
{
private const string DefaultThumbprint = "BC6B3F7414BE8F5C2632C3BCE199B6DC33092EE5";
private const StoreName _StoreName = StoreName.Root;
- private const StoreLocation _StoreLocation = StoreLocation.LocalMachine;
+ private const StoreLocation _StoreLocation = StoreLocation.CurrentUser;

     public static void Main(string[] args)
     {

• if (args.Length > 0)
• {
• Console.WriteLine("Importing {0}", args[0]);
• var cert = new X509Certificate2(args[0], (string)null);
• Console.WriteLine("cert={0} {1}", cert, cert.HasPrivateKey);
• var store = new X509Store(_StoreName, _StoreLocation);
• store.Open(OpenFlags.ReadWrite);
• store.Add(cert);
  +
• return;
• }
  var certificateThumbprint = DefaultThumbprint;
  ```

now you can run "dotnet run ../server.pfx" and than anytime after you'll get what you want.

furt@Ubuntu:~/git/netcore-certtest/src$ ~/dotnet/dotnet run
Found certificate with thumbprint [BC6B3F7414BE8F5C2632C3BCE199B6DC33092EE5] in the [CurrentUser]:[Root] store. Has private key: [True]

Also note, that CurrentUser must be used as .NET Core will not modify system stores. On Linux, there is no real standard way how to do that. I would need more information to see how that relates back to the problem with kubernetes.

[karelz]

Seems to be answered. Please let us know if it is not sufficient or if there were different related problems @wfurt didn't address. Thanks!