Dotnet-docker: HttpClient causing SSL certificate issues when running .NET 5.0 inside Docker (Linux Container Host)

[tareksharbak]

Trying to call any HTTPS endpoint using .NET 5.0 running in Docker (Linux Container Host) is causing the call to fail with the following exception:
System.Net.Http.HttpRequestException: 'The SSL connection could not be established, see inner exception.'
Inner Exception:
AuthenticationException: The remote certificate is invalid according to the validation procedure.

This is happening when running the code in a Docker container in both Kubernetes (AKS) or locally using Docker for Windows.

Affected .NET Version: .NET 5.0 Preview 3.

Repro steps:

• In Visual Studio 2019 Preview v16.6.0 Preview 5.0, create a new Console Application
• Add the following code:

HttpClient httpClient = new HttpClient();
var response = await httpClient.GetAsync("[Any HTTPS Endpoint");

• Right click on the project and add Docker Support
• Run inside Docker (Linux Containers)

Running the code either locally (without Docker) or by downgrading to .NET 3.1 solves the issue without any code changes.

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[wfurt]

can you run
```c#
using System;
using System.Net.Http;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Threading.Tasks;

namespace certchain
{
class Program
{
static async Task Main(string[] args)
{
X509Store store1 = new X509Store (StoreName.Root, StoreLocation.LocalMachine);
store1.Open(OpenFlags.ReadOnly);
foreach (X509Certificate cert in store1.Certificates)
{
Console.WriteLine(cert.Subject);
}

        HttpClientHandler handler = new HttpClientHandler();

        using (HttpClient httpClient = new HttpClient(handler))
        {
            handler.ServerCertificateCustomValidationCallback = (request, cert, chain, errors) =>
            {
                Console.WriteLine($"SslPolicyErrors: {errors}");

                if (chain == null)
                {
                    Console.WriteLine("No chain...");
                }
                else
                {
                    foreach (X509ChainElement element in chain.ChainElements)
                    {
                        Console.WriteLine();
                        Console.WriteLine(element.Certificate.Subject);
                        Console.WriteLine(element.ChainElementStatus.Length);
                        foreach (X509ChainStatus status in element.ChainElementStatus)
                        {
                            Console.WriteLine($"Status:  {status.Status}: {status.StatusInformation}");
                        }
                    }
                }

                return true;
            };
            Console.WriteLine("----------------------");

            using var response = await httpClient.GetAsync("https://microsoft.com/");
            Console.WriteLine(response);

        }
    }
}

}
```

you should see list of trusted CA available in the container as well as any validation errors.

[davidfowl]

@DamianEdwards hit this too

[tareksharbak]

@wfurt below is the output of the code running under Docker in Linux Containers.

---

SslPolicyErrors: RemoteCertificateChainErrors
CN=microsoft.com
0
CN=Microsoft IT TLS CA 2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
1
Status:  PartialChain: unable to get local issuer certificate
CN=www.microsoft.com, OU=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=WA, C=US
CN=Microsoft IT TLS CA 5, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
    Cache-Control: no-store, no-transform, no-cache
    Pragma: no-cache
    X-Activity-Id: 578e14ae-928d-4251-b133-4f7352b1d749
    MS-CV: rKvTMmux406MH7hn.0
    X-AppVersion: 1.0.7411.39016
    X-Az: {did:92e7dc58ca2143cfb2c818b047cc5cd1, rid: OneDeployContainer, sn: marketingsites-prod-odnortheurope, dt: 2018-05-03T20:14:23.4188992Z, bt: 2020-04-17T05:40:32.0000000Z}
    ms-operation-id: 8c06c59fd111de4c8a5622d5b4079f0d
    P3P: CP="CAO CONi OTR OUR DEM ONL"
    X-UA-Compatible: IE=Edge;chrome=1
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    Access-Control-Allow-Methods: HEAD,GET,POST,PATCH,PUT,OPTIONS
    X-XSS-Protection: 1; mode=block
    Date: Wed, 06 May 2020 07:29:59 GMT
    Transfer-Encoding: chunked
    Connection: keep-alive
    Connection: Transfer-Encoding
    TLS_version: tls1.2
    Strict-Transport-Security: max-age=31536000
    Set-Cookie: akacd_OneRF=1596526199~rv=7~id=4f2ff554b0e51cf3026ae653c24e4633; path=/; Expires=Tue, 04 Aug 2020 07:29:59 GMT; Secure; SameSite=None
    X-RTag: RT
    Content-Type: text/html; charset=utf-8
    Expires: -1
}

[wfurt]

So it seems like your StoreName.Root is empty. When I run this on my container, it starts with

O=Government Root Certification Authority, C=TW
CN=Certum Trusted Network CA, OU=Certum Certification Authority, O=Unizeto Technologies S.A., C=PL
CN=Certum Trusted Network CA 2, OU=Certum Certification Authority, O=Unizeto Technologies S.A., C=PL
CN=SwissSign Gold CA - G2, O=SwissSign AG, C=CH
OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US
CN=USERTrust ECC Certification Authority, O=The USERTRUST Network, L=Jersey City, S=New Jersey, C=US
CN=GeoTrust Universal CA, O=GeoTrust Inc., C=US
CN=OpenTrust Root CA G2, O=OpenTrust, C=FR
CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
...

What openssl s_client -connect www.microsoft.com:443 gives you? What bases OS do you have in the container?

You can run 'openssl version -a' and check content and permissions on OPENSSLDIR.
You can also try sudo update-ca-certificates.

[DamianEdwards]

@tareksharbak @wfurt Can you both verify exactly which container images you're using please? I only hit this after publishing the app to a runtime image that was deployed to AKS. Locally on Docker Desktop for Windows, when publishing the app to an SDK image it worked OK. In my case it was failing when StackExchange.Redis attempted to connect to my Azure Redis Cache instance. Other than that difference, on AKS the Linux kernel version was lower than on my local machine using Docker Desktop.

[wfurt]

I used mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-crossdeps e.g. Ubuntu18.04 with prerequisites for .NET and tools to build it. I'm running Docker on 18.04 VM running inside Parallels on my MacBook.

Could you run the snippet above @DamianEdwards to see if you have root certificates?
Also if you have repro, maybe we can debug it further. It really should not depend on Kernel version as that is irrelevant to SSL processing.

[davidfowl]

Could you run the snippet above @DamianEdwards to see if you have root certificates?
Also if you have repro, maybe we can debug it further. It really should not depend on Kernel version as that is irrelevant to SSL processing.

Can we make sure that's the image the customer used? An exact repro would be helpful so everyone can reproduce using the same codez

[tareksharbak]

I'm using the default (Visual Studio auto generated) Dockerfile, which has the following images:

FROM mcr.microsoft.com/dotnet/core/runtime:5.0-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:5.0-buster AS build
WORKDIR /src
...

I'm using the same images for my AKS deployment where I'm also having the same issue.

Downgrading to .NET 3.1 and using the following images instead:

FROM mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
...

then running it under the same exact Docker environment solves the problem and gives this output (I truncated the list of certificates):

CN=OISTE WISeKey Global Root GC CA, OU=OISTE Foundation Endorsed, O=WISeKey, C=CH
CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R4
CN=EC-ACC, OU=Jerarquia Entitats de Certificacio Catalanes, OU=Vegeu https://www.catcert.net/verarrel (c)03, OU=Serveis Publics de Certificacio, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), C=ES
CN=thawte Primary Root CA - G3, OU="(c) 2008 thawte, Inc. - For authorized use only", OU=Certification Services Division, O="thawte, Inc.", C=US
CN=SSL.com Root Certification Authority ECC, O=SSL Corporation, L=Houston, S=Texas, C=US
CN=Amazon Root CA 2, O=Amazon, C=US
CN=Chambers of Commerce Root - 2008, O=AC Camerfirma S.A., SERIALNUMBER=A82743287, L=Madrid (see current address at www.camerfirma.com/address), C=EU
CN=Cybertrust Global Root, O="Cybertrust, Inc"
CN=GeoTrust Primary Certification Authority - G3, OU=(c) 2008 GeoTrust Inc. - For authorized use only, O=GeoTrust Inc., C=US
...
----------------------
SslPolicyErrors: None

CN=microsoft.com
0
CN=Microsoft IT TLS CA 2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
CN=www.microsoft.com, OU=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=WA, C=US
CN=Microsoft IT TLS CA 5, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
  Cache-Control: no-store, no-transform, no-cache
  Pragma: no-cache
  X-Activity-Id: b81a096d-0e2a-4aac-b8f6-e445ffb42964
  MS-CV: jvG3kEwLxkOabK0x.0
  X-AppVersion: 1.0.7411.39016
  X-Az: {did:92e7dc58ca2143cfb2c818b047cc5cd1, rid: OneDeployContainer, sn: marketingsites-prod-odnortheurope, dt: 2018-05-03T20:14:23.4188992Z, bt: 2020-04-17T05:40:32.0000000Z}
  ms-operation-id: 4583d3d0b96eff438c31743f5e14059a
  P3P: CP="CAO CONi OTR OUR DEM ONL"
  X-UA-Compatible: IE=Edge;chrome=1
  X-Content-Type-Options: nosniff
  X-Frame-Options: SAMEORIGIN
  Access-Control-Allow-Methods: HEAD,GET,POST,PATCH,PUT,OPTIONS
  X-XSS-Protection: 1; mode=block
  Date: Thu, 07 May 2020 20:44:31 GMT
  Transfer-Encoding: chunked
  Connection: keep-alive
  Connection: Transfer-Encoding
  TLS_version: tls1.2
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: akacd_OneRF=1596660271~rv=4~id=fa22eb7d55f68fdca70c306befae3da1; path=/; Expires=Wed, 05 Aug 2020 20:44:31 GMT; Secure; SameSite=None
  X-RTag: RT
  Content-Type: text/html; charset=utf-8
  Expires: -1
}

---

Some information about my setup:

I'm running this in Docker Desktop in Windows 10 configured to run Linux containers.
If I run uname -a, I get the following:

# uname -a
Linux 9439be7bc506 4.19.76-linuxkit dotnet/runtime#1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 GNU/Linux
# more /etc/issue
Debian GNU/Linux 10 \n \l

Please let me know if this doesn't answer your questions exactly.

[davidfowl]

That sounds to be like we broke something in 5.0.
Either in the runtime or in the container base image. cc @richlander @MichaelSimons @bartonjs

[MichaelSimons]

Does adding the ca-certificates package to your image change anything?

[davidfowl]

@MichaelSimons did something change from 3.1 here?

[MichaelSimons]

curl and ca-certificates were dropped from the runtime images in 5.0.

[wfurt]

ca-certificates is what brings root CAs. I can do more testing since the images are clear now.

https://askubuntu.com/questions/857476/what-is-the-use-purpose-of-the-ca-certificates-package

[tareksharbak]

Does adding the ca-certificates package to your image change anything?

Adding ca-certificates package solves the issue, yes!

Will that be the intended behavior for .NET 5.0 going forward or will this behavior change?

Thanks everyone for your help here!

[davidfowl]

curl and ca-certificates were dropped from the runtime images in 5.0

Why? To shrink the image? This breaks networking as a whole. Surely this was a mistake....

[karelz]

Dropping curl was intentional as we do not depend on that. ca-certificates are likely a bit surprising / mistake.

[DamianEdwards]

It's somewhat concerning to me that a change of this magnitude made it into our runtime images and shipped in a preview (multiple previews?) without being detected. It literally breaks HTTPS communication. Seems we have a verification gap WRT to container scenarios?

[wfurt]

runtime repo seems to be suboptimal place for end2end verifications. Perhaps we should move the discussion elsewhere? https://github.com/dotnet/dotnet-docker/? It seems like there we only verify that images build as far as I can tell.

[MichaelSimons]

curl was removed intentionally from the runtime images as it is not a product requirement and has had some critical vulnerabilities which is undesirable. Dropping ca-certificates was an oversight and is something that should be covered in the container tests.

[davidsh]

Dropping ca-certificates was an oversight and is something that should be covered in the container tests.

Is there going to be a PR for dotnet/dotnet-docker to fix the images to add back ca-certificates ?

[MichaelSimons]

Is there going to be a PR for dotnet/dotnet-docker to fix the images to add back ca-certificates ?

Yes

[mthalman]

This fix is in the 5.0 Preview 4 images.

[mburumaxwell]

I am experiencing this same issue running net core 3.1. I have noticed that if the image is based on 3.1.3-buster-slim, the error does not show but all -buster-slim images afterward have this error. It surprisingly also not present when running with 3.1-alpine or 3.1-alpine. The particular certificate missing is the GeoTrust Global CA one. The endpoint being accessed is https://api.push.apple.com

[mthalman]

@mburumaxwell - This is due to the version of ca-certificates that is available in the Buster package feed. See https://github.com/dotnet/dotnet-docker/issues/2145#issuecomment-669217855. In the case of Alpine, it is still using an earlier version of ca-certificates.