Runtime: NTLM interop problems with MacOS

[davidsh]

.NET Core NTLM support is broken on MacOS. It has probably been broken for a while. But due to lack of test coverage we didn't notice. I did my testing on MacOS X 10.15.2 (Catalina).

The root cause is complex and due to possible interop issues with the NTLM support in the Heimdal library that ships with the Mac. Prior to MacOS 10.7, the Mac used MIT Kerberos open-source libraries. In 10.7, it switched to Heimdal open-source libraries. You can see the source code of the various open-source libraries used in the Mac here.

We have had some reports by customers of problems with Mac .NET Core clients using NTLM. At first, we thought that some NTLM support package was missing on the Mac. On Linux, we require the installation of the 'gss-ntlmssp' package which by default is not present on Linux distros nor on our .NET Core docker images. But the Mac already has NTLM support installed in the GSS-API mechanisms.

We have some simple NTLM tests in our repo:

https://github.com/dotnet/runtime/blob/57df20206691c6123b6e2d392b661cb3541fc24a/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs#L588-L593

And these were running and passing on the Mac. However, the tests only validate that an initial NTLM base64 token is generated and sent to the server. Those tests don't do end-to-end validation of a username/password.

We have other tests for HttpClient that do end-to-end validation:

https://github.com/dotnet/runtime/blob/57df20206691c6123b6e2d392b661cb3541fc24a/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs#L568-L570

But those are not turned on right now. They require environment variables to be enabled to point to a remote NTLM server. After turning on those tests on my local Mac, I discovered that NTLM is broken for Mac .NET Core.

More NTLM (and Kerberos) tests will be enabled for CI as part of follow-up PRs to #463

If I use Safari on the Mac, it works fine against the NTLM test server. But the Safari app on the Mac uses its own NTLM library and not the GSS-API Heimdal library that .NET Core uses.

Looking at several WireShark traces of working and non-working scenarios shows differences in various NTLM protocol fields. Although the Mac is sending NTLM packets to the server, the end result is that the server (Windows in this case) denies the authentication. We'll need to work with the Windows protocol team to help enable advanced diagnostics to see why the Windows server denied the authentication.

I will continue looking into the problem by trying out different NTLM servers to test against (such as Linux Apache) to get more information on working versus non-working scenarios. My current hypothesis is that there is a problem with Heimdal's NTLM v2 support. The NTLM protocol has many differences between NTLMv1 versus NTLMv2.

[scalablecory]

triage: please upvote this if you're seeing this issue.

[scholtz]

Hi @davidsh , I am trying to do windows authentication in linux according to this: https://docs.microsoft.com/cs-cz/aspnet/core/security/authentication/windowsauth?view=aspnetcore-3.1&tabs=visual-studio#kestrel

What I have observed, .net core 3.1 connects to System.Net.NTAuthentication which calls System.Net.Security.NegotiateStreamPal which calls unix Interop.NetSecurityNative.InitSecContext which invokes somehow the gss method in linux

without gss-ntlmssp library it throws error "An unsupported mechanism was requested." because there is no mechanism defined in /etc/gss/mech.d/ folder.

when i install gss-ntlmssp library, it throws error: "No credentials were supplied, or the credentials were unavailable or inaccessible"

i have tried to compile my own gss-ntlmssp, and found out that it does not even trigger any gss_ methods.
when i comment the line in mech.conf (gssntlmssp_v1 1.3.6.1.4.1.311.2.2.10 @libdir@/gssntlmssp/gssntlmssp.so) it shows unsupported mech, when i uncomment it show no credentials supplied without triggering the library.

i tested this on centos7 vm, ubuntu windows subsytem (18.04) and debian docker image (customized mcr.microsoft.com/dotnet/core/sdk:3.1-buster) - all behave the same

please see my full config in here: https://stackoverflow.com/questions/60296237/how-can-i-enable-debug-log-with-microsoft-aspnetcore-authentication-negotiate-pa

also note, that klist shows my spn, and i am able to connect to mssql using integrated security

what am i missing?? does this work for anybody?

[davidsh]

@scholtz The problem you are describing is not related to this particular issue. This issue is only about NTLM problems (client-side) on MacOS. So, it would be best for you to open a new issue to discuss your authentication issues on Linux.

That said, the problem you are describing is a server-side problem with asp.net core. Kestrel server does NOT support NTLM protocol. There is no way to tell it to send back 'Www-Authenticate: NTLM'. It only supports 'Www-Authenticate: Negotiate' protocol. 'Negotiate' protocol is usually Kerberos but can fall back to using NTLM packets (over SPNEGO protocol) in certain cases. NTLM fallback with Negotiate is something that isn't really supported either in Kestrel although I was able to make it work better in .NET 5. See: https://github.com/dotnet/corefx/pull/42522

So, are you really trying to get NTLM working? You mention 'klist shows my spn and mssql works with integrated security'. So, that probably means you want to use Kerberos anyways. The 'gss-ntlmssp' component is not going to help you here since it is only for NTLM and not Kerberos.

You might want to open an issue in aspnetcore repo to get help on getting Kestrel working with Kerberos.

I also recommend you set the following environment variable on Linux to get GSS-API traces, i.e

export KRB5_TRACE=/dev/stdout

@Tratcher

[scholtz]

Hi @davidsh Great work with the dotnet/corefx#42522 .. Is there any chance how i can test it in my environment? How can I compile .net5 source code? Would it take too much effort for you to port it to .net core 3 :) ?

I believe I am missing some very simple configuration thing to make it work. Could you plase take a look at here, and suggest me how to proceed?

The kerberos kinit thing and mssql authentication does not necessary needs to be setup the same as negotiate auth in kestrel. I believe that Kestrel does the auth negotiate, which goes to gssapi, which goes to gssntlmssp lib and somehow it should authenticate to AD. The thing is that the first negotiate request goes well, and i get the response to the user, so it fails at the challange verification somehow because the winbind library does not verify the user token. So the real question is what do you have setup on your linux machine to make it work.

According to your advise i have started the issue in aspnetcore: https://github.com/dotnet/aspnetcore/issues/19397 . I do not want to spam this thread, so if you will be so kind, please reply there. Thanks

[karelz]

@scholtz we do not backport fixes unless there is strong demand for them. You can try .NET 5 daily builds (or public previews coming soon) - see installers and binaries

[scholtz]

Hi @karelz , thanks for that.. I am trying to extend docker image mcr.microsoft.com/dotnet/core/sdk:3.1-buster with .net5, with no success yet.

I was able to install runtime using this:

RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-runtime-latest-x64.deb
RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-runtime-deps-latest-x64.deb
RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-hostfxr-latest-x64.deb
RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-host-latest-x64.deb

RUN dpkg -i dotnet-runtime-latest-x64.deb dotnet-runtime-deps-latest-x64.deb dotnet-hostfxr-latest-x64.deb dotnet-host-latest-x64.deb

but i am not able to find dependencies for the sdk:
for example it complains that runtime is in lower version than required runtime for sdk:

Step 39/42 : RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-x64.deb
 ---> Using cache
 ---> 3921bf94717f
Step 40/42 : RUN dpkg -i  dotnet-sdk-latest-x64.deb
 ---> Running in 844e40507c26
Selecting previously unselected package dotnet-sdk-5.0.
(Reading database ... 29940 files and directories currently installed.)
Preparing to unpack dotnet-sdk-latest-x64.deb ...
Unpacking dotnet-sdk-5.0 (5.0.100-preview.2.20151.5-1) ...
dpkg: dependency problems prevent configuration of dotnet-sdk-5.0:
 dotnet-sdk-5.0 depends on dotnet-runtime-5.0 (>= 5.0.0~preview.2.20129.8); however:
  Version of dotnet-runtime-5.0 on system is 5.0.0~alpha.1.19564.1-1.
 dotnet-sdk-5.0 depends on netstandard-targeting-pack-2.1 (>= 2.1.0); however:
  Package netstandard-targeting-pack-2.1 is not installed.
 dotnet-sdk-5.0 depends on aspnetcore-runtime-5.0 (>= 5.0.0~preview.2.20128.11); however:
  Package aspnetcore-runtime-5.0 is not installed.
 dotnet-sdk-5.0 depends on dotnet-targeting-pack-5.0 (>= 5.0.0~preview.2.20129.8); however:
  Package dotnet-targeting-pack-5.0 is not installed.
 dotnet-sdk-5.0 depends on aspnetcore-targeting-pack-5.0 (>= 5.0.0~preview.2.20128.11); however:
  Package aspnetcore-targeting-pack-5.0 is not installed.
 dotnet-sdk-5.0 depends on dotnet-apphost-pack-5.0 (>= 5.0.0~preview.2.20129.8); however:
  Package dotnet-apphost-pack-5.0 is not installed.

dpkg: error processing package dotnet-sdk-5.0 (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 dotnet-sdk-5.0

Any ideas where i can find

• dotnet-runtime > 5.0.0~preview.2.20129.8 ,
• netstandard-targeting-pack-2.1 (>= 2.1.0);
• aspnetcore-runtime-5.0 (>= 5.0.0~preview.2.20128.11);
• dotnet-targeting-pack-5.0 (>= 5.0.0~preview.2.20129.8);
• aspnetcore-runtime-5.0 (>= 5.0.0~preview.2.20128.11);
• dotnet-targeting-pack-5.0 (>= 5.0.0~preview.2.20129.8);
• aspnetcore-targeting-pack-5.0 (>= 5.0.0~preview.2.20128.11);

? or is there any .net5 sdk docker image available?

in here https://hub.docker.com/_/microsoft-dotnet-core-sdk/ i dont see any..

thanks

[karelz]

@scholtz I would expect to see .NET 5 Docker image. At minimum with Preview builds. @richlander @MichaelSimons do you know more about it? (or someone who does)

[MichaelSimons]

dotnet/core Docker images will be published for all preview releases of .NET 5 (e.g. preview 1). There are no .NET 5 images in the dotnet/core repositories yet because preview 1 hasn't been released. Don't worry though we have you covered 😸, you can always get the last-known-good (LKG) builds for the next release of .NET Core by going to the dotnet/core-nightly repositories.

@scholtz, take a look at the mcr.microsoft.com/dotnet/core-nightly/sdk:5.0-buster image and see if it contains what you are looking for

[scholtz]

@MichaelSimons Yes, exactly. Thanks for that :) I was able to to create project from tamplate, added the negotiate library, build and run the project however at the end it did not solved my issue :(

[jstlns]

It would be nice if significant defects like this would be back ported to the LTS version of the SDK.

[dustinsoftware]

For others running into this issue, I ran into an identical problem during dotnet restore for both macOS and a Docker container build.

GSSAPI operation failed with error - An invalid name was supplied (Configuration file does not specify default realm).

apt-get update && apt-get install -y gss-ntlmssp resolved the issue within the Docker container prior to build, but I could not find an equivalent for macOS. I'm on Catalina 10.15.7, and the 15.0.100-rc.1.20452.101 sdk.