Runtime: Unhandled Exception: System.Net.Mail.SmtpException On Linux

[qqbuby]

smtp ehlo

$ telnet mail-proxy.mycompanydomain.com 25
Trying xx.xx.xx.xx...
Connected to mail-proxy.mycompanydomain.com.
Escape character is '^]'.
220 mail.mycompanydomain.com Microsoft ESMTP MAIL Service ready at Fri, 13 Apr 2018 17:52:12 +0800
EHLO
250-mail.mycompanydomain.com Hello [xx.xx.xx.xx]
250-SIZE 209715200
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-AUTH GSSAPI NTLM LOGIN
250-8BITMIME
250-BINARYMIME
250 CHUNKING

Stacktrace

Unhandled Exception: System.Net.Mail.SmtpException: Failure sending mail. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid status code was supplied (Cannot find KDC for realm "mycompanydomain.com").
   at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential)
   at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
   at System.Net.Mail.SmtpNegotiateAuthenticationModule.Authenticate(String challenge, NetworkCredential credential, Object sessionCookie, String spn, ChannelBinding channelBindingToken)
   at System.Net.Mail.SmtpConnection.SetContextAndTryAuthenticate(ISmtpAuthenticationModule module, NetworkCredential credential, ContextAwareResult context)
   at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port)
   at System.Net.Mail.SmtpTransport.GetConnection(String host, Int32 port)
   at System.Net.Mail.SmtpClient.GetConnection()
   at System.Net.Mail.SmtpClient.Send(MailMessage message)
   --- End of inner exception stack trace ---
   at System.Net.Mail.SmtpClient.Send(MailMessage message)

dotnet --info

$ dotnet --info
.NET Command Line Tools (2.1.300-preview1-008174)

Product Information:
 Version:            2.1.300-preview1-008174
 Commit SHA-1 hash:  b8df89a54f

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.300-preview1-008174/

Microsoft .NET Core Shared Framework Host

  Version  : 2.1.0-preview1-26216-03
  Build    : f2c3216183d20416568a4bbf5bb7d153e826f153

Program.cs

```cs
using System;
using System.Net;
using System.Net.Mail;

namespace ConsoleApp1
{
class Program
{
static void Main(string[] args)
{
var host = "mail-proxy.mycompanydomain.com";
var user = "[email protected]";
var password = "xxxxxxxxxxxxx";
var from = "[email protected]";
var to = "[email protected]";

        var client = new SmtpClient(host, 25);
        client.UseDefaultCredentials = true;


        client.Credentials = new NetworkCredential(user, password);
        //  client.Credentials = new NetworkCredential(user, password).GetCredential(host, 25, "NLTM");

        var msg = new MailMessage(from, to);
        msg.Subject = "Hello 世界";
        msg.Body = "Hello World" + DateTime.Now.ToString();

        client.Send(msg);


        Console.WriteLine("Hello World!");
    }
}

}

[karelz]

@qqbuby are you able to reproduce the same problem on .NET Core 2.0? -- In other words, is it a regression since 2.0?
The error looks like configuration problem on the machine - can you please check everything is all right?

cc @wfurt

[qqbuby]

@karelz The same problem has been repreduced on .NET Core 2.0 as below:

Besides, the code works well on Windows platform

$ dotnet run

Unhandled Exception: System.Net.Mail.SmtpException: Failure sending mail. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid status code was supplied (Cannot find KDC for realm "xxxx.com").
   at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential)
   at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
   at System.Net.Mail.SmtpNegotiateAuthenticationModule.Authenticate(String challenge, NetworkCredential credential, Object sessionCookie, String spn, ChannelBinding channelBindingToken)
   at System.Net.Mail.SmtpConnection.SetContextAndTryAuthenticate(ISmtpAuthenticationModule module, NetworkCredential credential, ContextAwareResult context)
   at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port)
   at System.Net.Mail.SmtpTransport.GetConnection(String host, Int32 port)
   at System.Net.Mail.SmtpClient.GetConnection()
   at System.Net.Mail.SmtpClient.Send(MailMessage message)
   --- End of inner exception stack trace ---
   at System.Net.Mail.SmtpClient.Send(MailMessage message)
   at ConsoleApp1.Program.Main(String[] args) in /tmp/hwapp/Program.cs:line 28
$ dotnet --info
.NET Command Line Tools (2.1.104)

Product Information:
 Version:            2.1.104
 Commit SHA-1 hash:  48ec687460

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /opt/dotnet/sdk/2.1.104/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.6
  Build    : 74b1c703813c8910df5b96f304b0f2b78cdf194d

[wfurt]

Do you know if the code is trying to use ntlm or Kerberos? On windows both will work automatically. if on Kerberos path, can you login with kinit from command line? If so,what is your default principle?

[davidsh]

Do you know if the code is trying to use ntlm or Kerberos?

Just a guess here but the stack trace shows that it is trying to do "Negotiate" and not "NTLM" directly.

at System.Net.Mail.SmtpNegotiateAuthenticationModule.Authenticate(String challenge, NetworkCredential credential, Object sessionCookie, String spn, ChannelBinding channelBindingToken)

[karelz]

Good to know this is not a regression - it still seems like machine config problem to me (kinit as @wfurt suggested - it is different between Windows and Linux).

[qqbuby]

@karelz

There are three authentication types, 250-AUTH GSSAPI NTLM LOGIN, how can I specify the SmtpClient use the LOGIN one? e.g. by CredentialCache ?

[qqbuby]

The mail sever supports three authentication types, GSSAPI, NTLM and LOGIN, however both GSSAPI and NTLM cann't work well on Linux.

I write the some code to remove GSSAPI and NTLM, keep LOGIN using reflection as below to repair Smtp temporarily.

private static void RepairSmtpAuthenticationManager()
{
    var smtpAuthenticationManager = typeof(SmtpClient).Assembly.GetType("System.Net.Mail.SmtpAuthenticationManager", false);
    var modules = (System.Collections.IList)smtpAuthenticationManager
        .GetTypeInfo()
        .GetField("s_modules", BindingFlags.Static | BindingFlags.NonPublic)
        .GetValue(smtpAuthenticationManager);
    List<object> removeModules = new List<object>();
    for (int i = 0; i < modules.Count; i++)
    {
        var fullName = modules[i].GetType().FullName;
        if (string.Equals(fullName, "System.Net.Mail.SmtpNegotiateAuthenticationModule", StringComparison.OrdinalIgnoreCase))
        {
            removeModules.Add(modules[i]);
        }
        else if (string.Equals(fullName, "System.Net.Mail.SmtpNtlmAuthenticationModule", StringComparison.OrdinalIgnoreCase))
        {
            removeModules.Add(modules[i]);
        }
        //else if (string.Equals(fullName, "System.Net.Mail.SmtpLoginAuthenticationModule", StringComparison.OrdinalIgnoreCase))
        //{
        //    removeModules.Add(modules[i]);
        //}
    }
    removeModules.ForEach(m => modules.Remove(m));
}

---

using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Mail;
using System.Reflection;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            var host = "smtp.xxxx.com";
            var user = "[email protected]";
            var password = "**********";
            var from = "[email protected]";
            var to = "[email protected]";

            RepairSmtpAuthenticationManager();

            var client = new SmtpClient(host, 25)
            {
                UseDefaultCredentials = false,
                EnableSsl = false
            };

            client.Credentials = new NetworkCredential(user, password);

            var msg = new MailMessage(from, to)
            {
                Subject = "Hello 世界",
                Body = "Hello World, " + DateTime.Now.ToString()
            };

            client.Send(msg);


            Console.WriteLine("Hello World!");
        }

        private static void RepairSmtpAuthenticationManager()
        {
            var smtpAuthenticationManager = typeof(SmtpClient).Assembly.GetType("System.Net.Mail.SmtpAuthenticationManager", false);
            var modules = (System.Collections.IList)smtpAuthenticationManager
                .GetTypeInfo()
                .GetField("s_modules", BindingFlags.Static | BindingFlags.NonPublic)
                .GetValue(smtpAuthenticationManager);
            List<object> removeModules = new List<object>();
            for (int i = 0; i < modules.Count; i++)
            {
                var fullName = modules[i].GetType().FullName;
                if (string.Equals(fullName, "System.Net.Mail.SmtpNegotiateAuthenticationModule", StringComparison.OrdinalIgnoreCase))
                {
                    removeModules.Add(modules[i]);
                }
                else if (string.Equals(fullName, "System.Net.Mail.SmtpNtlmAuthenticationModule", StringComparison.OrdinalIgnoreCase))
                {
                    removeModules.Add(modules[i]);
                }
                //else if (string.Equals(fullName, "System.Net.Mail.SmtpLoginAuthenticationModule", StringComparison.OrdinalIgnoreCase))
                //{
                //    removeModules.Add(modules[i]);
                //}
            }
            removeModules.ForEach(m => modules.Remove(m));
        }
    }
}

$ dotnet run
Hello World!

[davidsh]

There are three authentication types, 250-AUTH GSSAPI NTLM LOGIN, how can I specify the SmtpClient use the LOGIN one? e.g. by CredentialCache ?

When a server (HTTP server or SMTP server) present multiple schemes for authentication, the "strongest" scheme for security is used by the client.

The order generally is:

• GSSAPI (Negotiate / Kerberos)
• NTLM
• LOGIN (this is similar to HTTP BASIC auth)

If you have a single NetworkCredential set into the SmptClient, then GSSAPI would be used. But if you have a CredentialsCache and only have the user/password marked as LOGIN/Basic, then it would use that scheme.

So you should change this code:

```c#
client.Credentials = new NetworkCredential(user, password);

to something that uses a CredentialCache if you want to force LOGIN/Basic to be used.

See: 
https://docs.microsoft.com/en-us/dotnet/api/system.net.credentialcache.add?view=netcore-2.0#System_Net_CredentialCache_Add_System_String_System_Int32_System_String_System_Net_NetworkCredential_

```c#
var cred = new NetworkCredential(user, password);
var cache = new CredentialCache();
cache.Add(host, 25, "Basic", cred);
client.Credentials = cache;

[karelz]

I wonder if we shouldn't be more resilient to exceptions in our code. The exception comes from:
https://github.com/dotnet/corefx/blob/a44c93063d24e29971891bfc2cf8b0aa8771eae9/src/System.Net.Mail/src/System/Net/Mail/SmtpConnection.cs#L275

due to Negotiate setup. We stop enumerating the remaining authentication protocols. Maybe we could treat some/all exceptions here as "SmtpStatusCode.CommandParameterNotImplemented". Thoughts?

[davidsh]

I wonder if we shouldn't be more resilient to exceptions in our code.

I'm not sure what you mean here. Yes, we should be resilient in general to errors. But we can't, for example, downgrade the authentication mechanism because of an error with a stronger mechanism. For example, in HTTP auth scenarios (.NET Framework, .NET Core) with any of our handlers, if Negotiate authentication fails for some reason, we don't try to use another, less secure, mechanism like Digest or Basic with the same or different credentials.

[karelz]

@davidsh what if we detect just the specific setup failure (as mentioned above Cannot find KDC for realm "mycompanydomain.com")?
That basically means the client does not have the stronger mechanism at all.

[davidsh]

@davidsh what if we detect just the specific setup failure

That could be a risky strategy. That means we need to look at errors (and maybe the error strings) and reason out whether it is safe to downgrade authentication schemes or not.

Perhaps "Cannot find KDC" is because the KDC (Kerberos domain controller/ticketing server) is down due to network problems or authentication errors and not because of some setup error. We should fail the authenticaion and not try a weaker scheme. This is similar to why TLS downgrade scenarios are bad where a client trying TLS 1.2 for example gets a TLS error. And then decides to try a different TLS version (lower) to get the connection. Browsers used to do that to try to maximize connection success but then changed to a more secure algorithm to prevent these downgrade scenarios.

We have no existing precedents that I'm aware of where we try other authentication schemes because the first one we used fails.

[karelz]

Fair, in that case we should treat it as machine configuration error. Fixing the configuration should help. Or using explicit authentication protocol as listed above: https://github.com/dotnet/corefx/issues/29087#issuecomment-382109615

@qqbuby please be warned that your "workaround" above https://github.com/dotnet/corefx/issues/29087#issuecomment-381930805 is unsupported and may break at any time - it is hacking on internals of framework classes. I would recommend to remove it ASAP.

[mackcoding]

I'm running into this issue but https://github.com/dotnet/corefx/issues/29087 isn't working for me. No matter what I do CredentialCache is being ignored and it's sending anonymous.

[wfurt]

Any chance the server is publicly accessible? Also could you try to add gss-ntlmssp package to your system. I have not test it with SMTP but it does fix NTLM for HTTP. (this still may not fix it if we go down the kerberos path)

[wfurt]

To comment on earlier note from @davidsh For example PAM has concept of chain of authentication modules. They are used in sequence and you can either authenticate, fail to authenticate or fall-through to next module if given authentication cannot proceed. In this case we fail very early even before trying to use credentials. That can perhaps return distinct error code and be handled more gracefully. This could perhaps be controlled via some property if we feel strongly about it.

In this case the server may for example offer GSSAPI/Kerberos to domain joined machines, but that may never work for clients from outside. If I rememebr well, @rmkerr recently fixed similar issue where we were selecting strongest mechanism even if the authentication could not proceed.
I doubt other mail clients would permanently fail using scheme they cannot use.

[rmkerr]

@mackcoding you might also consider capturing network traffic and examining it to verify that the server is advertising the authentication schemes you support.

[mackcoding]

@wfurt it is a private relay server, unfortunately. I added gss-ntlmssp to my system (for those wondering, apt-get install gss-ntlmssp) and I get a new error message:

dotnet/corefx#33[41m#033[30mfail#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HLDPPGHGA13B", Request id "0HLDPPGHGA13B:00000001": An unhandled exception was thrown by the application.
System.Net.Mail.SmtpException: Failure sending mail. ---> System.FormatException: Invalid length for a Base-64 char array or string.
at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
at System.Convert.FromBase64String(String s)
at System.Net.Mail.SmtpNegotiateAuthenticationModule.GetSecurityLayerOutgoingBlob(String challenge, NTAuthentication clientContext)
at System.Net.Mail.SmtpNegotiateAuthenticationModule.Authenticate(String challenge, NetworkCredential credential, Object sessionCookie, String spn, ChannelBinding channelBindingToken)
at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port)
at System.Net.Mail.SmtpTransport.GetConnection(String host, Int32 port)
at System.Net.Mail.SmtpClient.GetConnection()
at System.Net.Mail.SmtpClient.Send(MailMessage message)

@rmkerr It supports GSSAPI and NTLM. But for whatever reason NTLM refuses to work. With a new error message I might be able to figure this out..

[wfurt]

what is your mail server @mackcoding ? Exchange or something else?
If you share your code, I can give it try in my lab.

[mackcoding]

I can post the code tomorrow. It's just a basic few lines. We use an internal relay server (Microsoft ESMTP). Our mail server is Office 365. It's hard for me to say since I don't manage any of it.. I'll get you some deals so you can try to replicate and hopefully fix. Appreciate the attention to this!

[wfurt]

I did more testing on my Ubuntu 16.04 machine and NTLM is working for me. (with gss-ntlmssp installed)

I did try original code as well code modified as:

c# var cred = new NetworkCredential(user, password, domain); var cache = new CredentialCache(); cache.Add(host, 25, "NTLM", cred); client.Credentials = cache;

My server only offers NTLM (not GSSAPI) so it is not 100% same setup.