Embedio: Don't depend on Windows' SSL implementation

[ice6]

Is your feature request related to a problem? Please describe.
I'm always frustrated when I want to expose https, netsh is really a mess. I embed a golang reverse proxy server to resolve this problem in production.

Describe the solution you'd like
Is there a better way to do this in CSharp?

[rdeago]

Unfortunately there seems to be no way around it. Not unless we stop using Microsoft's and Mono's HttpListeners. Maybe not even then.

Even µHttpSharp, that handles the HTTP protocol on its own, relies on SslStream to handle encryption over TCP. Not a problem for them, since apparently it is a Windows-only library.

.NET Core has a fully managed implementation of HttpListener and related classes. It normally gets compiled only for other operating systems, but there's a build option to compile it for Windows too, so _maybe_ we could use that instead (or in addition to) current HttpListener options.

From a quick look, the .NET Core managed implementation seems to use OpenSSL. This brings up interesting questions about dependencies and deployment. I think a serious discussion about this is better left for after the release of EmbedIO v3.

[geoperez]

We can check also how Kestrel is handling this.

[rdeago]

You're right @geoperez, I forgot to mention Kestrel!

But alas, it uses SslStream too.

[geoperez]

But how is Kestrel binding to the address:port. I only see how it loads the certificate.

BTW this is an interesting way to load a default cert:

https://github.com/aspnet/AspNetCore/blob/e4fbd598b56c5582335b427dedfb68495d1b8489/src/Servers/Kestrel/Core/src/KestrelServerOptions.cs#L145

[rdeago]

The topic of address / port binding deserves a conversation of its own, but in short Kestrel does not use HttpListener; instead, it is meant to replace HttpListener entirely.

Besides, no matter how we load a certificate, SslStream under Windows uses SSPI. Therefore, Kestrel applications need NETSH just like us mere mortals. 😄

Maybe, and that's a _big_ maybe, we could try to build the Unix implementation of SslStream, which uses OpenSSL, and see if it can work under other operating systems. That's apt to be both fun and painful IMHO.

[rdeago]

For future reference: a managed wrapper for OpenSSL exists. It requires the unmanaged OpenSSL libraries, of course, and it seems to be for .NET 2.0, but it _may_ be a starting point.

[rdeago]

Mono also has managed TLS support using Google's BoringSSL. The latter is, in Google's own words, an OpenSSL fork "not intended for general use". I guess there has to be a reason why Mono uses it, other than thinking different without being Apple. Anyway, one more bit of food for thought, and one more pile of third-party code to look into as soon as we have both the time and courage.

[Genteure]

Besides, no matter how we load a certificate, SslStream under Windows uses SSPI. Therefore, Kestrel applications need NETSH just like us mere mortals. 😄

I don't think so? Isn't SslStream itself is sufficient to provide SSL/TLS encryption, at any platform, with the possibility to do SNI and client certificate?

The GitHub search link shows only three result (at least as of today, GitHub only search on master branch), and they have nothing to do with Kestrel and SSL.

---

µHttpSharp

is a .NET Framework only project.
HTTPS implemented using SslStream.

System.Net.HttpListener

Both .NET Framework and .NET Core use httpapi.dll/http.h on Windows.
HTTPS implemented by httpapi.dll, certificate is configured by netsh http.

On .NET Core it use a managed implementation, HTTPS implemented by SslStream. It doesn't support SNI despite SslStream can do it.

Reference:
.NET Framework (From ReferenceSource)

.NET Core Windows
.NET Core Other and call to SslStream

Kestrel

Kestrel is using SslStream, and it support SNI.

netsh

Netsh commands for HTTP writes:

You can use the Netsh commands for HTTP context to query and configure HTTP.sys settings and parameters.

I don't think it have anything to do with SSPI / SslStream on Windows.

SslStream

It use OpenSSL on Unix, AppleCrypto on OSX, SSPI on Windows.
But that's .NET internal implementation, we don't need to care about that as long as it provide a uniform behavior.

---

EmbedIO currently do support HTTPS on systems other than Windows if using HttpListenerMode.EmbedIO, since it use SslStream.

// v2

new WebServer(new string[]{"https://example.com/"}, RoutingStrategy.Regex, new Unosquare.Net.HttpListener(new System.Security.Cryptography.X509Certificates.X509Certificate2("cert.pfx")))

I suggest we just use SslStream, it has all the feature we need, it's in .NET Standard, it doesn't need any native dependency (at least from our point of view).
As for System.Net.HttpListener, it can stay with netsh. Users of EmbedIO can decide whether they want use EmbedIO's internal server to have HTTPS anywhere, or use System.Net.HttpListener and have to deal with netsh commands. (Or we remove support for System.Net.HttpListener altogether, though I'm not sure if anyone have any reason must to use it.)

[Genteure]

During test I also find out currently if SSL handshake failed for some reason, a common reason is client doesn't trust server's certificate, entire WebServer crashes.

proof of concept code:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
openssl pkcs12 -inkey key.pem -in cert.pem -export -out cert.pfx

// Program.cs
using System;
using System.Net;
using System.Threading.Tasks;
using Unosquare.Labs.EmbedIO;
using Unosquare.Labs.EmbedIO.Constants;
using Unosquare.Labs.EmbedIO.Modules;

namespace sslstreamtest
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var hl = new Unosquare.Net.HttpListener(new System.Security.Cryptography.X509Certificates.X509Certificate2("cert.pfx"));
            var server = new WebServer(new string[]{"https://*:8899/"}, RoutingStrategy.Regex, hl);
            await server.RunAsync();
        }
    }
}



md5-516ed815c88d5dadd38de990d46535ec



curl https://localhost:8899/test



md5-aedd3382bbd0bf70b0b4e9ca162cbe5b



Unhandled Exception: System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__48_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Unosquare.Net.HttpConnection..ctor(Socket sock, EndPointListener epl, X509Certificate cert)
   at Unosquare.Net.EndPointListener.ProcessAccept(SocketAsyncEventArgs args)
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
   at System.Net.Sockets.SocketAsyncEventArgs.ExecutionCallback(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Byte[] socketAddress, Int32 socketAddressSize, SocketError socketError)
   at System.Net.Sockets.SocketAsyncContext.AcceptOperation.InvokeCallback(Boolean allowPooling)
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op)
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.<>c.<.cctor>b__18_0(Object op)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()



md5-b73a8cb9edcba819cf55fadac75a4698



curl https://localhost:8899/test -kv
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 8899 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8899 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=Test EmbedIO Cert
*  start date: Jun 27 19:11:02 2019 GMT
*  expire date: Jun 26 19:11:02 2020 GMT
*  issuer: CN=Test EmbedIO Cert
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /test HTTP/1.1
> Host: localhost:8899
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Type: text/html; charset=utf-8
< Server: embedio/2.0
< Date: Thu, 27 Jun 2019 19:26:23 GMT
< Content-Length: 63
< Keep-Alive: timeout=15,max=100
< 
* Connection #0 to host localhost left intact
<html><head></head><body><h1>404 - Not Found</h1></body></html>

[rdeago]

@Genteure the problem with SslStream is that it uses SSPI!

SSPI (which http.sys and httpapi.dll use under the hood, or implement themselves, I don't know and it's not relevant here) is why we have to use netsh, which is a major pain in the posterior (admin rights required, different command line according to Windows language...)

The idea was to try and recompile .NET COre's or Mono's TLS support for Linux to make it work on other systems (primarily Windows). Since OpenSSL has Windows binaries available (not sure about BoringSSL but I'd say yes, since it's used by Chrome) it _should_ be possible to support secure connections without relying on SSPI.

[Genteure]

Can you write a example code to elaborate why SSPI would need to use netsh?
I can't find any result on Google with SSPI and netsh together that don't involve "httpapi" "httpsys" "Windows Web Services".

Here's my POC, it runs on .NET Core Windows, without netsh.
Tested with openssl as client: openssl s_client -connect 127.0.0.1:8877

static async Task Main(string[] args)
{
    var serverCert = new X509Certificate2("cert.pfx");

    var sock = new Socket(SocketType.Stream, ProtocolType.Tcp);
    sock.Bind(new IPEndPoint(IPAddress.Any, 8877));
    sock.Listen(8);

    while (true)
    {
        try
        {
            Socket accepted = await sock.AcceptAsync();
            var ns = new NetworkStream(accepted, true);
            var ss = new SslStream(ns);

            var options = new SslServerAuthenticationOptions
            {
                CertificateRevocationCheckMode = X509RevocationMode.NoCheck,
                ClientCertificateRequired = false,
                RemoteCertificateValidationCallback =
                (object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) =>
                {
                    // check client certificate
                    return true;
                },
                ServerCertificateSelectionCallback =
                (object sender, string hostName) =>
                {
                    // check hostName
                    return serverCert;
                }
            };

            await ss.AuthenticateAsServerAsync(options, default);

            ss.Close();
            ss.Dispose();

        }
        catch (Exception ex)
        {
            Console.WriteLine(ex);
        }
    }
}

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vhnatyk]

Sorry - seems quite obvious but nevertheless I would like to make it clear that as of today (october 2019) it doesn't support https on mobile platforms?

[geoperez]

Nop, the support to HTTPS right now is only with Windows.

[TomCJones]

ssl is part of windows (kerberos) auth - it belongs to the security team not the network team. If you want to work beyond windows you will need to grab some other code - or perhaps put the code into platform specific modules.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.