Mqttnet: AWS support?

[stefxx]

Describe the bug

Is this a bug? Of not supported? I am trying to connect to AWS IoT with client certificate over port 8883. I see some references to ALPN but that doesn't seem a hard requirement. But I couldn't find anyway to configure ALPN as well, so that might be related.

Which project is your bug related to?

• Client

Details

I checked that the certificates are valid by using "openssl s_client -CAfile ca.pem -cert client.pem -key client.key -connect your-endpoint.amazonaws.com:8883". Also, I catch the certificate errors in MQTTnet so that all seems fine. But when I try a ConnectAsync, this is the error I receive: "MQTTnet.Exceptions.MqttCommunicationTimedOutException' was thrown".

MQTTnet.Exceptions.MqttCommunicationTimedOutException
  HResult=0x80131500
  Message=Exception of type 'MQTTnet.Exceptions.MqttCommunicationTimedOutException' was thrown.
  Source=mscorlib
  StackTrace:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MQTTnet.PacketDispatcher.MqttPacketAwaiter`1.<WaitOneAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MQTTnet.Client.MqttClient.<SendAndReceiveAsync>d__50`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MQTTnet.Client.MqttClient.<AuthenticateAsync>d__44.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at MQTTnet.Client.MqttClient.<ConnectAsync>d__35.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at MQTTnet.Client.MqttClient.<ConnectAsync>d__35.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at MyProgram.MyProg.PD() in MyProgram.vb:line 577
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

[stefxx]

Some additional info here

[SeppPenner]

Did you already take a look at https://github.com/chkr1011/MQTTnet/wiki/Client#connecting-with-amazon-aws?

[chkr1011]

You can also try to use the WebSocket4Net extension of this lib because WebSocket4Net supports more protocols than the WebSocket implementation in .NET.

[stefxx]

Thanks for the pointers! I did look at the link but this is all about Sigv4, not certificate authentication. Also, I tried to use WebSocket4Net but i can't seem to find any docs or examples on how to use it. What should I do except Dim mqttClient As Client.MqttClient = New MqttFactory().UseWebSocket4Net.CreateMqttClient?

The more I read, the more it seems I need to use the "x-amzn-mqtt-ca" header somehow...

[dxmann]

Thanks for the pointers! I did look at the link but this is all about Sigv4, not certificate authentication. Also, I tried to use WebSocket4Net but i can't seem to find any docs or examples on how to use it. What should I do except Dim mqttClient As Client.MqttClient = New MqttFactory().UseWebSocket4Net.CreateMqttClient?

The more I read, the more it seems I need to use the "x-amzn-mqtt-ca" header somehow...

I'm working with the same problem: I want to authenticate in AWS IoT with certificate on port 8883.
When I have prepared the certificate and started the client I have this issue:

Error: MQTTnet.Exceptions.MqttCommunicationException: Authentication failed, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090327): Errore sconosciuto durante l'elaborazione del certificato.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__64_2(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at MQTTnet.Implementations.MqttTcpChannel.ConnectAsync(CancellationToken cancellationToken)
   at MQTTnet.Implementations.MqttTcpChannel.ConnectAsync(CancellationToken cancellationToken)
   at MQTTnet.Internal.MqttTaskTimeout.WaitAsync(Func`2 action, TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Adapter.MqttChannelAdapter.ConnectAsync(TimeSpan timeout, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MQTTnet.Adapter.MqttChannelAdapter.WrapException(Exception exception)
   at MQTTnet.Adapter.MqttChannelAdapter.ConnectAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ConnectAsync(IMqttClientOptions options, CancellationToken cancellationToken)
Verbose: MqttClient Disconnecting [Timeout={0}]
Verbose: MqttClient Disconnected from adapter.
Info: MqttClient Disconnected.

I don't think that the problem is in the certificate.pem.crt and privatekey.pen.key released by AWS...
so where's the problem?

[stefxx]

I haven't been able to get this working. But if you want to make sure your certificates are fine, you can test it like this:

openssl s_client -CAfile ca.pem -cert client.pem -key client.key -connect your-endpoint.amazonaws.com:8883

If you ever make this work, I am still very interested!

[dxmann]

Is useful to setup the rootCA certificate? AWS released 3 main files: dev certificate, dev private key and root CA certificate. How to setup the last one?

Note: I have found this function that takes the dev's certificate and private key and decodes the certificate (it works)

private async Task<X509Certificate2> LoadPemCertificate(string certificatePath, string privateKeyPath)
        {
            using var publicKey = new X509Certificate2(certificatePath);

            var privateKeyText = await File.ReadAllTextAsync(privateKeyPath);
            var privateKeyBlocks = privateKeyText.Split("-", StringSplitOptions.RemoveEmptyEntries);
            var privateKeyBytes = Convert.FromBase64String(privateKeyBlocks[1]);
            using var rsa = RSA.Create();

            if (privateKeyBlocks[0] == "BEGIN PRIVATE KEY")
                rsa.ImportPkcs8PrivateKey(privateKeyBytes, out _);
            else if (privateKeyBlocks[0] == "BEGIN RSA PRIVATE KEY")
                rsa.ImportRSAPrivateKey(privateKeyBytes, out _);
            else
                throw new ArgumentException("key type not recognized");

            // keyPair contains the decrypted certificate
            var keyPair = publicKey.CopyWithPrivateKey(rsa);
            return new X509Certificate2(keyPair.Export(X509ContentType.Cert));
        }

What can I do now?

[dxmann]

I haven't been able to get this working. But if you want to make sure your certificates are fine, you can test it like this:

openssl s_client -CAfile ca.pem -cert client.pem -key client.key -connect your-endpoint.amazonaws.com:8883

If you ever make this work, I am still very interested!

I have performed the test:

openssl s_client -CAfile /.../AmazonRootCA1.pem -cert /.../...-certificate.pem.crt -key /.../....-private.pem.key -connect <endpoint>.iot.<region>.amazonaws.com:8883
CONNECTED(00000003)
...
SSL handshake has read 2393 bytes and written 1608 bytes
Verification error: unable to get local issuer certificate
...

[dxmann]

ok, now the verification works: I have to remove the -CAfile argument... it's strange but works

[stefxx]

Great find!

Have you resolved the verification in the .Net code as well? I would love to see a working example of that.

[dxmann]

My terror was that the error is only in the win implementation of dotnet code... nope...
in linux I have a similar stacktrace: the error is inside the SslStram.Authenticate:

---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
... etc ...

but in the configuration I have specified that

var options = new MqttClientOptionsBuilder()
                    .WithClientId(account.ClientId)
                    .WithTcpServer(broker.Host, broker.Port)
                    .WithCleanSession()
                    .WithProtocolVersion(MQTTnet.Formatter.MqttProtocolVersion.V311)
                    //.WithCommunicationTimeout(TimeSpan.FromSeconds(30))
                    .WithTls(new MqttClientOptionsBuilderTlsParameters
                    {
                        UseTls = true,
                        SslProtocol = System.Security.Authentication.SslProtocols.Tls12,
                        IgnoreCertificateChainErrors = false,
                        IgnoreCertificateRevocationErrors = false,
                        AllowUntrustedCertificates = false,
                        Certificates = new List<X509Certificate> { cert },
                        //CertificateValidationHandler = delegate { return true; }
                    })
                    .Build();

_NOTE_
Debugging in Visual Studio I can confirm that the protocol that arrives in SslStream is correct: System.Security.Authentication.SslProtocols.Tls12; so why I have an error " Interop+Crypto+OpenSslCryptographicException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate"?

[dxmann]

ah my test with openss was:

openssl s_client -key <dev_id>-private.pem.key -cert <dev_id>-certificate.pem.crt -connect <code>.iot.<region>.amazonaws.com:8883

CONNECTED(00000003)
...
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: <bla>
Shared Requested Signature Algorithms: <bla>
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2393 bytes and written 1618 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: <bla>
    Session-ID-ctx: 
    Master-Key: <bla>
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: <bla>
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

[dasa-asen]

I had a similar problem when connecting to AWS IoT. The trick for me was to export and recreate the certificate in the MqttClientOptionsBuilderTlsParameters. Using only new[] { cert } did not work.

Certificates = new[] { new X509Certificate(cert.Export(X509ContentType.Pfx)) },

If there is a better solution, please share it.

[stefxx]

Hi @dasa-asen , thanks! Can you post a little more of your code? I still can't get it to work. Some larger example would be helpfull.

Thanks!

[dasa-asen]

Hi @dasa-asen , thanks! Can you post a little more of your code? I still can't get it to work. Some larger example would be helpfull.

Sure. Here is my code for opening the MQTT/TLS client:

X509Certificate2 cert = Aws.CertificateReader.ReadCertFromZipFile(certificateManager.PackageFile);

// Create a new MQTT client.
var factory = new MqttFactory();
mqttClient = factory.CreateMqttClient();

var tlsParameters = new MqttClientOptionsBuilderTlsParameters
{
    UseTls = true,
    Certificates =  new[] { new X509Certificate(cert.Export(X509ContentType.Pfx)) },
    IgnoreCertificateRevocationErrors = true, //Prevent online revocation check
    IgnoreCertificateChainErrors = disableServerValidation,
    AllowUntrustedCertificates = disableServerValidation,
};

// Create TCP based options using the builder.
var connectOptions = new MqttClientOptionsBuilder()
    .WithClientId(certificateManager.ThingName)
    .WithTcpServer(BrokerHostName, BrokerPort)
    .WithTls(tlsParameters)
    .WithCommunicationTimeout(TimeSpan.FromMilliseconds(connectionTimeout))
    .WithCleanSession()
    .Build();

var conResult = await mqttClient.ConnectAsync(connectOptions);

I receive the certificates and keys in PEM format packed in a single zip file. I unzip, read the PEM data and pass the binary content to the constructor of X509Certificate2.

I am certainly no TLS expert and have come up with this by trial and error. The CertificateValidationHandler specifically should probably be done in some other way, but this worked for me on my targets which is ASP.NET Core 3.1 running on either Windows or an embedded Linux target.

Edit: I removed the CertificateValidationHandler since it made no sense and made use of the ignore/allow options instead. My embedded Linux device were having problems with online revocation which is default. IgnoreCertificateRevocationErrors = true also changes the RevocationMode to NoCheck and that works a lot better for me. Currently I have not found a straight forward way to manually verify the server with a stand alone root certificate, but that is not important for my use case.

[nicolasr75]

Thanks guys for sharing your tests. I followed this description https://github.com/aws-samples/iot-dotnet-publisher-consumer
to set up the certs. First I got a timeout. I then realized that I need to set up a policy and assign that policy to the certificate as mentioned here https://github.com/aws-samples/iot-dotnet-publisher-consumer/issues/2

With these settings I can successfully connect and publish a message with MQTTnet to AWS!

[nicolasr75]

For others trying this I would like to mention two pitfalls that at first prevented me from publishing messages. Quoted from the AWS IoT docs:

AWS IoT supports MQTT quality of service (QoS) levels 0 and 1 only. AWS IoT doesn't support publishing or subscribing with QoS level 2. When QoS level 2 is requested, the message broker doesn't send a PUBACK or SUBACK.

The MQTT specification provides a provision for the publisher to request that the broker retain the last message sent to a topic and send it to all future topic subscribers. AWS IoT doesn't support retained messages. If a request is made to retain messages, the connection is disconnected.

Meaning, don't use .WithExactlyOnceQoS(), .WithRetainFlag()