Azure-iot-sdk-c: Not able to connect device to iothub through iotedge inside C module.

[rajaggrawal]

we are running iotedge module and inside module we are trying to create device-client using connection-string.
we are implementing this module similar to C# IdentityTranslationLite in C/C++ using azure-iot-sdk-c .
Following is the github link for C# IdentityTranslationLite

everything works well except device client.

following is format of my connection string.

string connectionString = "HostName=upi-devpoc01-iothub.azure-devices.net;GatewayHostName=raj-virtualbox;DeviceId=iot-edge-leaf-device-for-testing;SharedAccessKey=myDeviceKeyInBase64Format"

m_iothubHandle = IoTHubDeviceClient_CreateFromConnectionString(connectionString.c_str(), MQTT_Protocol);

but connection gets fail with following error.
AZURE_PLATFORM: ERROR: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
AZURE_PLATFORM: ERROR: Error: failure opening connection to endpoint

in C# it is done using following code.

IAuthenticationMethod authMethod = new DeviceAuthenticationWithRegistrySymmetricKey(leafDeviceId, signedKey);
                    leafDevice.DeviceClient = _useTransparentGateway ? DeviceClient.Create(_iothubHostName, _gatewayHostName, authMethod) : DeviceClient.Create(_iothubHostName, authMethod);

In C sdk method IoTHubDeviceClient_CreateFromConnectionString, we need to provide protocol(mqtt, http, amqp) as argument.
. But C# has different way of create device client and it works fine.

Could you please let me know how to create device client in C same way how it is created in C# IdentityTranslationLite module?

[rajaggrawal]

let me more specific

Following is in C#
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.devices.client.deviceclient.create?view=azure-dotnet#Microsoft_Azure_Devices_Client_DeviceClient_Create_System_String_System_String_Microsoft_Azure_Devices_Client_IAuthenticationMethod_Microsoft_Azure_Devices_Client_ClientOptions_

C# have this method.
Create(String, String, IAuthenticationMethod, ClientOptions)

i have to set IAuthenticationMethod as Microsoft.Azure.Devices.Client.DeviceAuthenticationWithRegistrySymmetricKey
please let me know how to set this using C sdk.

[ewertons]

Hi @rajaggrawal ,
this might be related to a well-known situation.
What platorm/OS are you using? do you have a certificate store in your system?
You might need to explicitly set the server root cert for validating the server cert during TLS negotation.
Are you setting the trusted cert (i.e., the root cert for server cert validation)?
Also, what version of the C SDK are you using?

[jspaith]

@rajaggrawal - The root issue here is how you are connecting the IoT C SDK to the IoT Edge (the GatewayHostName=<...> in the connection string.

Normally when the SDK connects directly to IoT Hub, the IoT Hub presents an SSL server certificate that the SDK trusts by default because all of Azure is based off of it and it chains up to a well-known certificate authority.

Your IoT Edge device has a certificate that presumably does not chain up to a common certificate authority (e.g. Baltimore). As far as the SDK is concerned and potentially in reality, this certificate is self-signed.

So when IoT Edge presents the SDK with its certificate, it doesn't chain up to anything the SDK trusts and we reject it. Hence AZURE_PLATFORM: ERROR: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed .

So the question becomes how to solve this? There are a few ways that will depend a lot on your solution and how much extra work you'd want to put in?

My first question is why do you need a device client running inside of an IoT Edge container in the first place? Can you be a module instead?

If you run your code as a Module and use the correct API, the SDK itself already has a mechanism where it will "magically" talk to IoT Edge to determine its server certificate and then the SDK will trust that.

As as added bonus, you don't have to provision out connection strings to your module because part of this process gets the module's authentication material too.

See here for an example. Specifically IoTHubModuleClient_LL_CreateFromEnvironment() instead of CreateFromConnectionString.
And this sample is a bit more complicated as it was designed to be run with modules talking to another module in the IoT Edge local routing. So it uses IoTHubModuleClient_LL_SendEventToOutputAsync. If you want to just send a message to IoT Hub directly use IoTHubModuleClient_LL_SendEventAsync instead.

I'd recommend this approach not knowing anything more about your setup, since it solves a bunch of provisioning problems for you and gets you into APIs we recommend for this case.

-

If for whatever reason you need to stay a device client, you need therefore to tell the DeviceClient "please trust this IoT Edge's" certificate.

For this I'd point you to the sample here. This is designed for a "down-stream" device - one theoretically not running on same physical hardware as Edge itself. But it will still work.
The most relevant code here is

cert_string = obtain_edge_ca_certificate();
(void)IoTHubDeviceClient_SetOption(device_handle, OPTION_TRUSTED_CERT, cert_string);

Which tells C SDK to trust the certificate. The hard part here though is that we do not e2e solve certificate provisioning for you; you need to put the certificate to trust per the IoT Edge instance into the docker container.

This means if you have 20 IoT Edge devices out there, you need to figure out how to provision the trusted certificate individual for each of the 20 devices.

The links at the top of the .c file in this sample have some more info about this, but we don't have any sort of sample e2e demonstrating this on a IoT Edge module directly because... The IoTHubModuleClient_LL_CreateFromEnvironment solves this for you anyway and is recommended approach.

[rajaggrawal]

@jspaith ,

Thanks for responding.

We are implementing identity translation module and leaf/downstream devices which doesnt have identity can communicate with cloud through this module, Hence we need a device client running inside of an IoT Edge container.

Our approach is exactly same as following registrationflow.png.
kindly refer following registrationflow.png.

https://github.com/Azure-Samples/azure-iot-edge-identity-translation-lite/blob/master/docs/media/registrationflow.png.

Yes, Authentication is successful after setting OPTION_TRUSTED_CERT with following method.
(void)IoTHubDeviceClient_SetOption(device_handle, OPTION_TRUSTED_CERT, cert_string);

My Question ->
Let say my device is registered to IoTHub through symmetric key. Do we have a way to connect my device client which is running inside edge container to iothub through IoTEdge without setting OPTION_TRUSTED_CERT ?
Authentication can be done with symmetric key without any need of OPTION_TRUSTED_CERT.

This can be done in C# without setting OPTION_TRUSTED_CERT.

Please check the following C# code .

https://github.com/Azure-Samples/azure-iot-edge-identity-translation-lite/blob/master/src/edge/modules/IdentityTranslationLite/Program.cs#L152

IAuthenticationMethod authMethod = new DeviceAuthenticationWithRegistrySymmetricKey(leafDeviceId, signedKey);
leafDevice.DeviceClient = _useTransparentGateway ? DeviceClient.Create(_iothubHostName, _gatewayHostName, authMethod) : DeviceClient.Create(_iothubHostName, authMethod);

C# have DeviceAuthenticationWithRegistrySymmetricKey which doesnt need OPTION_TRUSTED_CERT, but i am not able to find this in C sdk.

Could you please check and let me know how this can be done using C-sdk ?

[jspaith]

C doesn't have an equivalent of this to skip manually setting OPTION_TRUSTED_CERT. Either you get the certificate somehow and give it to C, or else you use the CreateFromEnvironment().

The C# sample is interesting. I'm not a C# expert by any stretch but I've spent some time staring at it. I'm not entirely convinced they're not doing some OPTION_TRUSTED_CERT equivalent on their side.

See specifically here where they are doing a CreateFromEnvironment equivalent which is going to plumb through to IoT Edge to get the trusted certificate.

Now in the code you referenced, it's not immediately clear how this would wire up the trusted certificate for a new DeviceClient but it might. In any case it's moot since C doesn't have C# style runtime authentication providers in any case.

---

I take it you don't want the OPTION_TRUSTED_CERT because it's hard to provision...? I'm going to guess yes to save back+forth latency between us and since you're commited on device client and since you're pretty deep into this anyway if just so you understand how this works. Let's look at how you can get this yourself.

IoT Edge puts a bunch of environment variables into your docker by default. The .cs you linked actually shows many of those being ready including the key thingy.

For C SDK, hsm_client_http_edge_get_trust_bundle is where we call over to IoT Edge for it to give us trusted certificate.

There's bunches of abstraction between that code and this code but the code I just linked (or a few lines beneath it) is where the C SDK turns this trusted certificate into the OPTION_TRUSTED_CERT "magically" for you.

So if you read in a bunch of environment variables that the actual hsm_client_http_edge_get_trust_bundle needs to build up REST request you could get that at runtime and then do the SetOption yourself. I'm not claiming this is super easy to do but it's easier if you at least see how it's done.

If you poke around hsm_client_http_edge.c you also get a sense of how we get the token material signed for authentication assuming that's needed.

Not sure if that helps or why you're worried on the trusted cert in 1st place?

[rajaggrawal]

@jspaith .

I have tested C# module and it doesn't depend on OPTION_TRUSTED_CERT to connect device client with cloud.
Please check the following code.

IAuthenticationMethod authMethod = new DeviceAuthenticationWithRegistrySymmetricKey(leafDeviceId, signedKey);
leafDevice.DeviceClient = _useTransparentGateway ? DeviceClient.Create(_iothubHostName, _gatewayHostName, authMethod) :

Device client is created with DeviceClient.Create method and this create method is taking IAuthenticationMethod as input argument. I dont see similar methods in C sdk.

Please refer documentation for DeviceClient.Create

https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.devices.client.deviceclient.create?view=azure-dotnet#Microsoft_Azure_Devices_Client_DeviceClient_Create_System_String_System_String_Microsoft_Azure_Devices_Client_IAuthenticationMethod_Microsoft_Azure_Devices_Client_ClientOptions_

I have run this module without adding any Trusted certs to docker container and it works fine.

Let say following is certificate section in my config.yaml file.

certificates:
device_ca_cert: "file:///iot-edge-device-raj-test-full-chain.cert.pem"
device_ca_pk: "file:///iot-edge-device-raj-test.key.pem"
trusted_ca_certs: "file:///azure-iot-test-only.root.ca.cert.pem"

I have also tested after deleting trusted_ca_certs from my config.yaml file to run C# module and my device-client is successfully connected with cloud.hence it doesn't depend on OPTION_TRUSTED_CERT.
But this doesn't work with C module.

We are looking similar authentication mechanism with C sdk.

[kiranpradeep]

The C# sample is interesting. I'm not a C# expert by any stretch but I've spent some time staring at it. I'm not entirely convinced they're not doing some OPTION_TRUSTED_CERT equivalent on their side.

@jspaith I could be wrong, but I assume the C# SDK installs! the iot-edge certificates into the user cert store. I had tried raising it as a bug[1] for ModuleClient.

@rajaggrawal If you chose to stay with DeviceClient, at the moment, it is easier if the C device client (inside the module) communicates directly to IoTHub (as then default certs in docker will do). C# DeviceClient is working magically only as a side effect of certificate install.

[rajaggrawal]

@kiranpradeep

I assume the C# SDK installs! the iot-edge certificates into the user cert store.
I am using test-root-certificate which is created locally. so this should not be be present in user cert store.

Test certs are created with following-:
https://docs.microsoft.com/en-us/azure/iot-edge/how-to-create-test-certificates

[rajaggrawal]

@jspaith

I still believe symmetric key passed in connection-string should be sufficient for authentication if parent child relationship is already established and C# module is authenticating through symmetric key without depending on OPTION_TRUSTED_CERT and this seems to be correct.

So edgehub should authenticate through symmetric key if parent/child is established already.

C# DeviceClient.Create takes IAuthenticationMethod as input and IAuthenticationMethod defines mutiple ways of authentication.
But not seeing similar support with C-SDK.

Please check this

https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.devices.client.iauthenticationmethod?view=azure-dotnet

[jspaith]

At the initial connection setup level where the OPTION_TRUSTED_CERT comes into play, the authentication mechanism for the SDK<=>Hub won't matter. The underlying SSL stack needs to trust the endpoint and if it doesn't - which we won't by default in this case - then none of the IoT Hub logic above will get a chance to kick in.

@kiranpradeep - thanks for insight into Edge auto-setting up certificates. I have reached out to them offline to see if they can provide help for this issue.

@rajaggrawal - could you let know what your system is specifically Windows/Linux/Mac? Based on GitHub issue kiranpradeep linked above this might impact how to proceed in this case.

[rajaggrawal]

@jspaith

My system is Linux.

[jspaith]

An IoT Edge developer got back to me and it turns out that the C# SDK is doing some extra logic here.

Looking at the C# SDK logic, I found this line interesting that whenever the module connected using the CreateFromEnvironmentAsync API, the X509Store was injected with the trust certs. There is no automatic linkage between the trusted store of the container and Edge's trust bundle on file system that I am aware of. I think it is just an accident that the device client is created within the same container where the moduleClient installs the cert within the container's X509 store and said C# SDK follows this practice. They would need to manually do this using OpenSSL in their C based module client to achieve the same effect for their C based device client.

I can't speak to C# design philosophy or the implementation requirements of their underlying TLS. However this is not a pattern we would bring onto C SDK. Our design principle is that each DeviceHandle or ModuleHandle is completely separate from others.

So that brings you back to needing to do OPTION_TRUSTED_CERT manual. I have some questions out about whether there's some docker magic you can do that would allow you to access the "trusted-cert.pem" from your app, poking through docker container and into the IoT Edge host OS itself. fopen and fread are much easier to deal with than the HTTP calls I gave you for getting the trust bundle. This also may be obvious already but I'm not an IoT Edge much less docker expert.

[jspaith]

I think we have an answer for you @rajaggrawal. You'll still need a SetOption on the trusted cert, but it's much easier than you directly calling into the IoT Edge http daemon:).

Shout out to my friend @massand for help with above and answer below.

They could stuff the same trust bundle location within the HostConfig > Binds docker options within the edgeHub twin config. See this article -
https://docs.microsoft.com/en-us/azure/iot-edge/how-to-access-host-storage-from-module

Please let us know if this addresses your questions or if there's other stuff we can help with.

[jspaith]

Since this thread has been quiet a few days, closing down. If anyone wants to get it going again please either create a new issue or else explicitly tag me on this one so I know to reactivate.

[az-iot-builder-01]

@rajaggrawal, @kiranpradeep, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey