Azure-iot-sdk-c: X.509 certificate not being read properly from its file. Only works by embedding in source code.

Created on 26 Jun 2020 · 22Comments · Source: Azure/azure-iot-sdk-c

I am working with the Azure IoT SDK for C. My device generates a signed X.509 device certificate file and private key file initially using openssl, then enrolls the device on the Azure IoT Portal.

The problem is, the certificate and key information don't seem to be getting read properly from the files into my C application. The authentication fails.

However, if I instead directly embed the certificate and key string into my application as follows, this works.

static const char* CERTIFICATE =
"-----BEGIN CERTIFICATE-----""\n"
"MIIDhjCCAm4CFFE3UVO9gg8Da3qjp1pjfNyrJJmjMA0GCSqGSIb3DQEBCwUAMHkx""\n"
"CzAJBgNVBAYTAkNaMQ8wDQYDVQQIDAZQcmFndWUxDzANBgNVBAcMBlByYWd1ZTEM""\n"
"MAoGA1UECgwDSW9UMQwwCgYDVQQLDANJb1QxDDAKBgNVBAMMA0lvVDEeMBwGCSqG""\n"
"SIb3DQEJARYPZW1haWxAZW1haWwuY29tMCAXDTIwMDYyMjE1NDgyMloYDzIyMDAw""\n"
"MTAyMTU0ODIyWjCBgzELMAkGA1UEBhMCQ1oxDzANBgNVBAgMBlByYWd1ZTEPMA0G""\n"
"A1UEBwwGUHJhZ3VlMQwwCgYDVQQKDANJb1QxDDAKBgNVBAsMA0lvVDEWMBQGA1UE""\n"
"AwwNcGF1bC1kZXZpY2UtMTEeMBwGCSqGSIb3DQEJARYPZW1haWxAZW1haWwuY29t""\n"
"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyEz0GNdXrdiX1BPOxulf""\n"
"MiabawwnwXUe9XypuVZ7YQyZGzjS+7onOvab3fcXGEs91fwClLGw8kt2Em6iMTlQ""\n"
"mEG4YAcFw/Ww5HbxPWkjxrKtwC3+1IMql2UJp3EGepBPS0OwGT8RM7PU1sgmOgN+""\n"
"8kjB2RKGkb7ifwjTEqD7PVAhTbPYlxhJsfw1BJI7eZuTm+7wiaMK5t9omW7SWMVV""\n"
"0vleXYME5P2wI1MfIgQwBV/KJ08043WyOmtIDfvPNhkG/EE2XS6TciNA+eJuraDi""\n"
"cBn05J+hcLMUUzE7w+j+YEGAVYGPYsbWT1Czfq6fDZT6NmFFAbZtXxjz5OQwT/cm""\n"
"2QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBld/HtPUJdwRW/rDQHDe1wdBw7Yadt""\n"
"wjXNJKBUDmmTzQof4cf8DaiSvV8u62TShfV19OXBWoY4uo3UmZp80w/bSOGennk9""\n"
"FTKjOLHZmRk0ejRuuNhD47QZi1IqNuiJKhX0hgBXRCNbGr+ZdSmYoVgGGOiiYJAZ""\n"
"AkvEaTXl0clwB1UnGttk5AH9/G0yIre53XpNRhZ2X/PDgWQNTlRHw0XAegFj/Rie""\n"
"xwmDyv3FtV0vOzhpdgpOKotUc3g+050yv9JJncJaM4IOlQM0usywP7uNKWw0bnVt""\n"
"Ue5ZTSsZ6XHUNBylB3z1HYsjaBTIpXDFdTd4KmAPSHyi2sQ9HXeBsrZW""\n"
"-----END CERTIFICATE-----";

Otherwise I am trying to parse the Device_cert.crt file which has the certificate as follows, into my String hsm_info->certificate :

static const char* const CERTIFICATE = "Device_cert.crt";   
hsm_info->certificate_file           = CERTIFICATE;

FILE* fCert = fopen(hsm_info->certificate_file, "r");

if (fCert != NULL)
{
   fseek(fCert, 0, SEEK_END);
   size_t len = ftell(fCert);
   rewind(fCert);

   if ((hsm_info->certificate = (char*)malloc(len + 1)) == NULL)
   {
      (void)printf("Failure allocating certificate cache\r\n");
      result = NULL;
   }
   else
   {
      len = fread(hsm_info->certificate, 1, len, fCert);
      fclose(fCert);
      hsm_info->certificate[len] = 0;
      (void)printf("%s\r\n", hsm_info->certificate);   // The certificate string prints exactly as expected here
      (void)printf("\n\n");
   }

It looks to have parsed properly via the output, but the certificate is still invalid when trying the authenticate. Would anyone know what could be the issue here reading the certificate from the file ?

question

Source

pmulligan-rk

All 22 comments

@pmulligan-rk , We see that the same question is posted on Stackoverflow as well X.509 certificate not being read properly from its file

SatishBoddu-MSFT on 26 Jun 2020

@pmulligan-rk , We see that the same question is posted on Stackoverflow as well X.509 certificate not being read properly from its file

@SatishBoddu-MSFT It's removed to avoid duplicates

pmulligan-rk on 26 Jun 2020

Hi @pmulligan-rk just to double check. I see that the file type you are attempting to read in from is .crt and not .pem. Those can generally be binary DER form or base64. Are you sure the cert is base64 encoded?

danewalton on 26 Jun 2020

Hi @pmulligan-rk just to double check. I see that the file type you are attempting to read in from is .crt and not .pem. Those can generally be binary DER form or base64. Are you sure the cert is base64 encoded?

@danewalton Thanks for your reply. I'm generating the device certificate and key via openssl on Linux as follows. deviceName and other variables are of course defined before this.

deviceCert=${deviceName}".crt"
deviceKey=${deviceName}".key"
deviceCSR=${deviceName}".csr"

openssl genrsa -rand device-certs/rand.txt -out device-certs/${deviceKey} 2048
openssl req -new -key device-certs/${deviceKey} -out device-certs/${deviceCSR} -subj "/C=$country/ST=$state/L=$locality/O=$organization/OU=$organizationalunit/CN=$commonname/emailAddress=$email"

openssl x509 -req -in device-certs/${deviceCSR} -CA ca-certs/CAcert.pem -CAkey ca-certs/CAprivatekey.key -CAcreateserial -out device-certs/${deviceCert} -days 365

pmulligan-rk on 26 Jun 2020

@pmulligan-rk can you try changing the output file extension from ".crt" to ".pem" near the top of that?

danewalton on 26 Jun 2020

If it works when you embed the certificate in the source but not when you read it from a file then I would suggest you should embed it in the source, read it from the file and compare the two strings.

markrad on 26 Jun 2020

👍1

If it works when you embed the certificate in the source but not when you read it from a file then I would suggest you should embed it in the source, read it from the file and compare the two strings.

@markrad I have done that. The strings look exactly the same when printed out via _printf()_. This is what baffles me.

pmulligan-rk on 27 Jun 2020

If it works when you embed the certificate in the source but not when you read it from a file then I would suggest you should embed it in the source, read it from the file and compare the two strings.

@markrad I've just compared the embedded certificate string with the one read from the file.

This is it embedded in the source code :

static const char* const CERT =
"-----BEGIN CERTIFICATE-----""\n"
"MIIDhjCCAm4CFFE3UVO9gg8Da3qjp1pjfNyrJJmkMA0GCSqGSIb3DQEBCwUAMHkx""\n"
"CzAJBgNVBAYTAkNaMQ8wDQYDVQQIDAZQcmFndWUxDzANBgNVBAcMBlByYWd1ZTEM""\n"
"MAoGA1UECgwDSW9UMQwwCgYDVQQLDANJb1QxDDAKBgNVBAMMA0lvVDEeMBwGCSqG""\n"
"SIb3DQEJARYPZW1haWxAZW1haWwuY29tMCAXDTIwMDYyNzA5MDkyMFoYDzIyMDAw""\n"
"MTA3MDkwOTIwWjCBgzELMAkGA1UEBhMCQ1oxDzANBgNVBAgMBlByYWd1ZTEPMA0G""\n"
"A1UEBwwGUHJhZ3VlMQwwCgYDVQQKDANJb1QxDDAKBgNVBAsMA0lvVDEWMBQGA1UE""\n"
"AwwNcGF1bC1kZXZpY2UtMjEeMBwGCSqGSIb3DQEJARYPZW1haWxAZW1haWwuY29t""\n"
"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuYAPInn7NXozHAnDCE3H""\n"
"QQ2N/8dPckQBUUozMOX5vnGruzD+qJYxps+70Mhuju7KoRA6OtY+PrePh8W5q7Ph""\n"
"QyZDa3UZRPg7BOhn6sXJ4j77qgM3mqrhf1qeQMLYbyWQYJsAEJ6F56K74mMjz8Eh""\n"
"YtaTIJXHxCqIt87FPgFNFry6DaiY+7wX/oHso8pxIbj/pIGTcgsahqB+3ilTx2tb""\n"
"ArU3JkeKZj5R/quuYQ0qaFHa8YSXIPfrsIY1DE3MlK1DkDCOVQM3ke8YTIrIFskt""\n"
"TvDgxPRNyTjw7VQkUsO0Wag2L2kWYkAWEjF418BBFKPVtDuj4qYIZPN3dhQAkMZI""\n"
"yQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQAkvNLSg+HF7hzDTyFJFGC3BLz/36z6""\n"
"AwxBRQFNsv6EdbsqhBceWDSG651ERO1NUk9gwXWIcKLMlQGKbavSVO6iuMRM+4nG""\n"
"Tv2Tuce0ybiiCWZFq8gAoXOV9rgV0vEjtaSOEEXb41c6DH9QUd2lyEwuSdfeTssS""\n"
"oRyRecMy2i0UsQ5DJLXqJTLdbgwWoneeTP4j2cDOODniLoA5UCRdHYFD/8UnHMYU""\n"
"CRNDN46s0uU0ukyVRsE4jpbVJD930GDugopg5ApZ8ov9S8zS57dpY6dqRXRcOb5k""\n"
"+m/PbHElpi85tOQMunxulPulMFO7auSACBVrlawg1AYtN3uzNQan8Smm""\n"
"-----END CERTIFICATE-----";

This is the printed out certificate string on the terminal when it is read from the file :

When I use strcmp in the code to compare both the embedded certificate string and the certificate string read from the file, the result is 10, so it actually means that they are not the same somehow. .

pmulligan-rk on 27 Jun 2020

You may have a memory corruption issue. I removed the quotes, newlines, and semi-colon and compared it to the other. They are identical. If they fail strcmp though then it is likely something has damaged one or the other.

markrad on 29 Jun 2020

A strcmp return of 10, implies the issue is 10 char in, which is the end of "-----BEGIN".

ericwol-msft on 29 Jun 2020

You may have a memory corruption issue. I removed the quotes, newlines, and semi-colon and compared it to the other. They are identical. If they fail strcmp though then it is likely something has damaged one or the other.

@markrad @ericwol-msft

Something is definitely wrong with the certificate and key files. When I check them with command : openssl x509 -noout -text -in my_cert.pem

This reads :

unable to load certificate
140134768538752:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE

The same thing with the key file. I don't undertsnad it tho. The start lines are there, as shown in the file and also printed out on the terminal when when reading the file into a string with fread.

pmulligan-rk on 30 Jun 2020

probably needs to look at the file with a binary editor to see if there are issues.

ericwol-msft on 30 Jun 2020

@ericwol-msft I've examined the files with a hex-editor. Everything looks ok to. I've uploaded the certificate file here for reference. I had to change the extension name to .txt as uploading .pem files is not supported here.
my-cert.txt

pmulligan-rk on 30 Jun 2020

@pmulligan-rk I passed authorization several days ago. But when I switch to SAS and switch back to X.509. My device failed to authorize.
I have same issue as yours: 1)strcmp return 10, 2)openssl x509 -noout -text -in key.pem return "unable to load certificate..PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE".
But When I copy both content to Sublime Text Editor and compare file via meld. All of them is identical.
I want to know how do youfix this issue? thanks

williamaccuenergy on 22 Jul 2020

@pmulligan-rk I passed authorization several days ago. But when I switch to SAS and switch back to X.509. My device failed to authorize.
I have same issue as yours: 1)strcmp return 10, 2)openssl x509 -noout -text -in key.pem return "unable to load certificate..PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE".
But When I copy both content to Sublime Text Editor and compare file via meld. All of them is identical.
I want to know how do youfix this issue? thanks

Hi @williamaccuenergy . I forgot to update here.

My problem turned out to be something I didn't expect :

Reading the device certificate and private key from the files wasn't actually the problem. However, reading the device identity from a file was.

On startup, my device reads the device identity from flash via a bash script and then writes this to a file called Device_identity.txt , which is of course also used in the authentication process afterwards in the C code.

From my script I was doing echo $device_name > Device_identity.txt . This was including a new line character, which made the device identity invalid to the one that's saved on the portal.

I fixed it with echo -n $device_name > Device_identity.txt

I hope it helps in some way:)

pmulligan-rk on 22 Jul 2020

👍1

@pmulligan-rk Thanks for your reply.

williamaccuenergy on 22 Jul 2020

@pmulligan-rk Thanks for your reply.

@williamaccuenergy You asked about why we need the device id on the device. This is the common name of the certificate, but I believe it also must be provided and is read via the API custom_hsm_get_common_name ?

I based my solution on this example :

https://kevinsaye.wordpress.com/2020/04/14/using-a-real-certificate-with-the-azure-iot-client-c-sdk-and-the-provisioning-service/

Are you doing something similar? Thanks

pmulligan-rk on 22 Jul 2020

@pmulligan-rk thank you very much. I believe that reading cert and key file is not problem. I create X509 cert following https://github.com/Azure/azure-iot-sdk-c/blob/master/tools/CACertificates/CACertificateOverview.md. and create IoT device coresponding cert's common name on Azure IoT portal. But I always faile to authorize. log information: "/third-party/azure-iot-sdk/azure-iot-sdk-c/iothub_client/src/iothubtransport_mqtt_common.c Func:mqtt_operation_complete_callback Line:1755 Connection Not Accepted: 0x5: Not Authorized". Do you have any idea? thanks advanced.

williamaccuenergy on 22 Jul 2020

@williamaccuenergy Another issue could be the system date. Does it have the correct date when it's trying to provision? If this is out of date it will also fail to authorize. I had problems with the system date just during boot-up of my device as it takes time to sync. I recently implemented a loop in my code to attempt to provision multiple times to solve it.

pmulligan-rk on 23 Jul 2020

@pmulligan-rk Thanks for your reply. I think copying cert file from different OS may be problem. I force NTP sync when system reboot, and force NTP sync every 5 minutes at system run time. Thanks again

williamaccuenergy on 23 Jul 2020

@pmulligan-rk , thanks for posting back the solution to this issue.
We really appreciate your contribution.

This issue will be closed for now, but please feel free to reopen it if you would like to follow up on this matter.
Thanks,
Azure IoT SDK Team.

ewertons on 28 Jul 2020

@SatishBoddu-MSFT, @pmulligan-rk, @williamaccuenergy, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey