Arduino: can't load x509 certificates

[copercini]

Basic Infos

Hardware

Hardware: ESP-12
Core Version: git

Description

I am trying use a x509 certificate to connect with a TLS1.2 host, but in if(espClient.loadCertificate(ca)) I get Success to open ca file and not loaded

I have tried with PEM and DER formats

Settings in IDE

Module: Generic ESP8266 Module
Flash Size: 4MB/1MB
CPU Frequency: 80Mhz
Flash Mode: dio
Flash Frequency: 40Mhz
Upload Using: SERIAL
Reset Method: ck

Sketch

...
#include "FS.h"
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

WiFiClientSecure espClient;

void setup() {
  Serial.begin(115200);
  setup_wifi();
  delay(1000);
  if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
  }

File ca = SPIFFS.open("/ca.crt", "r"); //replace ca.crt eith your uploaded file name
  if (!ca) {
    Serial.println("Failed to open ca file");
  }
  else
  Serial.println("Success to open ca file");

if(espClient.loadCertificate(ca))
  Serial.println("loaded");
  else
  Serial.println("not loaded");

}
...

I am trying use a x509 certificate to connect with a TLS1.2 host, but in if(espClient.loadCertificate(ca)) I get Success to open ca file and not loaded

I have tried with PEM and DER formats

[igrr]

Certificate should be in DER format. Could you please upload the one you have tried somewhere and post a link?

[igrr]

Also, please enable debug output (Core+SSL) and add Serial.setDebugOutput(true); after Serial.begin. Then post full output you get.

[copercini]

Thanks for the fast reply:

WiFi connected
IP address:
192.168.137.109
SPIFFSImpl: allocating 512+180+1400=2092 bytes
SPIFFSImpl: mounting fs @300000, size=fb000, block=2000, page=100
SPIFFSImpl: mount rc=0
Success to open ca file
not loaded
SPIFFS_close: fd=1
Attempting MQTT connection...:ref 1
please start sntp first !
State: sending Client Hello (1)
:sent 72
:rn 1460
:rd 5, 1460, 0
:rdi 1460, 5
:rd 1455, 1460, 5
:rdi 1455, 1455
:c0 1455, 1460
:rn 1310
:rd 1310, 1310, 0
:rdi 1310, 1310
:c0 1310, 1310
State: receiving Server Hello (2)
State: receiving Certificate (11)
Error: invalid handshake
Alert: unexpected message
Alert: close notify
failed, rc=-2 try again in 5 seconds
:sent 7
:rn 7
:rcl
:abort

Here is the zip with certs and the complete sketch
certificates.zip

[chaeplin]

I have checked with der format.
openssl x509 -in ca.crt -outform der -out ca.der

if(espClient.loadCertificate(ca)) with

bool WiFiClientSecure::loadCertificate(Stream& stream, size_t size)
{
    if (!_ssl) {
        DEBUGV("is _ssl?");
        return false;
    }
    return _ssl->loadObject(SSL_OBJ_X509_CERT, stream, size);
}

SPIFFSImpl: mount rc=0
Success to open ca file
825
is _ssl?not loaded
SPIFFS_close: fd=1
pm open,type:2 0

if (espClient.loadCACert(ca, ca.size())) with

bool WiFiClientSecure::loadCACert(Stream& stream, size_t size)
{
    if (!_ssl) {
        DEBUGV("is _ssl?");
        return false;
    }
    return _ssl->loadObject(SSL_OBJ_X509_CACERT, stream, size);
}

SPIFFSImpl: mount rc=0
Success to open ca file
825
is _ssl?not loaded
SPIFFS_close: fd=1
pm open,type:2 0

[igrr]

Ah, this is a regression I think... Previously you were able to load certs before calling connect. Now you can only load them after connect which isn't very useful if you would like to use a client side cert :)
Thanks for pointing this out, will fix.

[copercini]

@igrr downgrading this commit https://github.com/esp8266/Arduino/commit/b41266097fe6d39d0021c29bb09fdbb69c7d5df4 the certificate is loaded, but without support to CA root cert

[copercini]

And downgraded version still doesn't handshake =/

[madpilot]

It looks like a problem with last axTLS upgrade (35ee060c098a1a69ab5b545f54babffb7ada50df).

The last commit that works for me is d6e38f0abd2e1bf796a32e8b1a24d37fdc7daaf8

[copercini]

I think this bug is caused in upgrade to axtls v2. 0.0, here: https://github.com/igrr/axtls-8266/pull/19

[igrr]

Not exactly. The bug was added to WiFiClientSecure, axTLS is okay.

[hkartadi]

Hi all...

Just wondering if the bug already solved? I think I also encounter the same thing, I am able to load the file from FFS, however it is not able to load the certificate in WifiSecureClient.

Thank you for the help.
Below is some portions of my code:

void certver() {
  SPIFFS.begin();

  Dir dir = SPIFFS.openDir("/");
  while (dir.next()) {
      Serial.print(dir.fileName());
      File f = dir.openFile("r");
      Serial.println(f.size());
  }

  File ca = SPIFFS.open("/ca.crt.der", "r");
  if (!ca) {
    Serial.println("Couldn't load ca cert");
    return;
  }
  if (espClientSecure.loadCACert(ca, ca.size())) {
    Serial.println("Loaded ca cert");
  } else {
    Serial.println("Didn't load ca cert");
    return;
  }

  File cl = SPIFFS.open("/client.crt.der", "r");
  if (!cl) {
    Serial.println("Couldn't load client cert");
    return;
  }
  if (espClientSecure.loadCertificate(cl)) {
    Serial.println("Loaded client cert");
  } else {
    Serial.println("Didn't load client cert");
    return;
  }

  File key = SPIFFS.open("/client.key.der", "r");
  if (!key) {
    Serial.println("Couldn't load key");
    return;
  }

  if (espClientSecure.loadPrivateKey(key)) {
    Serial.println("Loaded Key");
  } else {
    Serial.println("Didn't load Key");
  }
}

OUTPUT:

WiFi connected
IP address:
[192.168.10.94]
/ca.crt.der759
/client.crt.der702
/client.key.der1192
Didn't load ca cert
Attempting MQTT connection...please start sntp first !
failed, rc=-2 try again in 5 seconds

[copercini]

@hkartadi

The WiFiClientSecure just create a SSLContext in https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp#L315 when you call _connectSSL.

To load the certificates before connect, it should create a SSLContext in loadCACert or loadCertificate or loadPrivateKey.... and verify when called _connectSSL if it has already been created.

[igrr]

PR #3271 has the fix for this issue.

[jogaraobommana]

i am getting this error

Success to open private cert file
loadObject: ssl_obj_memory_load returned -269
private key not loaded

[chegewara]

Hi @igrr
sorry for bothering with such trivial question, but how to convert pem or der certificate to array like in this example:
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequestCACert/CACert.ino

Also what is difference in use setCACert and setCACert_P?

[chadouming]

xxd -i -a

On Mon, Aug 27, 2018, 2:52 PM chegewara, [e-mail] wrote:

Hi @igrr https://github.com/igrr
sorry for bothering with such trivial question, but how to convert pem or
der certificate to array like in this example:

https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequestCACert/CACert.ino

Also what is difference in use setCACert and setCACert_P?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/esp8266/Arduino/issues/2470#issuecomment-416329462,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABSUGxqsGDWF4iLVAP52jMj23mVQIJl5ks5uVD_lgaJpZM4Jy2XN
.

[chegewara]

@chadouming thanks a lot

[DerGuteWolf]

What is difference in use setCACert and setCACert_P?

[ateeb327]

Hello, I was having the same issue and finally solved it by using
espClient.loadCACertificate(ca)
after the connect function.
@copercini
Look at my following working code:

`#include "FS.h"
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.

const char* ssid = "Your SSID name";
const char* password = "Your SSID password";
const char* mqtt_server = "test.mosquitto.org"; //MQTT broker ip 
//IPAddress server(172, 16, 0, 95);
void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();

}
WiFiClientSecure espClient;
//SPIFFS.begin();
//File ca = SPIFFS.open("/mosquitto_or.der", "r");
//if(!ca) {
//  Serial.println("FIle not Found!");
//  return;  
//}
//
//if(espClient.loadCertificate(ca)) {
//  Serial.println("Loaded Cert");
//} else {
//  Serial.println("Didn't load cert");
//  return;
//}

PubSubClient client(mqtt_server,8883,callback,espClient); //set  MQTT port number to 8883 as per //standard
long lastMsg = 0;
char msg[50];
int value = 0;




void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}


void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {

    File ca = SPIFFS.open("/mosquitto_or.der", "r"); //replace ca.crt eith your uploaded file name
  if (!ca) {
   Serial.println("Failed to open ca file");
  }
  else
 Serial.println("Success to open ca file");



    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("ESP8266Client1")) {
      Serial.println("connected");
      if(espClient.loadCACert(ca))
  Serial.println("loaded");
  else
 Serial.println("not loaded");

      // Once connected, publish an announcement...
      client.publish("outTopic", "hello world");
      // ... and resubscribe
      client.subscribe("inTopic");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  SPIFFS.begin();
  setup_wifi();
  delay(1000);
  if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
  }

  client.setServer(mqtt_server, 8883);
client.setCallback(callback);

}



void loop() {

  if (!client.connected()) {
    reconnect(); 
  }
  client.loop();

  long now = millis();
  if (now - lastMsg > 2000) {
    lastMsg = now;
    ++value;
    snprintf (msg, 75, "hello world #%ld", value);
    Serial.print("Publish message: ");
    Serial.println(msg);
    client.publish("outTopic", msg);
  }
}`

[tianmanik]

in my arduino IDE at tools menu, there is no spiffs flash size to upload the skecth, is it the problem?
my code :

include "FS.h"

include

include

include

const char* ssid = "MANIK";
const char* password = "manik123";

//-----pzem define-----//

include

PZEM004Tv30 pzem(2, 0);
//-----pzem define end-----//

include

const char* AWS_endpoint = "xxxxx-ats.iot.ap-southeast-1.amazonaws.com"; //MQTT broker ip

void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();

}

WiFiClientSecure espClient;
PubSubClient client(AWS_endpoint, 8883,espClient); //set MQTT port number to 8883 as per //standard
long lastMsg = 0;
char msg[90];
int value = 0;

void setup() {
Serial.begin(115200);
Serial.setDebugOutput(true);

espClient.setBufferSizes(512, 512);
Serial.println();
Serial.print("Connecting to ");
Serial.println(ssid);

WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}

Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());

//Initialize File system
if(SPIFFS.begin())
{
Serial.println("SPIFFS got Initialized successfully");
}
else
{
Serial.println("SPIFFS Initialization is failed");
}
//Format File system
if(SPIFFS.format())
{
Serial.println("File system Formatted");
}
else
{
Serial.println("File system formatting error");
}

// Load certificate file
File cert = SPIFFS.open("cert.der", "r"); //replace cert.crt eith your uploaded file name
if (!cert) {
Serial.println("Failed to open cert file");
}
else
Serial.println("Success to open cert file");

delay(1000);

if (espClient.loadCertificate(cert))
Serial.println("cert loaded");
else
Serial.println("cert not loaded");

// Load private key file
File private_key = SPIFFS.open("/private.cer", "r"); //replace private eith your uploaded file name
if (!private_key) {
Serial.println("Failed to open private file");
}
else
Serial.println("Success to open private file");

delay(1000);

if (espClient.loadPrivateKey(private_key))
Serial.println("private key loaded");
else
Serial.println("private key not loaded");

// Load CA file
File ca = SPIFFS.open("/CA.der", "r"); //replace ca eith your uploaded file name
if (!ca) {
Serial.println("Failed to open ca ");
}
else
Serial.println("Success to open ca");

delay(1000);

if(espClient.loadCACert(ca))
Serial.println("ca loaded");
else
Serial.println("ca not loaded");

Serial.print("Heap: "); Serial.println(ESP.getFreeHeap());
}

void loop() {

//-----pzem-----//
float voltage = pzem.voltage();
float current = pzem.current();
float power = pzem.power();
float energy = pzem.energy();
float frequency = pzem.frequency();
float pf = pzem.pf();
if(voltage == NAN || current == NAN || voltage == NAN || power == NAN || energy == NAN || frequency == NAN || pf == NAN){ Serial.println("Reading failed.");}
else{
//publishing data
snprintf (msg, 75, "{\"Voltage\": #%ld\"}", voltage);
snprintf (msg, 75, "{\"Current\": #%ld\"}", current);
snprintf (msg, 75, "{\"Power\": #%ld\"}", power);
snprintf (msg, 75, "{\"Energy\": #%ld\"}", energy);
snprintf (msg, 75, "{\"Frequency\": #%ld\"}", frequency);
snprintf (msg, 75, "{\"PF\": #%ld\"}", pf);
client.publish("outTopic", msg);
Serial.println("Publishing:- ");
Serial.print("Voltage: "); Serial.print(voltage); Serial.println("V");
Serial.print("Current: "); Serial.print(current); Serial.println("A");
Serial.print("Power: "); Serial.print(power); Serial.println("W");
Serial.print("Energy: "); Serial.print(energy, 3); Serial.println("kWh");
Serial.print("Frequency: "); Serial.print(frequency, 1); Serial.println("Hz");
Serial.print("PF: "); Serial.println(pf);

  Serial.println("Success\n");
}

delay(10);
}

output
cant open cert file
fail to loaded cert file
cant open private file
fail to loaded private file
cant open ca file
fail to loaded ca file

[ateebpk2]

@tianmanik
You need to follow the following steps in order to enable SPIFFS in your Arduino IDE.

1. > Download the tool: https://github.com/esp8266/arduino-esp8266fs-plugin/releases/download/0.5.0/ESP8266FS-0.5.0.zip
2. > In your Arduino sketchbook directory, create tools directory if it doesn’t exist yet.
3. > Unpack the tool into tools directory (the path will look like /Arduino/tools/ESP8266FS/tool/esp8266fs.jar) If upgrading, overwrite the existing JAR file with the newer version.
4. > Restart Arduino IDE.
5. > Open a sketch (or create a new one and save it).
6. > Go to sketch directory (choose Sketch > Show Sketch Folder).
7. > Create a directory named data and any files you want in the file system there.
8. > Make sure you have selected a board, port, and closed Serial Monitor.
9. > If your board requires you to press a button (or other action) to enter bootload mode for flashing a sketch, do that now.
10. > Select Tools > ESP8266 Sketch Data Upload. This should start uploading the files into ESP8266 flash file system. When done, IDE status bar will display SPIFFS Image Uploaded message.

You also need to select FileSystem Size of your CHip in order to upload data to SPIFFS. Look at the following image for Flash Sizes:

For more information kindly refer to the ESP8266 Filesystem Documentation at following link:
https://arduino-esp8266.readthedocs.io/en/latest/filesystem.html

[xmoulin]

Work fine for me, over a board ESP8266 : Wemos D1 R1
Thanks you all. I use this topic to do this, some more code to add Root certificate, private cert and private key.

_For this test, I use https://www.httpcs.com/fr/convertisseur-ssl to convert the AWS IoT Certificated to ".DER" file_

/**
 * Tester sur Wemos D1 R1
 * Utilisation des certificat AWS
 * Code basé sur l'exemple : https://github.com/esp8266/Arduino/issues/2470
 **/
#include "FS.h"
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.

const char* ssid = "MySSID";
const char* password = "MyPASSWORD";
const char* mqtt_server = "xxxxx.iot.eu-west-1.amazonaws.com"; //MQTT broker ip

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();

}
WiFiClientSecure espClient;


PubSubClient client(mqtt_server, 8883, callback, espClient); //set  MQTT port number to 8883 as per //standard
long lastMsg = 0;
char msg[50];
int value = 0;


void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}


void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("reconnect - Heap - before : "); Serial.println(ESP.getFreeHeap());

    // -------------------
    // ------- CA --------
    // -------------------
    File ca = SPIFFS.open("/CAcertificate.der", "r"); //replace ca.crt eith your uploaded file name
    if (!ca)
      Serial.println("Failed to open ca file");
    else
      Serial.println("Success to open ca file");
    if (espClient.loadCACert(ca))
      Serial.println("loaded CA");
    else
      Serial.println("not loaded CA");
    delay (100);

    // -------------------
    // ------- Certif-----
    // -------------------
    File certificate = SPIFFS.open("/certificate.der", "r"); //replace ca.crt eith your uploaded file name
    if (!certificate)
      Serial.println("Failed to open certificate file");
    else
      Serial.println("Success to open certificate file");
    //https://github.com/esp8266/Arduino/issues/2470
    //Serial.println("LoadCertificate");
    //espClient.loadCertificate(ca);
    if (espClient.loadCertificate(certificate))
      Serial.println("loaded certificate");
    else
      Serial.println("not loaded certificate");
    delay(100);

    // -------------------
    // ------- privateKey-
    // -------------------
    //Load private key file
    File private_key = SPIFFS.open("/092b035fee-private.pem.key", "r"); //replace private eith your uploaded file name
    if (!private_key)
      Serial.println("Failed to open private file");
    else
      Serial.println("Success to open private file");
    if (espClient.loadPrivateKey(private_key))
      Serial.println("private key loaded");
    else
      Serial.println("private key not loaded");
    delay(100);


    Serial.print("Attempting MQTT connection...");
    // Attempt to connect

    if (client.connect("ESP8266Client1")) {
      Serial.println("connected");


      // Once connected, publish an announcement...
      client.publish("topic/topic_1", "hello world");
      // ... and resubscribe
      client.subscribe("topic/topic_2");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
  Serial.print("reconnect - Heap - after : "); Serial.println(ESP.getFreeHeap());
}

void setup() {
  Serial.begin(115200);
  SPIFFS.begin();
  setup_wifi();
  delay(1000);
  if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
  }

  client.setServer(mqtt_server, 8883);
  client.setCallback(callback);

}



void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  long now = millis();
  if (now - lastMsg > 5000) {
    lastMsg = now;
    ++value;
    snprintf (msg, 75, "hello world #%ld", value);
    Serial.print("Publish message: ");
    Serial.println(msg);
    client.publish("topic/topic_1", msg);
    Serial.print("loop - Heap : "); Serial.println(ESP.getFreeHeap());
  }
}