Arduino-esp32: [HELP] How to post an HTTP with headers?

[sarkrui]

Hardware:

Board: Wemos Lolin32 Lite
Core Installation version: 1.0.4
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: Unknown
Upload Speed: 115200
Computer OS: Mac OSX

Description:

I was able to POST a header-less HTTP request to Firebase Realtime Database, while I failed to post HTTP requests with headers (http code 403.) to an Entity Dataset powered by TU Eindhoven. The server side seems to work well as I was able to POST, GET, PUT via Insomnia. On my ESP32, the http.getString returns Access with api_token failed., which seems headers of the request were missing (it's confirmed from the server side that there were attempts of HTTP requests without headers, although I did include them in the code).

Sketch: (leave the backquotes for code formatting)

//Change the code below by your sketch
#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h> //v6
const char* ssid     = "";
const char* password = "";
int sensor_0 = 1234;
int sensor_1 = 5678;
String postMessage;

void setup() {
void WPAConnect() {

  Serial.begin(115200);
  delay(500);
  WiFi.begin(ssid, password);
  Serial.print("Connecting to WPAPersonal Wi-Fi");
  while (WiFi.status() != WL_CONNECTED)
  {
    Serial.print(".");
    delay(200);
  }
  Serial.println();
  Serial.print("Connected with IP: ");
  Serial.println(WiFi.localIP());
  Serial.println();
}

void setup() {
  WPAConnect();
  curlRequest();
}
void curlRequest() {
  if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status
    HTTPClient http;
    const int capacity = JSON_OBJECT_SIZE(2);
    StaticJsonDocument<capacity> doc;
    doc["sensor_0"] = sensor_0;
    doc["sensor_1"] = sensor_1;
    serializeJsonPretty(doc, postMessage);
    serializeJsonPretty(doc, Serial);
    http.begin("https://data.id.tue.nl/datasets/entity/145/item/"); //Specify destination for HTTP request
    http.addHeader("Content-Type", "application/json");
    http.addHeader("api_token",  "pMXFOLpinNq7EzrVQ/nXXLWl+96c9Dk+f8K2iVj+QoQqajaRQJYMeWObg2XYmcX1");
    http.addHeader("resource_id", "breathing_0");
    http.addHeader("token", "1234");
    // http.begin("https://tu-eindhoven-shared-projects.firebaseio.com/.json"); //Specify destination for HTTP request
    int httpCode = http.POST(postMessage);
    if (httpCode > 0) {
      Serial.println();
      Serial.println(httpCode);
      if (httpCode == 200) {
        Serial.println("Hooray!");
      }
    }
  }
}
void loop() {
}

Debug Messages:

{
  "sensor_0": 1234,
  "sensor_1": 5678
}Access with api_token failed.
403

[Jeroen88]

Your POST request is correct, including the way you add request headers, and your server sends a reasonable reply.

The problem is that you do not send what your server needs to fulfill the request.

There is nothing wrong with the ESP32 libraries, this is not an issue for here.

[sarkrui]

Hi @Jeroen88,

Thanks for the prompt response!
I thought I have fulfilled the server's requirement as I was able to POST via cURL and Insomnia and they both returned successful responses.

Any idea what might went wrong with my Arduino code? Thanks!

[Jeroen88]

Well you should debug your own code :)

One thing I can think of is that the capacity of the json object is too small. You forget space for the string duplications (like "sensor_0"), see here

[Bmooij]

Could it be the problem that you are sending a https request without adding a certificate?

[sarkrui]

One thing I can think of is that the capacity of the json object is too small.

Hi @Jeroen88 , thanks for correcting me. Even though I have enlarged the capacity to JSON_OBJECT_SIZE(2) + 20 . It did not work still unfortunately. FYI, the serializeJsonPretty(doc, Serial); did work well, so this should not be problematic.

Hi @Bmooij , I tried including the rootCACertificate based on the BasicHTTPSClient example, no luck. I do really know if it's necessary to generate my own root Certificate as the one in the code below is from the example.

#include <ArduinoJson.h> //v6
const char* ssid     = "";
const char* password = "";
int sensor_0 = 1234;
int sensor_1 = 5678;
String postMessage;

#include <Arduino.h>
#include <WiFi.h>
#include <WiFiMulti.h>
#include <HTTPClient.h>
#include <WiFiClientSecure.h>

// This is GandiStandardSSLCA2.pem, the root Certificate Authority that signed
// the server certifcate for the demo server https://jigsaw.w3.org in this
// example. This certificate is valid until Sep 11 23:59:59 2024 GMT
const char* rootCACertificate = \
                                "-----BEGIN CERTIFICATE-----\n" \
                                "MIIF6TCCA9GgAwIBAgIQBeTcO5Q4qzuFl8umoZhQ4zANBgkqhkiG9w0BAQwFADCB\n" \
                                "iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \
                                "cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\n" \
                                "BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQw\n" \
                                "OTEyMDAwMDAwWhcNMjQwOTExMjM1OTU5WjBfMQswCQYDVQQGEwJGUjEOMAwGA1UE\n" \
                                "CBMFUGFyaXMxDjAMBgNVBAcTBVBhcmlzMQ4wDAYDVQQKEwVHYW5kaTEgMB4GA1UE\n" \
                                "AxMXR2FuZGkgU3RhbmRhcmQgU1NMIENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IB\n" \
                                "DwAwggEKAoIBAQCUBC2meZV0/9UAPPWu2JSxKXzAjwsLibmCg5duNyj1ohrP0pIL\n" \
                                "m6jTh5RzhBCf3DXLwi2SrCG5yzv8QMHBgyHwv/j2nPqcghDA0I5O5Q1MsJFckLSk\n" \
                                "QFEW2uSEEi0FXKEfFxkkUap66uEHG4aNAXLy59SDIzme4OFMH2sio7QQZrDtgpbX\n" \
                                "bmq08j+1QvzdirWrui0dOnWbMdw+naxb00ENbLAb9Tr1eeohovj0M1JLJC0epJmx\n" \
                                "bUi8uBL+cnB89/sCdfSN3tbawKAyGlLfOGsuRTg/PwSWAP2h9KK71RfWJ3wbWFmV\n" \
                                "XooS/ZyrgT5SKEhRhWvzkbKGPym1bgNi7tYFAgMBAAGjggF1MIIBcTAfBgNVHSME\n" \
                                "GDAWgBRTeb9aqitKz1SA4dibwJ3ysgNmyzAdBgNVHQ4EFgQUs5Cn2MmvTs1hPJ98\n" \
                                "rV1/Qf1pMOowDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYD\n" \
                                "VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCIGA1UdIAQbMBkwDQYLKwYBBAGy\n" \
                                "MQECAhowCAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNl\n" \
                                "cnRydXN0LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNy\n" \
                                "bDB2BggrBgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRy\n" \
                                "dXN0LmNvbS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZ\n" \
                                "aHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAWGf9\n" \
                                "crJq13xhlhl+2UNG0SZ9yFP6ZrBrLafTqlb3OojQO3LJUP33WbKqaPWMcwO7lWUX\n" \
                                "zi8c3ZgTopHJ7qFAbjyY1lzzsiI8Le4bpOHeICQW8owRc5E69vrOJAKHypPstLbI\n" \
                                "FhfFcvwnQPYT/pOmnVHvPCvYd1ebjGU6NSU2t7WKY28HJ5OxYI2A25bUeo8tqxyI\n" \
                                "yW5+1mUfr13KFj8oRtygNeX56eXVlogMT8a3d2dIhCe2H7Bo26y/d7CQuKLJHDJd\n" \
                                "ArolQ4FCR7vY4Y8MDEZf7kYzawMUgtN+zY+vkNaOJH1AQrRqahfGlZfh8jjNp+20\n" \
                                "J0CT33KpuMZmYzc4ZCIwojvxuch7yPspOqsactIGEk72gtQjbz7Dk+XYtsDe3CMW\n" \
                                "1hMwt6CaDixVBgBwAc/qOR2A24j3pSC4W/0xJmmPLQphgzpHphNULB7j7UTKvGof\n" \
                                "KA5R2d4On3XNDgOVyvnFqSot/kGkoUeuDcL5OWYzSlvhhChZbH2UF3bkRYKtcCD9\n" \
                                "0m9jqNf6oDP6N8v3smWe2lBvP+Sn845dWDKXcCMu5/3EFZucJ48y7RetWIExKREa\n" \
                                "m9T8bJUox04FB6b9HbwZ4ui3uRGKLXASUoWNjDNKD/yZkuBjcNqllEdjB+dYxzFf\n" \
                                "BT02Vf6Dsuimrdfp5gJ0iHRc2jTbkNJtUQoj1iM=\n" \
                                "-----END CERTIFICATE-----\n";
void loop() {
  WiFiClientSecure *client = new WiFiClientSecure;
  if (client) {
    client -> setCACert(rootCACertificate);
    {
      // Add a scoping block for HTTPClient https to make sure it is destroyed before WiFiClientSecure *client is
      HTTPClient http;
      Serial.print("[HTTPS] begin...\n");
      curlRequest();
    }
    delete client;
  } else {
    Serial.println("Unable to create client");
  }
  Serial.println();
  Serial.println("Waiting 10s before the next round...");
  delay(5000);
}

[Bmooij]

You can't use the certificate from the example, because it referrers to https://jigsaw.w3.org.
You need to use the certificate from data.id.tue.nl.

Tutorial: https://techtutorialsx.com/2017/11/18/esp32-arduino-https-get-request/

You can try the following certificate:

const char* rootCACertificate = \
"-----BEGIN CERTIFICATE-----\n"
"MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/\n" \
"MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\n" \
"DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow\n" \
"PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD\n" \
"Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB\n" \
"AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O\n" \
"rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq\n" \
"OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b\n" \
"xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw\n" \
"7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD\n" \
"aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV\n" \
"HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG\n" \
"SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69\n" \
"ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr\n" \
"AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz\n" \
"R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5\n" \
"JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo\n" \
"Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ\n" \
"-----END CERTIFICATE-----\n";

[Jeroen88]

You can try https but I don't think it will solve the problem since the server did send a reply over http. The server would have dropped the connection earlier if it didn't accept http

[Jeroen88]

Could you try

int httpCode = http.POST("");

Instead of your current POST?

[sarkrui]

You can't use the certificate from the example, because it referrers to https://jigsaw.w3.org.
You need to use the certificate from data.id.tue.nl.

Tutorial: https://techtutorialsx.com/2017/11/18/esp32-arduino-https-get-request/

You can try the following certificate:

@Bmooij Thanks for your suggestion. No luck so far. It still returns a 403 Access with api_token failed error.

int httpCode = http.POST("");

@Jeroen88 It still returned the same error.

[Bmooij]

The issue is in the library. The header token overrides the header api_token.
api_token is never send.

Try to reverse the order. First add the token and then the api_token:

http.addHeader("token", "1234");
http.addHeader("api_token", "pMXFOLpinNq7EzrVQ/nXXLWl+96c9Dk+f8K2iVj+QoQqajaRQJYMeWObg2XYmcX1");

[sarkrui]

@Bmooij You're definitely a genius! It works like a charm now. So, the api_token should be placed at the last line the addHeader section then? That's odd but glad that we now know what went wrong.

[Jeroen88]

@Bmooij why do you think the header is never sent?

[Bmooij]

@Jeroen88 I know it because I added debug lines to the HTTPClient and printed the values that are send with the client.

I didn't go full into it (because it is weekend), but I expected the error around here: https://github.com/espressif/arduino-esp32/blob/cec3fca4ad4a39feb463f9298ab3238819732d50/libraries/HTTPClient/src/HTTPClient.cpp#L912

[Bmooij]

There are 2 possible options to workaround the issue:

Option 1: Change the order

http.addHeader("token", "1234");
http.addHeader("api_token", "pMXFOLpinNq7EzrVQ/nXXLWl+96c9Dk+f8K2iVj+QoQqajaRQJYMeWObg2XYmcX1");

Option 2: Set replace to false

http.addHeader("api_token", "pMXFOLpinNq7EzrVQ/nXXLWl+96c9Dk+f8K2iVj+QoQqajaRQJYMeWObg2XYmcX1");
http.addHeader("token", "1234", false, false); // set replace to false

[Bmooij]

@Jeroen88 It is definitely the indexOf.

String _headers = "api_token : pMXFOLpin...";
// add token header:
_headers.indexOf("token :") != -1

[Jeroen88]

@Bmooij thnx, I may check it tomorrow and make a PR for it :)

[Jeroen88]

@sarkrui by the way, you do not need arduino json for such simple usage. You may POST "{'sensor_0':1234', 'sensor_1':45678}" directly to the http client.

If the response from the server is also a json, it is more useful to parse the reply with arduino json.

[sarkrui]

@Jeroen88 Thanks for the information although that seems not working for me. Also, the sensor_x values are going to be read from analog pins. In that case, ArduinoJson should be necessary.

[Jeroen88]

@sarkrui I made a typo, maybe that is because it did not work. Even if you read from the pins, you can just construct the JSON yourself:

char buffer[20];
sprintf(buffer, "{\"sensor_0\": %d, \"sensor_1\": %d}", 1234, 5678);
http.POST(buffer);

[Jeroen88]

I just made PR #3487 to fix the original problem

[sarkrui]

@Jeroen88 Thanks again! Although you made another tiny typo haha. A backslash is necessary before a double-quotation mark.

char buffer[20];
sprintf(buffer, "{\"sensor_0\": %d, \"sensor_1\": %d}", 1234, 5678);
http.POST(buffer);