Arduino: Reading response headers received by HTTPClient

[Potato-Matic]

Hi.
I'm currently working on talking to and from google scripts using POST messages, over https. Right now, POSTing to google is working quite well (except for when they change certs :/) but in trying to receive data from google scripts I've run into a gap in the documentation.

I have it working with the linux utility cURL making the POST from my desktop, but I have to tell cURL to follow redirects as google's content service redirects to a new temporary page with the information returned by the script, rather than sending it directly. (aside: the google scripts content service allows for the returning of arbitrary data, like text, json, and files but does so by redirecting to a temporary one-use URL with the content, while the html service allows a script to return HTML pages, in an invisible frame of course)

I have a pretty good idea of the behavior of cURL that I want to emulate (thanks manpages!), but it requires that I read the headers in the response received back from google in order to implement it _properly_. I read around a bit here:
https://links2004.github.io/Arduino/dd/d8d/class_h_t_t_p_client.html
and saw lots of stuff dealing with headers, but not a lot of explanation on how to use what. If someone could explain some of the potentially useful functions or provide some snippet of example code I'd really appreciate it.

There's a lot of potential in google scripts doing the heavy-lifting for ESP8266s, now that https makes it possible, with complete access to all of google's services. I'd really like to open this up for people to play with (I have an example up on github for logging temp & humidity data to a google spreadsheet) and this would further enhance the utility of this combo.

My testing google script preforms some arbitrary manipulation of the text it's sent and the key part of my ESP8266 code is as follows (source, destination, and text are form entries all used by the google script for it's own purposes):

bool POSTData(String source, String destination, String text){
if(!http.begin(https_site, fingerprint)){
return false;
}

http.addHeader("Content-Type", "application/x-www-form-urlencoded");
String data = "";//construction of the URL encoded data to be POSTed
data += "source="
data += source
data += "&destination="
data += destination
data += "&text="
data += text
int code = http.POST(data);

Serial.print("POST returned code:");
Serial.println(code);
if(code == 302 || code == 303){
Serial.println("Redirect detected. (currently unhandled)");
}
Serial.println("Returned data:");
Serial.println(http.getString());

//presumably get the headers around here and preform the redirect

http.end();

return true;
}

[krzychb]

@Matthew-Bradley,
I do not have an answer to your question regarding snippets to analyze headers but would like to thank you for digging out this link https://links2004.github.io/Arduino/index.html

@Links2004,
Do you have a process in place to keep the above doxygen in sync with https://github.com/esp8266/Arduino git repository or this is one off snapshot?

Krzysztof

[Potato-Matic]

To be honest, I'd just be happy figuring out which function returns the headers from the httpclient. Even if they need further parsing, it's waaaay better than ditching it and trying to parse/generate http on my own.

[igrr]

To obtain response headers from HTTPClient you need to specify header names in advance.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.h#L168

Once request finished, you may query values of collected headers using one of these functions:
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.h#L169-L173

For instance you may want to collect headers "Location" and "Last-Modified":

HTTPClient http;
http.begin(host, post, fingerprint);

const char* headerNames[] = { "Location", "Last-Modified" };
httpClient.collectHeaders(headerNames, sizeof(headerNames)/sizeof(headerNames[0]));

int rc = http.GET();
if (rc >0) {
  if (httpClient.hasHeader("Location")) {
    // do something...
  }
  String lastModified = httpClient.header("Last-Modified");
  // ...
}

This follows same design as ESP8266WebServer. ESP8266WebServer has similar functions, but for collecting request headers.

[Potato-Matic]

Ok, thanks a bunch! I'll get started putting that to use ASAP. Thanks for the example code!

Ps. Can you explain the size calculation:
sizeof(headerNames)/sizeof(headerNames[0])
I'm curious as to what exactly it's returning to the function (the fraction taken up by the first header name?)

PPs. Is the following valid in general, with the above in mind?
int rc = http.GET();
if (rc >0) {
if (httpClient.hasHeader("header1")) {
String header_1 = httpClient.header("header1");
}
if (httpClient.hasHeader("header2")) {
String header_2= httpClient.header("header2");
}
// ...
}

[krzychb]

Ps. Can you explain the size calculation:

Hi @Matthew-Bradley,

sizeof(headerNames) returns the size of the whole table.
The table itself contains pointers to headers (not the actual headers).

sizeof(headerNames[0]) returns the size of a single pointer.

Therefore sizeof(headerNames) / sizeof(headerNames[0]) return the number of table elements. i.e. number of headers

Krzysztof

[Potato-Matic]

Ah. Gotcha. I was having a moment of stupid, trying to remember my C programming courses.

I assume then that the slightly re-arranged code I posted is valid then.

[arnolde]

      if (https.hasHeader("Set-Cookie")) {
        String cookies = https.header("Set-Cookie");
        Serial.println("Got these cookies: "+cookies);
      }

This will only return 1 cookie header. How do I get all of them, if there are multiple Set-Cookie headers?
(See also https://github.com/esp8266/Arduino/issues/1368#issuecomment-423071242)