Arduino: WiFiClientSecure GET reads weird chars in response

[DayDreamiiing]

Basic Infos

• [x] This issue complies with the issue POLICY doc.
• [x] I have read the documentation at readthedocs and the issue is not addressed there.
• [x] I have tested that the issue is present in current master branch (aka latest git).
• [x] I have searched the issue tracker for a similar issue.
• [x] If there is a stack dump, I have decoded it.
• [x] I have filled out all fields below.

Platform

• Hardware: ESP8266
• Core Version: 2019/10/31
• Development Env: Arduino IDE
• Operating System: Windows

Settings in IDE

• Module: Generic ESP8266 Module
• Flash Mode: DOUT
• Flash Size: 4MB
• lwip Variant: Lower Memory
• Reset Method: ck
• Flash Frequency: 40Mhz
• CPU Frequency: 80Mhz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

I am using my esp8266 to collect weather data using bme280. I want the data to be stored in a database. I managed to set up a codeigniter php web application which stores and returns this data using HTTPS GET/POST.
This API works fine and I do get the information I want inside the Arduino script. But I would like to know why I get these "45e" and "0" in front and after my default GET reply from. They are not showing inside the printed html code in the web browser. So where do they come from?

You can check "whateverweather.bplaced.net/api/weather_stations/81:1b:09:65:9d:b6" in your browser to see the real output.

Additionally I used an answer from this issue: https://github.com/esp8266/Arduino/issues/5041 to get all rows. Because in the original example of HTTPSRequest I could only get this first "45e" which confused me very hard.

MCVE Sketch

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

const char* ssid = "SSID";
const char* password = "PASSWORD";

const char* host = "whateverweather.bplaced.net"; <-- My Website AND REST API 
const int httpsPort = 443;

void setup() {


  ... connecting to wifi ...

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);

  client.setInsecure();

  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  String url = "/api/weather_stations/81:1b:09:65:9d:b6";
  Serial.print("requesting URL: ");
  Serial.print(host);
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String chunk = "";
  int limit = 1;
  String response="";
 // String line = client.readStringUntil('z');
    do {
    if (client.connected()) {
      client.setTimeout(2000);
      chunk = client.readStringUntil('\n');
      response += chunk;
      Serial.println(chunk);
    }
  } while (chunk.length() > 0 && ++limit < 100);


  Serial.println("reply was:");
  Serial.println("==========");
 Serial.println(response);
  Serial.println("==========");
}

void loop() {
}

Debug Messages

Output:

[{"id":"1","location":"Paderborn Wohnheim","location_alt":"Z.Nr. 4.0.2","created_at":"2019-10-24 16:59:22","mac":"81:1b:09:65:9d:b6","ip":"103.80.152.37","measureIntervall":"60","updateIntervall":"600","rebootIntervall":"86400","settings_utd":"0","last_report":"2019-10-23 12:26:24"},{"id":"2","location":"Paderborn FHDW","location_alt":"P-212","created_at":"2019-10-22 16:18:10","mac":"a9:89:a2:20:47:25","ip":"99.138.70.193","measureIntervall":"60","updateIntervall":"600","rebootIntervall":"86400","settings_utd":"0","last_report":"2019-08-21 15:47:26"},{"id":"3","location":"Paderborn FHDW","location_alt":"Serverraum","created_at":"2019-10-23 01:02:43","mac":"da:3a:ad:bf:32:d9","ip":"246.99.14.204","measureIntervall":"60","updateIntervall":"600","rebootIntervall":"86400","settings_utd":"1","last_report":"2019-10-20 17:38:28"},{"id":"7","location":"Paderborn Wohnheim","location_alt":"Grillplatz","created_at":"2019-10-24 09:56:33","mac":"81:75:4e:af:01:48","ip":"120.77.169.181","measureIntervall":"60","updateIntervall":"700","rebootIntervall":"86400","settings_utd":"1","last_report":"2019-10-21 21:49:02"}]

0

[Jeroen88]

45e is exactly the size in hex of the string on the second line (1119 bytes in decimal), so your server sends the data 'chunked'. If you check the HTTP header 'Transfer-Encoding' is should show 'Transfer-Encoding: chunked'.

If you use ESP8266HTTPClient that library should be able to handle chunked response correctly.

[DayDreamiiing]

45e is exactly the size in hex of the string on the second line (1119 bytes in decimal), so your server sends the data 'chunked'. If you check the HTTP header 'Transfer-Encoding' is should show 'Transfer-Encoding: chunked'.

If you use ESP8266HTTPClient that library should be able to handle chunked response correctly.

Thanks for your reply. Unfortunately I did not managed to get a correct request with the ESP8266HTTPClient library because the server is https secured. You now how to do https request with this library? That would help me A LOT.

[Jeroen88]

Look around here where you find a little outdated example. Instead of a finger print I advise you to use certificate validation if security is important for you. Here you can find the various validation methods of BearSSL, now the default secure client for ESP8266
If you combine both examples by preparing a client including it's security from the second link and passing that into the basicHttpsClient of the first link, it should work.
BEWARE: memory is limited on the ESP8266. Make sure you don't declare variables locally to functions because of stack size limitations. Look at ways to reduce the buffer sizes involved ('standard' https uses 16k of buffers which consumes a lot of ram space). The send buffer can always be reduced, the receive buffer only if your server supports max fragment length negotiation (which is rather novel).
Good luck!

[earlephilhower]

Thanks @Jeroen88 .