Arduino: http.POST() limit of payload size?

Created on 6 Apr 2016  Â·  25Comments  Â·  Source: esp8266/Arduino

Hi!

I'm trying to send data as an HTTPClient to a server with http.POST() and it works great but the payload gets cut off around the 3000 character mark. Does this have something to do with "application/x-www-form-urlencoded"? It shouldn't, iirc, cause POST shouldn't have a size limit?

HTTPClient http;
http.begin("http://example.com/weather/espdata.php");
http.addHeader("Content-Type", "application/x-www-form-urlencoded");
http.POST(textToSend);
http.end();

I just wanted to know if there is a limit on payload size so I can rule out some other issue with my server. The String I'm sending gets put out normally in the serial monitor, so that shouldn't be the problem.

Thanks!

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

network web bug waiting for feedback
Source
picture TheAustrian

Most helpful comment

Ok ... this is what I wondered. I actually created a workaround that I'm using successfully, just in case anyone else requires a similar fix. To do it I had to use WifiClient:

#include <ESP8266HTTPClient.h>

String jsonPayload = "{ my big giant 3.5k byte json payload ....}";
int contentLength = jsonPayload.length();

  // create the request and headers
  String request = "POST " + String(endpointURL) + " HTTP/1.1\r\n" +
    "Host: " + String(endpointHost) + "\r\n" + 
    "Accept: application/json" + "\r\n" + 
    "Content-Type: application/json\r\n" +
    "Content-Length: " + String(contentLength) + "\r\n\r\n";

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  connectResult = client.connect(endpointHost, atoi(endpointPort));

  // This will send the request and headers to the server
  client.print(request);

  // now we need to chunk the payload into 1000 byte chunks
  int cIndex;
  for (cIndex = 0; cIndex < contentLength; cIndex = cIndex + 1000) {
    client.print(jsonPayload.substring(cIndex, cIndex+1000));
    //Serial.print(jsonPayload.substring(cIndex, cIndex+1000));
  }
  client.print(jsonPayload.substring(cIndex));
  //Serial.println(jsonPayload.substring(cIndex));

I have a little more response checking and other stuff ... but you get the idea. Using this I am now sending pretty big payloads, and it seems to work fine. The 1000 bytes was just an arbitrary guess ... if I have time I might try to adjust this to fine tune the operation.

Obviously there are limitations in this approach as I can't do SSL easily (or can I?) ... but maybe something like this ought to be rolled into the handling of the payload on the call to http.POST in the HTTPClient?

picture Humancell on 5 May 2016
🎉2 👍1

All 25 comments

Bump!

I'm seeing this same issue ... when my payload crossed the 3k mark I'm now getting a http.POST() return of -3 ... HTTPC_ERROR_SEND_PAYLOAD_FAILED ... but can't find any info on why this is, and what exactly failed?

Is there a buffer that needs to be adjusted, or?

Humancell picture Humancell on 5 May 2016

I think payload size > one packet size --> err.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp#L376-L380

chaeplin picture chaeplin on 5 May 2016

Ok ... this is what I wondered. I actually created a workaround that I'm using successfully, just in case anyone else requires a similar fix. To do it I had to use WifiClient:

#include <ESP8266HTTPClient.h>

String jsonPayload = "{ my big giant 3.5k byte json payload ....}";
int contentLength = jsonPayload.length();

  // create the request and headers
  String request = "POST " + String(endpointURL) + " HTTP/1.1\r\n" +
    "Host: " + String(endpointHost) + "\r\n" + 
    "Accept: application/json" + "\r\n" + 
    "Content-Type: application/json\r\n" +
    "Content-Length: " + String(contentLength) + "\r\n\r\n";

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  connectResult = client.connect(endpointHost, atoi(endpointPort));

  // This will send the request and headers to the server
  client.print(request);

  // now we need to chunk the payload into 1000 byte chunks
  int cIndex;
  for (cIndex = 0; cIndex < contentLength; cIndex = cIndex + 1000) {
    client.print(jsonPayload.substring(cIndex, cIndex+1000));
    //Serial.print(jsonPayload.substring(cIndex, cIndex+1000));
  }
  client.print(jsonPayload.substring(cIndex));
  //Serial.println(jsonPayload.substring(cIndex));

I have a little more response checking and other stuff ... but you get the idea. Using this I am now sending pretty big payloads, and it seems to work fine. The 1000 bytes was just an arbitrary guess ... if I have time I might try to adjust this to fine tune the operation.

Obviously there are limitations in this approach as I can't do SSL easily (or can I?) ... but maybe something like this ought to be rolled into the handling of the payload on the call to http.POST in the HTTPClient?

Humancell picture Humancell on 5 May 2016
🎉2 👍1

@Humancell use WiFiClientSecure.h for ssl.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

Chunk can be large to 1460.

chaeplin picture chaeplin on 5 May 2016

Thanks! Nice bonus! I'll make my changes. :-)

Humancell picture Humancell on 5 May 2016

I just ran into this issue as well. I believe the limit is just under 2000 Bytes. I hope there is a work around or solution for this. I'd rather use HTTPClient.

espmut picture espmut on 15 May 2017

Is there a work around for this issue? I need to send 8000bytes on POST>

melvinpmathew picture melvinpmathew on 31 May 2017

@TheAustrian @Humancell is this issue still valid with latest git?

devyte picture devyte on 17 Oct 2017
👍1

I am trying to upload a csv file with size of around 200 kB and using the method int HTTPClient::sendRequest(const char * type, Stream * stream, size_t size) and get the error HTTPC_ERROR_SEND_PAYLOAD_FAILED. Smaller size files work perfectly so I am assuming there is an unwritten file size limit or kind of a bug. This is mentioned in few stackoverflow questions as well but they have no answer (here.
Any new information, suggestions, progress, workaround?

mcanyucel picture mcanyucel on 29 Jan 2018

For me, I broke up the post into multiple posts.

On Mon, Jan 29, 2018 at 6:54 AM, Mustafa Can Yücel <[email protected]

wrote:

I am trying to upload a csv file with size of around 200 kB and using the
method int HTTPClient::sendRequest(const char * type, Stream * stream,
size_t size) and get the error HTTPC_ERROR_SEND_PAYLOAD_FAILED. Smaller
size files work perfectly so I am assuming there is an unwritten file size
limit or kind of a bug. This is mentioned in few stackoverflow questions as
well but they have no answer (here
https://stackoverflow.com/questions/41388417/send-payload-error-in-http-post-arduino-esp8266
.
Any new information, suggestions, progress, workaround?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/esp8266/Arduino/issues/1872#issuecomment-361251877,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbYZfq_JO_OP1ezEystbN2ttXG2DpOgyks5tPc2ngaJpZM4IBZhy
.

--
Machine Utilization Tracker
www.mutcloud.com
602.524.0192

espmut picture espmut on 29 Jan 2018

I see an issue when receiving rss feeds from a news portal over http GET (get string to be clear)

One RSS page with kind of high priority news gets transmitted normally and the esp can send it to a lcd display.

But whenever I try to get the "normal" news feeds, the esp just doesn't do anything at all.
Maybe it restarted?

The normal rss feed has 14527 Characters, the high priority one has 2676 Characters...

I've gone with the high priority one for now and I can see why nearly 15k characters is a bit overkill for this little thing, but still it's an issue....

Silberlachs picture Silberlachs on 30 Jan 2018

@espmut How did you manage to divide the file? I used another approach that reads the file into a WiFiClient using MTU_Size chunks but it floods everywhere with stack errors :(

mcanyucel picture mcanyucel on 30 Jan 2018

I could reproduce the error.
Can you retry sending big data without chunking into pieces, with DebugLevel HTTP_CLIENT (in Tools menu, with Serial for DebugPort) ?
I get:

[HTTP-Client][sendRequest] short write, asked for xxxx but got yyyy retry...

2MB data are working flawlessly with all debug options activated (they slow down processing a lot).
Bug is located, fix will come hopefully.

Full sketch:

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

#define ssid "open"
#define password ""

void setup()
{
  Serial.begin(115200);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  // Print the IP address
  Serial.println(WiFi.localIP());

}

class FakeStream: public Stream
{
  protected:
    size_t fakesize;

  public:
    FakeStream (size_t size): fakesize(size) { }

    virtual int available() {
      return fakesize;
    }
    virtual int read() {
      if (fakesize) {
        --fakesize;
        return 0x5a;
      }
      return -1;
    }
    virtual int peek() {
      return 0x5a;
    }
    virtual void flush() {
      fakesize = 0;
    };
    virtual size_t write (uint8_t data) {
      (void)data;
      return 1;
    }
    size_t size () {
      return fakesize;
    }
    const char* name () {
      return "5a";
    }
};


void loop()
{
  // put your main code here, to run repeatedly:
  HTTPClient http;
  http.begin("http://10.43.1.254:80/test/upload.php");

  int contentLength = 2000000;

  http.addHeader("Content-Type", "application/octet-stream");
  http.addHeader("Content-Length", String(contentLength));

  Serial.println("a----------------------------------------------");
  FakeStream s(contentLength);
  int httpCode = http.sendRequest("POST", &s, contentLength);
  Serial.println(httpCode);
  Serial.println("b----------------------------------------------");

  while (true)delay(1000);
}

The upload.php (got in some other issue):

<?php
file_put_contents('/var/www/html/test/test.txt', file_get_contents('php://input'));
?>
d-a-v picture d-a-v on 30 Jan 2018
👍2

@TheAustrian @mcanyucel @espmut can you unbreak-to-one-piece your POST and try this pull-request #4265 ?

@Silberlachs the fix is about outgoing data (not incoming data). If it still does not work for you, then please open a new issue with a minimal sketch to highlight the bug. It will be a lot easier for us to reproduce then fix.

d-a-v picture d-a-v on 31 Jan 2018

Why people use HTTPclient, when they can use normal HTTP request under ESP8266Wifi.h library?

martinius96 picture martinius96 on 1 Feb 2018

@martinius96 Are you referring the WiFiClient? I could not see any http request under the ESP8266Wifi.h library.

mcanyucel picture mcanyucel on 1 Feb 2018

@mcanyucel Yes WIficlient. What is difference between wificlient and HTTPclient (BasicHttpClient.ino file)? It do the same thing..

martinius96 picture martinius96 on 1 Feb 2018

In a manner of speaking. As far as I have seen WiFiClient has a very broad virtual size_t write(const uint8_t *buf, size_t size); method, therefore you have to generate all the POST headers manually. If you are making a post request of form-url-encoded with a short string, it may not be very hard to do that. But if you are attaching a binary file, the post request file identifiers are not very user friendly. moreover you have to read the file in buffer sized chunks and I am not very proficient with cpp so I keep getting stack errors all over the place. I think I could not find a single example that works on the internet that uploads a file on the SD card to a server using WiFiClient,
HttpClient on the other hand exposes more streamlined methods that accept the post parameters and generates the necessary strings automatically. It also handles buffer operations better than me; it does not flood everywhere with stack overflows. You can use a single line int sendRequest(const char * type, Stream * stream, size_t size = 0); which in my opinion better and more appealing than to have this.
In the end it is "not reinventing the wheel"; the SD card operations or OTA updates can be done manually using basic core libraries, but it is easier to use the already developed "advanced" libraries.

mcanyucel picture mcanyucel on 1 Feb 2018

@martinius96 HTTPClient uses WiFiClient to make it easier to deal especially with http requests and answers.

@TheAustrian @mcanyucel @espmut the fix is now in master. Can you test it ?

d-a-v picture d-a-v on 1 Feb 2018

@d-a-v Thank you very much for your extremely quick response. I am a little bit overwhelmed with some other stuff nowadays but hopefully I will be able to test it within a few days or maybe early next week.

mcanyucel picture mcanyucel on 1 Feb 2018

@mcanyucel @TheAustrian @Humancell could you please confirm the fix? I think this is the only pending for release 2.4.1.

devyte picture devyte on 20 Feb 2018

@devyte I really wish to try but we have deployed all our MCU boards to field. I have ordered several ESP8266s but it will take some time for them to arrive. If I can find a spare around I will try it.

mcanyucel picture mcanyucel on 20 Feb 2018

I'm sorry that I somehow missed your request, and the Github notice back in February!

It was funny, but I ran into this issue again last night, with the payload being too large, and found this thread to check back on the status. I did update to v2.4.2 ... and this does fix the issue that I was seeing! I'm now POSTing bigger payloads just fine!

One other issue that I did see, is that I has to add a line to set:

httpClient->setTimeout(250);

Before reading the response stream. Without this, my code that was running just fine before, (v2.3.0) now seemed to have a default timeout of 4000ms! It would just hang. I added the line above, and now it's fine.

Humancell picture Humancell on 15 Sep 2018

I seem to be suffering a similar problem, however in my case I think it might be related to free heap and fragmentation but have not been able to find any workarounds.
I'm using v2.5.0-beta2 and ArduinoJSON v5.13.4 (this might not be relevant).

So in the code I have arrays with a bunch of data which im trying to send to a backend. Due to the fact that the arrays are of considerable size, im splitting up posts to aprox less then 2kB (It should not be necessary but im running out of heap). I do this by calling this function several times, the code surrounding it retries several times in case of error:

int8_t sendSens(uint16_t arrayStart, uint16_t arrayEnd) {

    if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status
        int httpCode = 0;
        DEBUG_PRINT(F("Free Heap before jsonBuffer: "));
        DEBUG_PRINT(ESP.getFreeHeap());
        DEBUG_PRINT(F(" Fragmentation: "));
        DEBUG_PRINT(ESP.getHeapFragmentation());
        DEBUG_PRINT(F("% MaxBlock: "));
        uint16_t maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINTLN(maxFreeBlock);
        const size_t bufferSize = 2 * JSON_ARRAY_SIZE(arrayEnd - arrayStart + 1)
                + JSON_OBJECT_SIZE(4);
        DEBUG_PRINT(F("bufferSize: "));
        DEBUG_PRINTLN(bufferSize);
        if (maxFreeBlock < bufferSize) {
            return MAX_BLOCK_SIZE_TOO_SMALL; //TOO FRAGMENTED
        }
        DynamicJsonBuffer jsonBuffer(bufferSize);
        JsonObject& root = jsonBuffer.createObject();
        root["cmd"] = "insert";
        root["device"] = DEVICE;
        JsonArray& sens = root.createNestedArray("sens");
        for (int i = arrayStart; i < arrayEnd; i++) {
            sens.add(dataSens[i]);
        }
        JsonArray& sensTime = root.createNestedArray("sensTime");
        for (int i = arrayStart; i < arrayEnd; i++) {
            sensTime.add(dataSensTime[i]);
        }
#ifdef DEBUG_MODE
        root.printTo(Serial);
#endif
        size_t len = root.measureLength() + 1;
        DEBUG_PRINT(F("\njson Length: "));
        DEBUG_PRINTLN(len);
        maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINT(F("Max Block: "));
        DEBUG_PRINTLN(maxFreeBlock);
        if (maxFreeBlock < len) {
            return MAX_BLOCK_SIZE_TOO_SMALL; //TOO FRAGMENTED
        }
        DEBUG_PRINT(F("Free Heap before postRequest malloc: "));
        DEBUG_PRINT(ESP.getFreeHeap());
        DEBUG_PRINT(F(" Fragmentation: "));
        DEBUG_PRINT(ESP.getHeapFragmentation());
        DEBUG_PRINT(F("% MaxBlock: "));
        maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINTLN(maxFreeBlock);
        char *postRequest = (char *) malloc(len);
        root.printTo(postRequest, len);
        DEBUG_PRINT(F("postRequest: "));
        DEBUG_PRINTLN(postRequest);
        DEBUG_PRINT(F("Free Heap before jsonBuffer.clear: "));
        DEBUG_PRINT(ESP.getFreeHeap());
        DEBUG_PRINT(F(" Fragmentation: "));
        DEBUG_PRINT(ESP.getHeapFragmentation());
        DEBUG_PRINT(F("% MaxBlock: "));
        maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINTLN(maxFreeBlock);
        jsonBuffer.clear();
        ESP.wdtFeed();
        DEBUG_PRINT(F("Free Heap before http.begin: "));
        DEBUG_PRINT(ESP.getFreeHeap());
        DEBUG_PRINT(F(" Fragmentation: "));
        DEBUG_PRINT(ESP.getHeapFragmentation());
        DEBUG_PRINT(F("% MaxBlock: "));
        maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINTLN(maxFreeBlock);
        WiFiClient client;
        HTTPClient http;
        http.setTimeout(2000);
        http.begin(F("http://mywebpage.com/dataReciever.php")); //Specify request destination
        http.addHeader(F("Content-Type"), F("application/json")); //Specify content-type header
        DEBUG_PRINT(F("Free Heap before http.POST: "));
        DEBUG_PRINT(ESP.getFreeHeap());
        DEBUG_PRINT(F(" Fragmentation: "));
        DEBUG_PRINT(ESP.getHeapFragmentation());
        DEBUG_PRINT(F("% MaxBlock: "));
        maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINTLN(maxFreeBlock);
        httpCode = http.POST(postRequest); //Send the request
        DEBUG_PRINT(F("Free Heap after http.POST: "));
        DEBUG_PRINT(ESP.getFreeHeap());
        DEBUG_PRINT(F(" Fragmentation: "));
        DEBUG_PRINT(ESP.getHeapFragmentation());
        DEBUG_PRINT(F("% MaxBlock: "));
        maxFreeBlock = ESP.getMaxFreeBlockSize();
        DEBUG_PRINTLN(maxFreeBlock);
        DEBUG_PRINTLN(F("JSON Data Sent"));
        DEBUG_PRINT(F("Response: "));
        DEBUG_PRINTLN(httpCode);
        ESP.wdtFeed();
        if (httpCode == 200) {
            DEBUG_PRINT(F("Response Payload: "));
            DEBUG_PRINTLN(http.getString());
            DEBUG_PRINT(F("Free Heap before HTTP.END: "));
            DEBUG_PRINT(ESP.getFreeHeap());
            DEBUG_PRINT(F(" Fragmentation: "));
            DEBUG_PRINT(ESP.getHeapFragmentation());
            DEBUG_PRINT(F("% MaxBlock: "));
            maxFreeBlock = ESP.getMaxFreeBlockSize();
            DEBUG_PRINTLN(maxFreeBlock);
            http.end();  //Close connection
            DEBUG_PRINT(F("Free Heap before FREE POST REQ: "));
            DEBUG_PRINT(ESP.getFreeHeap());
            DEBUG_PRINT(F(" Fragmentation: "));
            DEBUG_PRINT(ESP.getHeapFragmentation());
            DEBUG_PRINT(F("% MaxBlock: "));
            maxFreeBlock = ESP.getMaxFreeBlockSize();
            DEBUG_PRINTLN(maxFreeBlock);
            free(postRequest);
            DEBUG_PRINT(F("Free Heap before RETURN: "));
            DEBUG_PRINT(ESP.getFreeHeap());
            DEBUG_PRINT(F(" Fragmentation: "));
            DEBUG_PRINT(ESP.getHeapFragmentation());
            DEBUG_PRINT(F("% MaxBlock: "));
            maxFreeBlock = ESP.getMaxFreeBlockSize();
            DEBUG_PRINTLN(maxFreeBlock);
            return JSON_POST_OK;
        }
        DEBUG_PRINT(F("Error to String: "));
        DEBUG_PRINTLN(http.errorToString(httpCode));
        DEBUG_PRINT(F("FAILED: Response Payload: "));
        DEBUG_PRINTLN(http.getString());
        http.end();  //Close connection
        free(postRequest);
        return SERVER_BAD_RESPONSE;
    } else {
        DEBUG_PRINTLN(F("WiFi not connected during temp report"));
        return NOT_CONNECTED_TO_WIFI;
    }
}

Sorry for the long post, most of it is debug output...

Next is the output:

Send sensors from 0 to 99
Free Heap before jsonBuffer: 5776 Fragmentation: 8% MaxBlock: 5344
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
json Length: 1750
Max Block: 2816
Free Heap before postRequest malloc: 3248 Fragmentation: 13% MaxBlock: 2816
postRequest: {"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
Free Heap before jsonBuffer.clear: 1488 Fragmentation: 26% MaxBlock: 1056
Free Heap before http.begin: 4016 Fragmentation: 32% MaxBlock: 2528
Free Heap before http.POST: 3800 Fragmentation: 28% MaxBlock: 2528
Free Heap after http.POST: 3368 Fragmentation: 38% MaxBlock: 1768
JSON Data Sent
Response: -3
Error to String: send payload failed
FAILED: Response Payload: 

Send sensors from 0 to 99
Free Heap before jsonBuffer: 5448 Fragmentation: 32% MaxBlock: 3272
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
json Length: 1750
Max Block: 1768
Free Heap before postRequest malloc: 2920 Fragmentation: 34% MaxBlock: 1768
postRequest: {"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
Free Heap before jsonBuffer.clear: 1160 Fragmentation: 31% MaxBlock: 744
Free Heap before http.begin: 3688 Fragmentation: 11% MaxBlock: 3272
Free Heap before http.POST: 3472 Fragmentation: 6% MaxBlock: 3272
Free Heap after http.POST: 2840 Fragmentation: 28% MaxBlock: 1880
JSON Data Sent
Response: 200
Response Payload: 100 OK
Free Heap before HTTP.END: 3544 Fragmentation: 25% MaxBlock: 2544
Free Heap before FREE POST REQ: 3600 Fragmentation: 6% MaxBlock: 3384
Free Heap before RETURN: 5360 Fragmentation: 29% MaxBlock: 3384

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5584 Fragmentation: 32% MaxBlock: 3384
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368162,1548368162,1548368162,1548368162,1548368162,1548368162]}
json Length: 1750
Max Block: 1768
Free Heap before postRequest malloc: 3056 Fragmentation: 35% MaxBlock: 1768
postRequest: {"cmd":"insert","device":2,"sens":[59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368162,1548368162,1548368162,1548368162,1548368162,1548368162]}
Free Heap before jsonBuffer.clear: 1296 Fragmentation: 30% MaxBlock: 856
Free Heap before http.begin: 3824 Fragmentation: 12% MaxBlock: 3384
Free Heap before http.POST: 3608 Fragmentation: 7% MaxBlock: 3384
Free Heap after http.POST: 3176 Fragmentation: 32% MaxBlock: 1768
JSON Data Sent
Response: -3
Error to String: send payload failed
FAILED: Response Payload: 

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512
WIFI turned OFF
Free Heap: 4992

As you can see from the output, the exact same size of data succeeds only on the second try and eventually aborts due to lack of minimum free heap required for the json parser buffer.
The response of my http.post seems to arbitrarily fragment my memory unnecessarily, I'm not sure if it requires a minimum of heap to operate "normally"...
Until now I had been solving this issue by reducing the packet size, but now it just seems weird for my esp not to being able to send more than 1.5kB of data on one post, splitting it into smaller pieces would take forever to send.

Anyone has this problem? Is this a bug in HTTPClient? Am I asking too much for my esp?
General recommendations are welcome :)

EdoMon93 picture EdoMon93 on 25 Jan 2019

Ok ... this is what I wondered. I actually created a workaround that I'm using successfully, just in case anyone else requires a similar fix. To do it I had to use WifiClient:

#include <ESP8266HTTPClient.h>

String jsonPayload = "{ my big giant 3.5k byte json payload ....}";
int contentLength = jsonPayload.length();

  // create the request and headers
  String request = "POST " + String(endpointURL) + " HTTP/1.1\r\n" +
    "Host: " + String(endpointHost) + "\r\n" + 
    "Accept: application/json" + "\r\n" + 
    "Content-Type: application/json\r\n" +
    "Content-Length: " + String(contentLength) + "\r\n\r\n";

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  connectResult = client.connect(endpointHost, atoi(endpointPort));

  // This will send the request and headers to the server
  client.print(request);

  // now we need to chunk the payload into 1000 byte chunks
  int cIndex;
  for (cIndex = 0; cIndex < contentLength; cIndex = cIndex + 1000) {
    client.print(jsonPayload.substring(cIndex, cIndex+1000));
    //Serial.print(jsonPayload.substring(cIndex, cIndex+1000));
  }
  client.print(jsonPayload.substring(cIndex));
  //Serial.println(jsonPayload.substring(cIndex));

I have a little more response checking and other stuff ... but you get the idea. Using this I am now sending pretty big payloads, and it seems to work fine. The 1000 bytes was just an arbitrary guess ... if I have time I might try to adjust this to fine tune the operation.

Obviously there are limitations in this approach as I can't do SSL easily (or can I?) ... but maybe something like this ought to be rolled into the handling of the payload on the call to http.POST in the HTTPClient?

After spending MANY hours trying to upload ESP32-CAM photos, i found out the problem was only with big photos (e.g. bigger than VGA) and big files (txt) in general, so i finally found the problem was with the POST body in general..
The quote solution works fine with Strings but esp32-cam produce a uint8_t * buf that has to be used differently..
So, the code below that i wrote worked fine with every FRAMESIZE (even UXGA - 1600 x 1200):

camera_fb_t * fb = esp_camera_fb_get();
int imgLenInt = fb->len;
String imgLenStr = (String)imgLenInt;
//headers already POSTed
int howMuch = 0;
int sumHowMuch = 0;
int i = 0;
int myChunkSize = 1300;  //try 1000-2000 values if you want
for (i = 0; i < imgLenInt - myChunkSize; i = i + myChunkSize) {
  howMuch = client.write(fb->buf + i, myChunkSize);
  sumHowMuch += howMuch;
}
int left = imgLenInt % myChunkSize;
howMuch = client.write(fb->buf + (i - myChunkSize), left);
sumHowMuch += howMuch;
Serial.print("you had to send:");
Serial.print(imgLenStr);
Serial.print(", and you send:");
Serial.println(sumHowMuch);`
ligantx picture ligantx on 15 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Marcelphilippeandrade picture Marcelphilippeandrade  Â·  3Comments
treii28 picture treii28  Â·  3Comments
hoacvxd picture hoacvxd  Â·  3Comments
tiestvangool picture tiestvangool  Â·  3Comments
rudydevolder picture rudydevolder  Â·  3Comments