Arduinojson: Is this message too big? How can I optimize it?

[sblantipodi]

Hi all,
I'm am trying to send a big message to the ESP via MQTT using the mighty ArduinoJSON and PubSubClient.

I need to send a big object like this containing a big array.

{"total":500,"part":3,"stream":[-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16119028,-16711677,-16711677,-16579836,-16579836,-16514043,-16514043,-16579836,-16579837,-16514300,-16579837,-16579837,-16579837,-16382458,-16382457,-16382457,-16382201,-16381943,-16381943,-16381942,-16381942,-16381942,-16250613]}

To do it I have done this:

void callback(char *topic, byte *payload, unsigned int length) {
  DynamicJsonDocument doc(4096);
  deserializeJson(doc, (const byte*) payload, length);
  JsonArray stream = doc["stream"];
}

I even tried the StaticJsonDocument but nothing change.
Object seems to be too big and my ESP8266 reset without giving me any details as soon as I try to transfer it via MQTT.

Array contains 190 values... If I reduce it to 160 it works well.

Is there a way to transfer this array or even bigger by doing some optimizations or by using better Arduino JSON kind?

Thanks!!!

[sblantipodi]

I think that chapter 5.3 of the book (deserializing in chunks) have the answer... trying it right now... but...

// Jump to the first element of the "nodes" array
response.find("nodes:[");

// Repeat for each element of the array
do {
  StaticJsonDocument doc;
  // Deserialize one element of the node array
  deserializeJson(doc, response);
} while (response.findUntil(",", "]"));

I have this callback from pubsubclient:

void callback(char *topic, byte *payload, unsigned int length) {

}

How can I transform my payload in something on where I can call the .find function?
payload.find("nodes:[");
obviously don't work...

[bblanchon]

Hi @sblantipodi,

Unfortunately, I don't have any simple fix for this issue: JSON is just the wrong tool for the job (chapter 1).

I highly recommend sending the integer array as binary (make sure both machines transmit as big-endian); it will be much faster and smaller.
If you need to transmit other data in JSON, use two different MQTT topics.

I don't know why the ESP resets, but I suspect this is a problem related to PubSubClient.

Best Regards,
Benoit