Arduinojson: StaticJsonBuffer or Dynamic on Arduino/Esp8266

[dmarkey]

I'm seeing strange behaviour on my esp8266 over time. Multiple JSON buffer being mangled etc. What is the recommended way to manage memory on the esp8266 with ArduinoJson?

[bblanchon]

My recommendation would be to use a DynamicJsonBuffer with an initial size that correspond to your needs:

DynamicJsonBuffer jsonBuffer(4000);

However, I don't think the issue you're having will be fixed like this; it looks more complicated.

[sticilface]

I'm also wondering this. I'm seeing some fairly heavy heap usage adding doubles to an array. For example adding 300 doubles to an array (which has a memory footprint of 1.5k) will eat up nearly 20k of heap! Which seams like a lot. Does giving a size to the dynamic buffer help that?

Could u elaborate a bit more so I understand what's going on?

[bblanchon]

First of all, in version 5.0.5, I changed the storage type from double to float because the overhead wasn't worth it. In older versions, the memory consumption was nearly twice bigger on ESP8266.
You can switch back to the double storage by editing ArduinoJson/Internals/JsonFloat.hpp.

Second remark, in version 5.0.6, I added the ability to set the initial size of DynamicJsonBuffer and changed the default from 32 to 256.

From there, I'm going to assume that you're using Arduino 5.0.6+ and kept the default float storage.

A JsonArray is just a linked list of JsonVariant.
So on an ESP8266:

1. A float is 4 bytes, so 300 of them is 1.2k
2. A JsonVariant is 8 bytes, so 300 of them is 2.4k
3. A node of the linked list is 12 bytes, so 300 of them is 3.6k
4. A JsonArray has an initial size of 8 bytes.

So the total is 3608 bytes, which can be confirmed by:

static_assert(JSON_ARRAY_SIZE(300) == 3608, "@bblanchon is full of sh*t");

Now, this is too big for the stack of the ESP8266, so using a DynamicJsonBuffer is appropriate.

DynamicJsonBuffer allocates blocks of memory using malloc(); each block is like a StaticJsonBuffer.
As soon as it doesn't find enough memory in a block, it allocates a new one twice as big (like std::vector).

In our case, it will allocate four blocks:

1. 256 bytes + 12 bytes
2. 512 bytes + 12 bytes
3. 1024 bytes + 12 bytes
4. 2048 bytes + 12 bytes

Total: 3888 bytes

But the actual memory usage will be slightly higher due to the plumbing of malloc() and free().
Also it introduces memory fragmentation that reduces the memory you can actually use.

Specifying the right initial size will allow DynamicJsonBuffer to make only one malloc():

The benefits are:

1. Better performance because malloc() is called once
2. Less overhead, because 3888-3608=280 bytes are saved
3. No memory fragmentation
4. Mininal overhead due to malloc()/free() internal plumbing.

You can see that, in these conditions, the following:

DynamicJsonBuffer jsonBuffer(JSON_ARRAY_SIZE(300));

is almost identical to

JsonBuffer* jsonBuffer = new StaticJsonBuffer<JSON_ARRAY_SIZE(300)>();

That's why I'm now recommending using DynamicJsonBuffer with an initial size.

As a conclusion, let's compare with ArduinoJson 5.0.4:.

1. JSON_ARRAY_SIZE(300) was 7208
2. 8 calls to malloc() were made, for a total of 8256 bytes

If we include the overhead and the fragmentation, it's getting close to the 20K you're observing.

[sticilface]

Thanks for the uber detailed response! much appreciated, i just felt like a learnt a lot.

1. 1st thing i learnt update the lib. i was behind on 5.0.4, common sense, but something i forget often. Some libs are quite static, but you make a lot of improvements to this one, so good work.
2. just upgrading sorted the issue.
3. Thank you very much!!

[sticilface]

Just thought of an extra question.

When allocating memory for the dynamic buffer, is there any way to know that you've got it right!

if my understanding is correct, using dynamic is recommended with the ESP as static might get too big to be allocated. but... say i calculate i need X.. based on using your macros. if i've done too little, then i assume the dynamic will just allocate more. what is your advice? is there a way to check?

my heap usage has now shrunk a lot, instead of 9k free, i have 21k! and the jsonbuffer is 5k. (there is other stuff there not just the array i mentioned)

[bblanchon]

You can call JsonBuffer::size() to know how much bytes have been allocated in the buffer.
So that's a very good way to check if the size you gave to the constructor is appropriate.

[sticilface]

That works thanks.

[Antuanett]

I am working with the ArduinoJson library, I use DynamicJsonBuffer but when I write in the console several times it does not work. Can someone help me?
In addition I display the size of the buffer each time and it arrives at 0 and it does not work.
Thanks for replying.
I leave you my code

void loop() {
// put your main code here, to run repeatedly:
while (!Serial.available());
while (Serial.available())
{
String json = Serial.readString();

    DynamicJsonBuffer  jsonBuffer;
    JsonObject& root = jsonBuffer.parseObject(json);
    Serial.println(jsonBuffer.size());
    if (root.success()) 
    {
      parserJson(root); //Here, I take the json to work with him
    }
    else
    {

      return;
    }
  }

}

[bblanchon]

@Antuanett, I suggest that you let parseObject() read directly from Serial:

while (!Serial.available()) {}

DynamicJsonBuffer jsonBuffer;
JsonObject& root = jsonBuffer.parseObject(Serial);

if (root.success()) {
  parserJson(root); //Here, I take the json to work with him
}

[Antuanett]

Thank you, I had tried like that and it works