Rapidjson: Memory leak - Parse

[acachy]

I use rapidjson in std::thread-based socket-server, it simply decodes requests:

rapidjson::Document parsedQuery;
char query[QUERY_BUFFER_SIZE];
...
recv(this->socket, query, QUERY_BUFFER_SIZE, 0);
this->parsedQuery.Parse(query);

In this way memory grows constantly. If I use pointer for Document and new\delete each iteration - no problem.
However, it is too simple example to have real bugs and there is should be some my mistake.
SetObject()\Swap cleaning before Parse and alternative parse-methods makes no difference.

• Debian 8, g++ 7.3, current master #2bbd33b

[miloyip]

If you reuse the same Document object to parse JSON, you can swap it with a new Document when you can ensure the existing data are not referenced by others. E.g.,

~cpp
Document d;
d.Parse(...);
document.Swap(d);
~

[ArArgyridis]

@miloyip I am trying to do a similar thing, however the memory leak still exists.
Here is my example code (tmpRes is an array of json strings):

map> myMap;
for ( size_t k = 0; k < 1000; k++) {
for (size_t i = 0; i < tmpRes.size(); i++) {
rapidjson::Document d;
d.Parse(tmpRes[i][1].c_str());
myMap["id"][i].Swap(d);
}
}

[miloyip]

@ArArgyridis I think this code fragment should not cause any "memory leak".

[petr-shumilov]

Hi, @miloyip
I have the same problem as @ArArgyridis.
As proof, this is a log of Valgrind memcheck util:

==1497== 458,920 (65,560 direct, 393,360 indirect) bytes in 1 blocks are definitely lost in loss record 1,013 of 1,013
==1497==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==1497==    by 0x4B9893: rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::EndObject(unsigned int) (in /collector/build/bin/Collector)
==1497==    by 0x4B82DE: void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) (in /collector/build/bin/Collector)
==1497==    by 0x4B845B: void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseArray<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) (in /collector/build/bin/Collector)
==1497==    by 0x4B823E: void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) (in /collector/build/bin/Collector)
==1497==    by 0x4B823E: void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) (in /collector/build/bin/Collector)
==1497==    by 0x4B7CCD: rapidjson::ParseResult rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::Parse<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) (in /collector/build/bin/Collector)
==1497==    by 0x4B7B7B: rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>& rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::ParseStream<0u, rapidjson::UTF8<char>, rapidjson::GenericStringStream<rapidjson::UTF8<char> > >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&) (in /collector/build/bin/Collector)

[ehasting]

I think i might see some of the same issue, however i am struggeling getting the same valgrind error - i will try to isolate the software part, so i can run it more focused.

[eyjian]

I have the same problem. Can Rapidjson optimize the Parse() function to fix MLK?

[acachy]

I've tried several ways, included Swap() and all caused leak for me

Finally I've go simply with "this->parsedQuery = new rapidjson::Document; ... delete this->parsedQuery;" on each iteration

But probable performance loss is still eating my brain each night

[pah]

By default, RapidJSON uses a MemoryPoolAllocator tied to the Document, which will never release memory until the owning Document is destroyed. The swap trick will achieve that "destruction" by swapping a document instance with a temporary:

  Document().Swap( *this->parsedQuery );  // <-- swap with temporary to wipe out memory pool

If you want to avoid dealing with memory pool lifetimes, simply use the CrtAllocator instead:

using JsonDocument = rapidjson::Document<rapidjson::UTF8<>, rapidjson::CrtAllocator>;

[MikaGuChina]

Anyone help?
I've tried : Value(kObjectType).Swap(document);
it sitll leak

[coffeeaddict19]

By default, RapidJSON uses a MemoryPoolAllocator tied to the Document, which will never release memory until the owning Document is destroyed. The swap trick will achieve that "destruction" by swapping a document instance with a temporary:

  Document().Swap( *this->parsedQuery );  // <-- swap with temporary to wipe out memory pool

If you want to avoid dealing with memory pool lifetimes, simply use the CrtAllocator instead:

using JsonDocument = rapidjson::Document<rapidjson::UTF8<>, rapidjson::CrtAllocator>;

After profiling my program with massif from the valgrind suite I realized something with rapidjson or how I was using it was the source of my memory accumulation problem. This comment led me to switching over to rapidjson::CrtAllocator, like so:

rapidjson::GenericDocument

Which solved my problem. Thanks.