Jmsserializerbundle: JMS Serializer perfomance issues with more thant 10000 entries

[malcomhamelin]

Currently I'm building a PHP command that can update my ElasticSearch indices.

But, a big thing I've noticed is that serializing entities when my array holds more than 10000 of them is taking way too much time. I thought it would be linear, but either 6 or 9k entities takes like a minute (not much difference between 6 or 9k), but when you go past 10k, it just slows down to the point of taking up to 10 minutes.

...
                // we iterate on the documents previously requested to the sql database
                foreach($entities as $index_name => $entity_array) {
                    $underscoreClassName = $this->toUnderscore($index_name); // elasticsearch understands underscored names
                    $camelcaseClassName = $this->toCamelCase($index_name); // sql understands camelcase names

                    // we get the serialization groups for each index from the config file
                    $groups = $indexesInfos[$underscoreClassName]['types'][$underscoreClassName]['serializer']['groups']; 

                    foreach($entity_array as $entity) {
                        // each entity is serialized as a json array
                        $data = $this->serializer->serialize($entity, 'json', SerializationContext::create()->setGroups($groups));
                        // each serialized entity as json is converted as an Elastica document
                        $documents[$index_name][] = new \Elastica\Document($entityToFind[$index_name][$entity->getId()], $data);
                    }
                }
...

There's a whole class around that but that's what is taking the most of the time.

I can get that serialiazing is a heavy operation and that it takes time, but why is there next to no difference between 6, 7, 8 or 9k, but when above 10k entities it juste takes a lot of time ?

PS : for reference, I've asked the same thing on StackOverflow.

[goetas]

The code you have posted mentions "entities", are you using doctrine?

If you are using doctrine, I do not see any memory cleanup in the loops. This means that the memory goes up on each iteration since doctrine has to instantiate all the visited objects, thus inevitably you will see slowdowns.

[goetas]

Here some info on how to clean up memory in doctrine when dealing with big datasets https://www.doctrine-project.org/projects/doctrine-orm/en/2.6/reference/batch-processing.html