Mongodb-odm: Memory leaking when used query builder in the loop

[TheHett]

Bug Report

I try to use ODM in daemon, but have problem with memory consumption.

| Q | A
|------------ | ------
| Version | 2.0.1

I created minimal example that leads to an error:

        ini_set('memory_limit', '16m');
        gc_enable();
        $i = 0;
        while (true) {
            /** @var File $file */
            $file = $this->dm
                ->createQueryBuilder(File::class)
                ->getQuery()
                ->getSingleResult();

            $this->dm->flush();
            $this->dm->clear();
            unset($file);
            gc_collect_cycles();
            echo $i++, PHP_EOL;
        }

The code crushes regardless of the size allocated memory after some iterations (the more memory, the more iterations)

PHP Fatal error:  Allowed memory size of 16777216 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php on line 91
PHP Stack trace:
PHP   1. {main}() /usr/local/app/yii:0
PHP   2. yii\console\Application->run() /usr/local/app/yii:32
PHP   3. yii\console\Application->handleRequest($request = *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/base/Application.php:386
PHP   4. yii\console\Application->runAction($route = *uninitialized*, $params = *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/console/Application.php:147
PHP   5. yii\console\Application->runAction($route = *uninitialized*, $params = *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/console/Application.php:180
PHP   6. app\commands\TestController->runAction($id = *uninitialized*, $params = *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/base/Module.php:528
PHP   7. app\commands\TestController->runAction($id = *uninitialized*, $params = *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/console/Controller.php:164
PHP   8. yii\base\InlineAction->runWithParams($params = *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/base/Controller.php:157
PHP   9. call_user_func_array:{/usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57}(*uninitialized*, *uninitialized*) /usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP  10. app\commands\TestController->actionIndex() /usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP  11. Doctrine\ODM\MongoDB\Query\Query->getSingleResult() /usr/local/app/commands/TestController.php:39
PHP  12. Doctrine\ODM\MongoDB\Query\Query->getIterator() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Query/Query.php:277
PHP  13. Doctrine\ODM\MongoDB\Query\Query->execute() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Query/Query.php:249
PHP  14. Doctrine\ODM\MongoDB\Query\Query->runQuery() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Query/Query.php:181
PHP  15. Doctrine\ODM\MongoDB\Query\Query->makeIterator($cursor = *uninitialized*) /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Query/Query.php:418
PHP  16. Doctrine\ODM\MongoDB\Iterator\CachingIterator->__construct($iterator = *uninitialized*) /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Query/Query.php:356
PHP  17. Doctrine\ODM\MongoDB\Iterator\CachingIterator->storeCurrentItem() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:47
PHP  18. Generator->key() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:128
PHP  19. Doctrine\ODM\MongoDB\Iterator\CachingIterator->wrapTraversable($traversable = *uninitialized*) /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:128
PHP  20. Doctrine\ODM\MongoDB\Iterator\HydratingIterator->rewind() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:139
PHP  21. Generator->rewind() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:73
PHP  22. Doctrine\ODM\MongoDB\Iterator\HydratingIterator->wrapTraversable($traversable = *uninitialized*) /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:73
2019-10-29 07:44:48 [-][-][-][error][yii\base\ErrorException:1] yii\base\ErrorException: Allowed memory size of 16777216 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:91

[malarzm]

What is File? Is it any document?

[TheHett]

Hi, yes
but collection must not be empty.

<?php

declare(strict_types=1);

namespace app\doctrine\entities;

use Doctrine\ODM\MongoDB\Mapping\Annotations as ODM;

/**
 * @ODM\Document(collection="file")
 */
class File
{
    /**
     * @var string|null
     * @ODM\Id()
     */
    private $id;

    /**
     * @return string|null
     */
    public function getId(): ?string
    {
        return $this->id;
    }
}

[alcaeus]

Thanks for the report! It took me a few hours to figure out the root cause, but apparently the generators we use to wrap MongoDB cursor objects stick around. In my tests, removing the generator in __destruct removed the memory leak, but it'd be good to get confirmation for this. @TheHett would you be able to test the code in the PR to see if it fixes your problems?

[TheHett]

@alcaeus Now works fine, thanks!

[alcaeus]

Thanks for confirming! I'll try to get a patch release out today!