Mongodb-odm: Can not clear document manager cache when using readonly option

[douglasjam]

Feature Request

| Q | A
|------------ | ------
| New Feature | yes
| RFC | yes
| BC Break | no

Summary

I have some routines where I have to iterate over 3 millions objects.

Because of this use a cursor, where I iterate through mongo collection using doctrine with hydration true and partial objects. And for every X elements retrieved we clean the document manager in order to avoid memory going up and up.

One day we found a bug in our code, where objects were being updated in the background, and when calling flush in another collection, the collection iterated was being changed, due to a mistake made with default value in the class definition.

Therefore we decided to add readOnly in our queries for security (we fixed the original bug). But, by doing this, the clear document manager doesn't work anymore, since the documents are not registered in the UnitOfWork. So our memory grow and grow until it throws an OutOfMemoryException.

We've searched everywhere, but haven't find a way to clear the doctrine cache.

$query = $this->repository->createQueryBuilder()
            ->readOnly()
            ->getQuery();

/** @var Cursor $cursor */
$cursor = $query->execute();

$cursor->batchSize($batchSize);

$iterator = 0;
foreach($cursor as $product){
    $i++;
    // ... operations within the product ...

    if($i % $batchSize === 0){
        $this->repository->clear(); // DOES NOT WORK WHEN USING READ ONLY
    }
}

I'd like to know if it's possible to somehow clear this cache made by doctrine as it works for notReadOnly objects.

[Steveb-p]

readOnly should cause objects to be literally ignored by DocumentManager and UnitOfWork. Since they don't hold references to objects, they should be automatically garbage collected unless you're holding references elsewhere.

[douglasjam]

Indeed, but the garbage collector won't work for me since, it executes just when the iteration ends. Would be cool to somehow force the clean of doctrine caching.

[Steveb-p]

@douglasjam You're using ODM v1.3 (with mongo-php-adapter) or the upcoming v2.0?

I'm not sure but maybe the underlying library and/or cursor object is storing references. Or maybe it's something inside your foreach loop that keeps refences - because I've been using readOnly for the same purpose and I had constant memory usage.

You're using embedded documents maybe? What's your $batchSize variable value?

Otherwise, check if UoW or DocumentManager contain references to any other documents. Try the 2.0 branch if you can to see if the issue persists there (since 1.3 is basically starting to become legacy and might not receive a fix, if it's a bug).

[malarzm]

Indeed, the garbage collector won't work for me since, it executes just when the iteration ends.

You can put some of your logic to a private method, calling it should force GC to run after the method is finished.

since 1.3 is basically starting to become legacy and might not receive a fix, if it's a bug

FWIW we will provide bugfixes for some time after releasing 2.0 ;) It wouldn't be fair not to given amount of changes happening in 2.0

[virgiliolino]

@douglasjam I investigated the bug and found the solution thanks also to the suggestion from @malarzm . Short story: call clear on the documentManager without any parameter and unset the embedded properties on your document.
Long story: I recreated the case and noticed that embeddeddocumentRegistry always increase. This contains the references to our embedded documents. Normally the clear should remove this, but debugging their clear method I found a bug, clear together with readonly keep forever the references to embeddeddocumentRegistry. But, if you call just clear without any document as parameter, it actually removes everything the trick is that doctrine just remove the objects. But it will still not be enough: php garbage collector is not able to remove the embedded documents because doctrine was keeping reference to those. so I unset the embedded documents from the entity with an unset.

[douglasjam]

Thanks @virgiliolino. The global document manager cleaning worked.
Should the entity repository be responsible for cleaning readonly objects and its embedded ones? If yes, it could be interpreted as a bug, and the same happens in v2.