Orm: [Question] Why is memory still increasing by using EM flush ?

[poolerMF]

example:

echo "first :".memory_get_usage(true)."<br>";

$em = $this->getDoctrine();

for($y = 0;$y<5;$y++) {
    $user = $em->getRepository(User::class)->find(1);

    for($x = 0;$x<500+$y;$x++) {
        $token = new CentralToken();
        $token->setType('test');
        $token->setToken(StringUtils::getRandomString(10));
        $token->setUser($user);

        $em->persist($token);
    }
    $em->flush();
    $em->clear();
    var_dump(memory_get_usage(true));
}

echo "<br>last :".memory_get_usage(true);

result:

first :2097152
int(4194304) int(4194304) int(4194304) int(6291456) int(8388608) 
last :8388608

and then I refresh page, result:

first :2097152
int(2097152) int(2097152) int(2097152) int(2097152) int(4194304) 
last :4194304

after X refreshes in a row, result will be:

first :2097152
int(2097152) int(2097152) int(2097152) int(2097152) int(2097152) 
last :2097152

then ... I'm waiting X second and making another refresh:

first :2097152
int(8388608) int(8388608) int(8388608) int(10485760) int(12582912) 
last :12582912

why is it using different memory every first time ?
when I'm not making flush, memory is STILL SAME -> 2097152
is it bug of UnitOfWork .. or something with apache ? php ? db ?

I have never ending loop for selecting & creating entities ... after 1 hour apache process is using more than 500MB of memory

[jvasseur]

My guess would be that you have some kind of logger active that keep in memory queries executed by flush.

[poolerMF]

@jvasseur no I don't have
if I had, memory will be incerased EVERY time ... you see, that memory is not increased every time

I tested it in another NEW project and result is same

[Ocramius]

Can you try using https://github.com/Ocramius/DoctrineBatchUtils and see if
the problem still occurs for that one loop?

Do you have any listeners or loggers attached to the connection or the
entitymanager?

On 26 Jan 2018 09:12, "poolerMF" [e-mail] wrote:

@jvasseur https://github.com/jvasseur no I don't have
if I had, memory will be incerased EVERY time ... you see, that memory is
not increased every time

I tested it in another NEW project and result is same

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/doctrine/doctrine2/issues/7009#issuecomment-360712299,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJakOBLufk2hj8iUCPF7fvqLbfP9LUdks5tOYkIgaJpZM4Rs6yY
.

[coudenysj]

@poolerMF It is a pretty basic approach, but you could dump the entity manager in a file a couple of times, and use diff to figure out what's going on: https://blog.jachim.be/2013/01/finding-memory-leaks-in-php-objects/.

[poolerMF]

I tried to test it in NEW CLEAN project as I wrote ... and result was same

@Ocramius SimpleBatchIteratorAggregate is doing same thing as me in script ... I'm calling flush and clear after 500 iterations ... I tried It with transaction, but result is same

@coudenysj it's not possible to dump EM or UnitOfWork - apache will get out of memory

• this occured every time ... even without selecting/persisting something

I need better tool for measuring it ... any ideas ?

[Ocramius]

Dumping the keys of the UnitOfWork#identityMap would be sufficient - no need to dump the contents.

[poolerMF]

I tried and identityMap is empty after every EM->clear

I found DebugUnitOfWorkListener .. tried it and result after every flush:
Flush Operation [] - Empty identity map.

but it's OK that identity map is empty, it's normal behaviour after calling EM->clear

[Ocramius]

@poolerMF are you running the latest ORM version? Any other properties we can check?

What about the listeners? Anything registered?

[coudenysj]

@poolerMF Can you describe your setup a bit more? It is kind of strange that your memory keeps going up when you're doing page refreshes (as that would be new PHP runs every time).

[poolerMF]

@coudenysj look at my first post ...

• for testing you can use simple entity with 1-3 properties (less variables = you need to run loop more times .. call inner loop 700 ... 1000 times)
• anything special
• everything was default in new clean symfony project (v 3.3)

[coudenysj]

@poolerMF Can you push the Symfony project to github somewhere? By using Symfony, you have SQL logging, etc... in the Symfony profiler.

[poolerMF]

class DDC7009Test extends \Doctrine\Tests\OrmFunctionalTestCase
{
    public function setUp()
    {
        parent::setUp();
        try {
            $this->schemaTool->createSchema(
                [
                    $this->em->getClassMetadata(DDC7009Anything::class),
                ]
            );
        } catch (\Exception $e) {
        }
    }
    public function testIssue()
    {
        $startMemory = memory_get_usage(true);

        for($y = 0;$y<10;$y++) {
            for($x = 0;$x<1000+$y;$x++) {
                $object = new DDC7009Anything();
                $object->setText(uniqid());
                $this->em->persist($object);
            }
            $this->em->flush();
            $this->em->clear();
        }

        $endMemory = memory_get_usage(true);

        self::assertSame($startMemory, $endMemory);
    }
}
/**
 * @ORM\Entity
 */
class DDC7009Anything
{
    /**
     * @ORM\Id
     * @ORM\GeneratedValue
     * @ORM\Column(type="integer")
     */
    public $id;

    /**
     * @ORM\Column(name="_text", type="string", length=23)
     */
    public $text;
}

@coudenysj is it enough for you ?

or you can dump memory in every loop (1-10)

[coudenysj]

@poolerMF I've add the code in a branch on my fork.

First of all, the assertSame can never be the same, because you declare loop variables (which impact the memory usage).

The reason why the memory is increasing, is the SQL Logger in the tests. If you disabled it (like I did in the setUp), you don't get the extra memory usage.

[poolerMF]

@coudenysj I didn't test that code, I made it quickly

finnaly I find out, that memory increasing was maybe becouse of SQL logger ... but still it has weird behaviour if using SQL logger (memory is still same ... it should increase every time)

[Ocramius]

Closing here then :)