Orm: findBy didn't return back a refresh result

[jerryladoo]

findBy always returns me back previous entity cached in entity map if I call it in an infinity loop. I created two php file, one is called keep_fetching.php, another one is called test.php; and also opened up two terminals, one is for 'keep_fetching.php' and another one is for test.php. I ran keep_fetching.php first and it outputed whatever I have in database as below, which is good

email (fetched by 'find' of export repo): [email protected]
email (fetched by 'findBy' of export repo): [email protected]
email (fetched by 'find' of user repo): [email protected]
email (fetched by 'findBy' of user repo): [email protected]

after that, in another terminal I ran test.php to update user's email to [email protected], and I was expecting to see the outputs like so:

email (fetched by 'find' of export repo): [email protected]
email (fetched by 'findBy' of export repo): [email protected]
email (fetched by 'find' of user repo): [email protected]
email (fetched by 'findBy' of user repo): [email protected]

but unfortunately the outputs were still exactly same as the previous one, no any changes.

In order to figure out what is goning on, I dived into the source code, and found that findBy actually did fetch the fresh result from db, but it didn't update the existing entity cached as the hint by default was set to UnitOfWork::HINT_DEFEREAGERLOAD => true, which further resulted in a false to $overrideLocalValues inside method createEntity in UnitOfWork.php.

And from source code, for findBy to update the existing entity is only through the Query::HINT_REFRESH or clear entire entity map by invoking $em->clear().

Now I a little bit confused about what result findBy should be expected to yield, the cached one or refreshed one? Is there a document that clearly states the relationships between find* methods and Query::HINT_*?

The followings are the two file I was using to do this experiment.

// keep_fetching.php
$kernel = new AppKernel('dev', true);
$kernel->boot();

$em = $kernel->getContainer()->get("doctrine.orm.entity_manager");

while (true) {
    $repo = $em->getRepository(\Acme\ExportBundle\Entity\Export::class);
    $userRepo = $em->getRepository(\Acme\UserBundle\Entity\User::class);

    $export3 = $repo->find(7);                        // <- this returned what expects, as it loads entity from cache first according to doc
    echo "email (fetched by 'find' of export repo): " . $export3->getUser()->getEmail(); echo PHP_EOL;
    $export4 = $repo->findBy(["id" => 7]);      // <- this fetched the fresh result from db but didn't overwrite the existing entity cached in entity map;
    echo "email (fetched by 'findBy' of export repo): " . $export4[0]->getUser()->getEmail(); echo PHP_EOL;

    $user2 = $userRepo->find(6);
    echo "email (fetched by 'find' of user repo): " . $user2->getEmail(); echo PHP_EOL;

    $user3 = $userRepo->findBy(["id" => 6]);
    echo "email (fetched by 'findBy' of user repo): " . $user3[0]->getEmail(); echo PHP_EOL;

    sleep(10);
    echo PHP_EOL;
}

// test.php

$kernel = new AppKernel('dev', true);
$kernel->boot();

$em = $kernel->getContainer()->get("doctrine.orm.entity_manager");

$userRepo = $em->getRepository(\Acme\UserBundle\Entity\User::class);
$user1 = $userRepo->find(6);

$email = "[email protected]";
$em->flush($user1->setEmail($email));
echo "email has been changed to ' . $email . ' with 'flush'";echo PHP_EOL;
die();

[Ocramius]

Closing as invalid.

Specifically, the issue is about a misunderstanding of the behaviour of the UnitOfWork, which is an abstraction around the current open transaction.

Refreshing entities by default would lead to overwritten state of your in-memory entities (which you are working with), whereas what you are doing is executing multiple subsequent reads outside of a serialized transaction (ideal scenario).

The results to be produced by default should always be the in-memory cached ones.

[jerryladoo]

@Ocramius thanks for replying this in such short time. But I still have a question about findBy. If it should always return the cached ones, then why it followed this pattern:

1. execute SQL query to get the latest result from db
2. create entity
   
   
   
   • check to see if current one is cached and load it
   
   

instead of the pattern find uses:

1. try to load from cache
2. load from db if it doesn't hit

?

[chrisdlangton]

@jerryladoo

Reading entity-state;
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/working-with-objects.html#entity-state

and unitofwork-being-out-of-sync;

http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/working-with-objects.html#effects-of-database-and-unitofwork-being-out-of-sync

It is quite clear that both find and findBy when called in a single php process, will;

• On first call to find or findBy return an Entity with data from the database record
• Entity will be UnitOfWork::STATE_MANAGED in both cases
• All further calls to find or findBy in the same php process will return same Entity, with the same data as the first call.

Per transactions-and-concurrency;
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/transactions-and-concurrency.html

@Ocramius I assume calling flush at any point after the first call and any subsequent calls should force Doctrine to fetch fresh data from database, as EntityManager#flush() completes the transaction and therefore the Entity and it's data is released.

Conclusion: for fresh data between calls, call EntityManager#flush() which completes the previous calls open transaction.

[jerryladoo]

@chrisdlangton thanks for bringing up the docs

On first call to find or findBy return an Entity with data from the database record

This is what doc says, and it is a result I am expecting, and it is only one I have question about not others. But the actual result is not always from db(only first time to be specific), it is from cached by default (confirmed by @Ocramius and the source code itself), and @Ocramius also explained why by default it should always return the in-memory cached ones. You can run the two scripts above to see the actual result.

[farazpartoei]

I know its a little bit late! but I used EntityManager:refresh method and issue was recovered

[k00ni]

For documentation purposes on a similar issue, because it might not be a bug, but AFAIK less known behavior.

When running update operations on the same database inside the same script using EntityManager and a DBAL connection, you might get cached results after changes to the database, even if you run EntityManager::flush in between.

Solution: I solved it by calling EntityManager::clear after running an update command with DBAL connection. Unfortunately the "Synchronization with database" doesn't contain any information about that.

In the following some example code to illustrate it:

$user = new User();
$user->setFoo("bar");
$em->persist($user);

$users = $em->getRepository(User::class)->findAll();

// change DB outside of Doctrine
$connection->exec('DELETE FROM user WHERE ...;');

// $em->flush(); would not work here
$em->clear();

// after calling clear(), the $em will use fresh data on the next call

Thank your for raising the issue and give helpful hints.