Yii2: BaseActiveRecord::populateWith()

[paulzi]

Suggestion for improvement.

If I already have an array of models, and I want to get all related data for this, I can do this:

$data = Product::find()->all(); // in other method
// ... some code ...
//$data = Product::find()->with('colors')->all(); // i can not do this, because i already have array
$query = new ActiveQuery(Product::className());
$query->findWith(['colors'], $data);

Now relate colors is populated for all items by single query.

My suggestion, make a convenient method for obtaining such data:

/**
 * @param string|array $with
 * @param BaseActiveRecord[] $data
 */
BaseActiveRecord::populateWith($with, $data);

And example of use:

$data = Product::find()->all(); // in other method
// ... some code ...
Product::populateWith('colors', $data);
foreach ($data as $product) {
    echo count($product->colors);
}

See: https://github.com/paulzi/yii2/commit/7173292627f19ffe97763ed60f3680acd7072e15
If interested, I can make the pull-request.

[samdark]

That could be interesting... @yiisoft/core-developers what do you think?

[dynasource]

because Im already confused about the given usecase at first hand, I would not prefer to have this in BaseActiveRecord.

This is about an operation on arrays and I would assume to be in that context, like ie. ArrayHelper or ActiveRecordHelper

[samdark]

@dynasource it's like being able to fill an array of models having an array of data.

[dynasource]

I know. But like you are saying, its about filling an array that is already created, so out of the scope of an AR.

Secondary, you also mentioned to have the intention to clean the inheritance tree with composition. If thats your goal, you cannot accept this implementation.

[paulzi]

@dynasource, this method is closely related to ActiveRecord, ie. to:
1) A set of relations dependeds on called class, see: BaseActiveRecord.php#L1647
2) This is not just an array, it is array of ActiveRecord same type.

[dynasource]

I know.

My point is to put this outside the AR scope as I can think of thousands of methods comparable to this. Do you want a fat AR with many functions that will not often be used, or do we want a lean AR supported with helpers for usecases like this.

[dynasource]

Another thing, this is some kind of an anti pattern. Functionality is added for having 2 queries, while this can also be solved in 1 query.

[paulzi]

Number of queries corresponding to the number of queries by using Product::find()->with('colors')->all();.
You propose to create ActiveRecordHelper?

[dynasource]

from a PHP perspective you are doing two 2 queries, while it can be done with one. This could add confusion for developers using your code in the future. Its no ideal.

From DB perspective, I understand that the current implementation of 'with' generates more than 1 query but Im not 100% sure if that will hold in the future.

About a ActiveRecordHelper, yes, thats sounds better.

[samdark]

The populating itself is usually called "hydration". There could be multiple strategies doing that:

• Just assigning properties as it's done now.
• Passing array to constructor.
• Filling private properties through reflection which is necessary if you have clean objects.

I think it could be a good idea to introduce a separate interface for it and have it as a separate non-static class.

[PowerGamer1]

I have already suggested adding an ability to perform with() on an already existing object here: https://github.com/yiisoft/yii2/issues/11000. Now OP goes even further by wanting to perform with() on _array_ of already existing objects. This is only logical, so to reiterate, the following is needed:

// Can do now (for SINGLE object):
$singleModel1 = Model1::find()->with('model2Relation')->one();
// Need to be able to do:
$singleModel1 = Model1::find()->one();
BaseActiveRecord::populateWith($singleModel1, 'model2Relation');

// Can do now (for ARRAY of objects):
$arrayOfModels1 = Model1::find()->with('model2Relation')->all();
// Need to be able to do:
$arrayOfModels1 = Model1::find()->all();
BaseActiveRecord::populateWith($arrayOfModels1, 'model2Relation');

[PowerGamer1]

@dynasource

from a PHP perspective you are doing two 2 queries, while it can be done with one. This could add confusion for developers using your code in the future. Its no ideal.

I don't really understand what you mean here by "queries from PHP perspective" but I don't see anything in suggested method interface that can cause any confusion here.

From DB perspective, I understand that the current implementation of 'with' generates more than 1 query but Im not 100% sure if that will hold in the future.

Yes, there will always be more than 1 query because if you try doing it with 1 query (join and select data from all joined tables in main query) you will be pulling a lot of duplicate data from database.

About a ActiveRecordHelper, yes, thats sounds better.

This functionality is not about arrays it is about database relations. We already have with() method in ActiveQuery interface and now we want to perform a "delayed" execution of with() so a new method populateWith should be added somewhere to ActiveRecord/ActiveQuery interface too (i.e. no need for separate helpers or trying to generalize this functionality into something more).

[dynasource]

now we want to perform a "delayed" execution of with

I cannot visualize a usecase in which a delayed 'with' is needed. I guess it a matter of architecture. IMHO, one should have 1 place for queries in your architecture. For this reason I think its more 'pollution' than 'population' in BaseActiveRecord.

[PowerGamer1]

I cannot visualize a usecase in which a delayed 'with' is needed. I guess it a matter of architecture. IMHO, one should have 1 place for queries in your architecture.

There are such use cases, I tried to explain (probably not good enough) one here : https://github.com/yiisoft/yii2/issues/11000. And indeed, in that example all queries are executed in a single place (single transaction).

[dynasource]

would you be willing(/able) to formulate a usecase in 1 sentence?

[PowerGamer1]

would you be willing(/able) to formulate a usecase in 1 sentence?

Well, how about 2 sentences: you need the main model always but may or may not need its related models depending on some condition. Both main model and related models must be loaded in the same transaction but related models will be used after transaction completes somewhere else in the code (other function/PHP file which is called/included depending on that condition).

[dynasource]

Well, how about 2 sentences: you need the main model always but may or may not need its related models depending on some condition. Both main model and related models must be loaded in the same transaction but related models will be used after transaction completes somewhere else in the code (other function/PHP file which is called/included depending on that condition).

So it is about performance?:

You are loading everything from the DB in 1 transaction, but keep the populated AR objects as lean as possible, until you'll decide to use the relations?

[paulzi]

@dynasource
I needed this functionality for function populateTree() in my NestedSets behavior. This function gets all descendants of the node, and fills their children relations:
https://github.com/paulzi/yii2-nested-sets/blob/master/NestedSetsBehavior.php#L224
Users want to populate tree with some relations (with()). But descendants can be already populated. Now, to get the related data, I have to re-request them from the database. I do not want to re-request the descendants, I just need to get additional relations for array of models.
If there is described a function ::populateWith(), I could get related data for already populated ->descendants.

[paulzi]

paulzi/yii2-nested-sets#5

[dynasource]

I see. You are using an external library in which population already has been done. Its getting clearer.

However, I see you have created your own fork. Whats your reason for not using:

• $this->owner->getDescendants()->with(['A','B'])->all()

Performance?

[PowerGamer1]

@dynasource

So it is about performance?:

You are loading everything from the DB in 1 transaction, but keep the populated AR objects as lean as possible, until you'll decide to use the relations?

Indeed it is about performance - in particular about not performing extra SELECTs to database if it can be avoided.

[dynasource]

particular about not performing extra SELECTs to database if it can be avoided.

Doesnt this contradict your earlier statement:

Both main model and related models must be loaded in the same transaction

[paulzi]

@dynasource
If I execute $this->owner->getDescendants()->with(['A','B'])->all(), it will anyway result in a 3 database queries:

SELECT * FROM main WHERE id IN (1, 2, 3)
SELECT * FROM A WHERE main_id IN (1, 2, 3)
SELECT * FROM B WHERE main_id IN (1, 2, 3)

Also related data is not written into the internal variable _related, because you are call direct method, not magic __get:
https://github.com/paulzi/yii2/blob/master/framework/db/BaseActiveRecord.php#L254

If I execute for already populated descenants:

$nodes = $this->owner->descentants;
$this->owner->populateWith(['A', 'B'], $nodes);

it will result in a 2 database queries:

SELECT * FROM A WHERE main_id IN (1, 2, 3)
SELECT * FROM B WHERE main_id IN (1, 2, 3)

Relation descendants is defined in the same repository:
https://github.com/paulzi/yii2-nested-sets/blob/master/NestedSetsBehavior.php#L143

But descendants can be already populated by user before run populateTree():

$amount = count($node->descendant);
$node->populateTree();
// out tree

This code will lead to a double-execute the query SELECT * FROM main WHERE id IN (1, 2, 3).

[PowerGamer1]

@dynasource

Doesnt this contradict your earlier statement:

Nope, it doesn't because as I said related models are needed not always but depending on condition. If I was not worrying about performance I would always load related models along with main model using with():

// begin transaction
$mainModel = MainModel::find()->with('relations')->one();
// end transaction
if(isConditionTrue($mainModel))
    call_func_that_uses_related_models;

But if that condition is false I want to avoid extra SELECTs for related models:

// begin transaction
$mainModel = MainModel::find()->one();
if(isConditionTrue($mainModel))
    BaseActiveRecord::populateWith($mainModel, 'relations');
// end transaction
if(isConditionTrue($mainModel))
    call_func_that_uses_related_models;

[dynasource]

Thank you both for your examples. So the main purpose is performance improvement. I can only favor this ;)

[PowerGamer1]

Suggested static method BaseActiveRecord::populateWith() should have a non-static shortcut for when a single object needs to be populated with some relations, analogous to non-static \yii\base\Model::validate() and static \yii\base\Model::validateMultiple().

BaseActiveRecord::populateMultipleWith($manyModels, 'relations'); // static
$singleModel->populateWith('relations'); // non-static
// analogous to:
\yii\base\Model::validateMultiple($manyModels); // static
$singleModel->validate(); // non-static

[Renkas]

Any news on this functionality?

I was searching for this functionality from docs and codebase and to my surprise found that it's not currently possible.

I'll leave a description for my usecase here for future reference:
```
// Array of freshly initiated Product models - nothing loaded from DB
$products = [new Product(), new Product(), new Product(), ...];

// Load data from form submission
Model::loadMultiple($products, Yii::$app->request->post(), 'Product');
```
Now I have many relations in Product model that I want to use for diffrent purposes.

• company
• category
• ... many more

Form data includes company_id, category_id etc that make it possible to load related info through relation.
But the problem is that if I have 50 products then there will be 50 separate queries for each relation.
Eager loading (Late eager loading - maybe a better name for this) would help a lot here.

[cebe]

I have started experimenting with an implementation of a model collection: https://github.com/yiisoft/yii2-model-collection

Such functionality could be implemented there. eager loading of relational data is implemented like this: https://github.com/yiisoft/yii2-model-collection/tree/master/docs/guide#lazy-find-with

I am interested in your feedback about it.

[Renkas]

For my usecase the example in first post actually works like a charm

$query = new ActiveQuery(Product::className());
$query->findWith(['company', 'category'], $products);

And the suggested shortcut method would suit very well.

/**
 * @param string|array $with
 * @param BaseActiveRecord[] $data
 */
BaseActiveRecord::populateWith($with, $data);

[cebe]

why do you need a shortcut method for something that is two lines of code?

[Renkas]

The possibility to use it like that was not clear at all even after reading docs or looking through methods in source. I only realized it can be done like that after stumbling on this issue here.

As I now know how to handle this situation it actually is a nonissue for me. But for people in the future to get to that point it would be nice to have a specific method for that or maybe document it somehow.

Just for reference - I searched for this functionality with queries like "late eager loading", "eager loading after find". So if there will be any addon to documentation then maybe use these keywords.