Objection.js: Default eager loading for Model class

[zuck]

Let's imagine we have two models: Person and Pet, where each Person could have many Pets.

Now let's say Person needs a virtual attribute, e.g. petAddicted, which counts the number of related Pets to measure the love of each Person for the animals in general:

Person.prototype.petAddicted = function() {
  return this.pets.length;
}

Person.virtualAttributes = ['petAddicted'];

This virtual attribute is used everywhere: this means I have to eager load the Pet relation every time a Person is fetched, both in main or related queries and where the instance is translated to JSON format.

How this goal could be achieved? Is it currently possible?

My idea is to associate a custom QueryBuilder to the Person model but I can't figure out where exactly I should implement the "default eager loading" behavior...maybe on forClass method?

I open this issue to keep track of this feature which, IMHO, could be integrated as "native" in the future (e.g. adding a new static property to Model class, like defaultEager, to define default eager loading behavior in default QueryBuilders):

Person.defaultEager = ['pets'];

Pet
.query()
.findById(1)
.eager('owner')
.then(function(bobby) {
  console.log("Pet addiction level: " + bobby.owner.petAddicted);
});

// Pet addiction level: 5

[zuck]

I tried to implement it creating a custom QueryBuilder class and overriding the forClass method, but it doesn't work as expected:

// Create a custom QueryBuilder for current Model.
function MyQueryBuilder() {
  objection.QueryBuilder.apply(this, arguments);
}

// Basic ES6 compatible prototypal inheritance.
objection.QueryBuilder.extend(MyQueryBuilder);

MyQueryBuilder.forClass = function(modelClass) {
  var qb = objection.QueryBuilder.forClass.call(this, modelClass);
  return qb.eager(modelClass.defaultEager || "");
}

model.QueryBuilder = MyQueryBuilder;
model.RelatedQueryBuilder = MyQueryBuilder;

All my models use this custom MyQueryBuilder and the trick seems to work when model instances are fetched directly, but not when they are fetched as part of a relation (related model instances are returned, but no eager loading is performed on them).

[fl0w]

There's a number of ways to attack this I think.
If I understand your problem correctly, I'd personally try to do a static property in Model definition, e.g.:

// es6
class MyModel {
  static get defaultEager () {
    return '[eagerRelation]'
  }
}

Of the top of my head, the way to accomplish this would be to perhaps overload .then in your CustomQueryBuilder check for defaultEager in modelClass.

If you would want to control this, I'd work off of context and settings via it.

Edit _removed the use of word "related" as its confusing in this context._

[koskimas]

@zuck It's not that forClass doesn't work correctly, but the eager method doesn't work as you expect. If eager method is called multiple times, the later call overwrites the previous one. You call eager with the default eager, but it gets replaced by another call later.

The default WHERE IN based eager loading algorithm calls eager recursively. For example expression children.pets creates a query that executes the children fetch and the result of that query is passed as an input to the next level query with eager expression pets.

You need to reimplement eager to merge the expressions passed in.

[zuck]

@fl0w Yes, adding the defaultEager property to Model is the trivial part, the problem is with the QueryBuilder implementation, as you noticed in the last part of your comment.

@koskimas so, the default eager implementation overrides previous expressions with the latest: just to be sure, is the following behavior correct (with Person, Pet and Race models)?

// Person -> hasMany -> Pet -> belongsToOne -> Race

Person.defaultEager = '[pets]';

Pet.defaultEager = '[race]'; // <- Overridden during final query...

Person
  .query()  // Here the defaultEager is performed in custom QueryBuilder...
  .findById(1)
  .then(function(person) {
    console.log(JSON.stringify(person));
  });

/* Output:

{ 
  ...
  pets: [{
    ...
    // no 'race' is returned...
  }]
}

*/

[koskimas]

If you didn't write any code that starts the race query, then yes that is correct. Why would objection start a race eager query for pets if all you've done is described in the second post?

[koskimas]

Maybe you can traverse the relation hierarchy in the forModel method and create the whole eager expression in there. That is not easy because the eager expression parsing code is not documented. You can access the RelationExpression class that handles all the parsing and stuff through require('objection').RelationExpression but the class is not public API and can change in the future. Maybe I could make that API public though...

[zuck]

Yes, I was exactly looking at RelationExpression to accomplish my goal, with something like:

MyQueryBuilder.prototype.eager = function(exp, filters) {
  var _eagerExp = exp || null;
  var _eagerFilters = filters || null;

  if (_.isString(_eagerExp)) {
    _eagerExp = objection.RelationExpression.parse(_eagerExp);
  }

  _eagerExp = _.assignIn(
    this._eagerExpression,
    _eagerExp
  );

  if (_.isObject(_eagerFilters)) {
    _eagerFilters = _.assignIn(
      this._eagerExpression.filters,
      _eagerFilters
    );
  }

  return objection.QueryBuilder.prototype.eager.call(this, _eagerExp, _eagerFilters);
}

Where, basically, I'm trying to merge eager expressions instead of replacing them.

The issues at the moment are:

1. The merging algorithm is not correct, because we need to take in consideration the name of relation in eager loaded fields (e.g. in the example above we have to append pets to each entry of Pet.defaultEager property and pets.race to each of Race.defaultEager attributes). This is the (quite) easy issue to solve.
2. @koskimas I don't understand exactly how relations are traversed and fetched: if I try to execute this code, different eager operations are performed for the various models in the tree, so where are they actually merged/overridden? Could you please explain me some details of current eager loading algorithm?

I mean, with the query above:

  .query( // with modified forClass )
  .findById(1)
  .then( // .... );

where is the loading performed and eager expressions taken in consideration?

[zuck]

I did it! :tada: :tada:

You can explore a showcase project at: https://github.com/zuck/objection-default-eager-example

I implemented the feature overriding the forClass method of QueryBuilder as originally planned:

MyQueryBuilder.forClass = function(modelClass) {
  var qb = objection.QueryBuilder.forClass.call(this, modelClass);

  var _eagerExp = '';
  var _eagerFilters = null;
  var _eagerMaxDepth = modelClass.defaultEagerMaxDepth || 2;
  var _eagerCurrentDepth = 0;
  var _eagerExpObject = {};

  console.log("Using max eager depth of " + _eagerMaxDepth + ".");

  function _traverseEagerExpression(model, prefix, additionalKey, depth) {
    // Recursion guard.
    if (depth > _eagerMaxDepth)
      return;

    var exp = model.defaultEager || null;
    var filters = model.defaultEagerFilters || null;
    var relations = model.getRelations() || {};
    var relNames = Object.keys(relations);

    if (_.isString(exp)) {
      // Removes square bracket for beginning and end of the string.
      exp = exp.replace(/^[\[]+|[\]]+$/g, "");

      // Split and clean up eager expression terms.
      var expTokens = exp
        .split(',')
        .map(function(str) {
          return str.trim();
        });

      // Add additional term from previous cycle to current expression.
      // An additional term is generated when in a default eager expression
      // terms like 'family,father.name' are used. In those cases, the
      // relation associated to the field 'family' is added at the first
      // step, while the additional term '.father.name' is passed to next
      // recursion step.
      // If the additional term is already present in the default eager
      // expression of the current model, don't duplicate it.
      if (additionalKey && expTokens.indexOf(additionalKey) == -1) {
        expTokens.push(additionalKey);
      }

      // Merge default eager filters from current recursion level with those
      // generated by previous recursion cycles.
      if (_.isObject(filters)) {
        _eagerFilters = _.merge(
          _eagerFilters,
          filters
        );
      }

      // For each term in default eager expression, follow the chain
      // if it represents a relation with another model (recursion).
      expTokens.forEach(function(key) {
        var _keyTokens = key.split('.');
        var _key = _keyTokens.pop()
        var _additionalKey = _keyTokens.slice(1).join('.');

        if (relNames.indexOf(_key) > -1) {
          var relatedModel = relations[_key].relatedModelClass;
          var _prefix = _key;
          _traverseEagerExpression(relatedModel, _prefix, _additionalKey, depth + 1);
        }
      });

      // Merge current expression terms with those generated by previous
      // recursion cycles. An object is used to store all of them in order
      // to eliminate overlapped or duplicated terms.
      expTokens.forEach(function(str) {
        var _key = str.trim();
        if (prefix) {
          // Prefix is used to keep track of relation chain.
          _key = prefix + '.' + _key;
        }
        if (!_.has(_eagerExpObject, _key)) {
          _.set(_eagerExpObject, _key);
        }
      });
    }
  }

  // Traverse default eager expression of current model, following
  // the relation chain and merging the results in one expression.
  _traverseEagerExpression(modelClass, '', null, 1);

  // Flatten the generated object in the final eager expression.
  function _flatten(obj) {
    var output = [];
    if (typeof obj === 'object') {
      Object
        .keys(obj)
        .forEach(function(key) {
          if (obj[key]) {
            if (Object.keys(obj[key]).length > 1) {
              output.push(key + '.[' + _flatten(obj[key]) + ']');
            } else {
              output.push(key + '.' + _flatten(obj[key]));
            }
          } else {
            output.push(key);
          }
        });
    }
    return output.join(', ');
  }

  _eagerExp = '[' + _flatten(_eagerExpObject) + ']';

  console.log("Default eager loading of " + _eagerExp + " for model " + modelClass.name);

  return qb.eager(_eagerExp, _eagerFilters);
}

Apart from defaultEager property, I've also integrated two new properties:

1. defaultEagerFilters
2. defaultEagerMaxDepth

[zuck]

After almost a year, I've found another (simpler) way to accomplish the same result (but probably less efficient because it works on instance-level):

objection.Model.prototype.$afterGet = function(queryContext) {
  queryContext.levelCounter = (queryContext.levelCounter || 0) + 1;

  if (queryContext.levelCounter <= (this.constructor.defaultEagerMaxDepth || 2)) {
    const _eagerExpr = objection.RelationExpression.parse(this.constructor.defaultEager);
    const _eagerFilters = this.constructor.defaultEagerFilters;

    return this
      .$query()
      .skipUndefined()
      .context(queryContext)
      .allowEager(_eagerExpr)
      .eager(_eagerExpr, _eagerFilters)
      .first()
      .then(reFetched => {
        this.$set(reFetched);
      });
  }

  return this;
}

Do you have any suggestion to improve the solution?

What is the best way to handle virtual attributes based on relations (which is the original goal of my implementation)?

[lehni]

Wouldn't this simple solution here work?

class MyQueryBuilder {
  ...
  execute() {
    const { unscoped } = this.internalContext()
    if (!unscoped) {
      const { defaultEager } = this.modelClass()
      if (defaultEager) {
        this.internalOptions({ unscoped: false })
        this.mergeEager(defaultEager)
      }
    }
    return super(builder)
  }

  unscoped() {
    return this.internalOptions({ unscoped: true })
  }
}

That way, if you don't want defaultEager to be applied, you call unscoped() on the builder. This is a bit of a misnomer, but I couldn't find a better name. Seems to work for me!

[koskimas]

@lehni I don't recommend using the private internalOptions and internalContext methods. mergeContext() and context() methods are your friends. I would always use mergeContext when setting a value so that you don't accidentally erase context values set by plugins and such.

[lehni]

Oh that's good to know, thanks!

[jtlapp]

Hey @koskimas, I noticed that mergeContext() always merges into a new clone of the current user context. Why doesn't it just merge into the existing user context, or better yet, just let the user directly edit the user context? I just want to double-check that creating a new object for each context change is necessary.

[hroland]

Oh that's good to know, thanks!

Any chance you could update your solution with @koskimas's suggestion?

[stefanvanherwijnen]

Is there a recommended way to achieve a default eager load for a model (yet)? It seems like a trivial task, but I can't find how to do it.

Edit:
This seems to work:

class MyQueryBuilder extends QueryBuilder {
  execute () {
    this.mergeEager('relationship')
    return super.execute()
  }
}

export default class Person extends Model {
...
  static get QueryBuilder() {
    return MyQueryBuilder
  }
...
}

[maprangsoft]

Is there a recommended way to achieve a default eager load for a model (yet)? It seems like a trivial task, but I can't find how to do it.

Edit:
This seems to work:

class MyQueryBuilder extends QueryBuilder {
  execute () {
    this.mergeEager('relationship')
    return super.execute()
  }
}

export default class Person extends Model {
...
  static get QueryBuilder() {
    return MyQueryBuilder
  }
...
}

thank you it work for me.