Lektor: Filtered query adds non-filtered results as page dependencies

[davidferguson]

When performing a query in a template, or a plugin, all the results of the base query are added as dependencies of that page, even if the query is filtered or further refined.

For example, say I have the following site structure:

/people/
  contents.lr
  /james/
    contents.lr
  /john/
    contents.lr
  /steven/
    contents.lr
  /kevin/
    contents.lr

If on the template of /people/contents.lr I have the following to display all people with names starting with j:

{% for page in site.query('/people/').filter(F.filename.startswith('j')) %}
{{ page.name }}
{% endfor %}

Then while the output is correct, all the results of the base query site.query('/people/') are added as dependencies to the page, rather than just the filtered results.

This is particularly problematic when dealing with thousands of subpages (as I was) as the process of updating the database in _memorize_dependencies in lektor/builder.py#L668 can be very slow for lots of dependencies.

For the moment I'm "solving" this in my site by querying the database using the pad object directly, however it probably should be properly fixed.

I think it can be fixed by modifying _iterate in db.py to only record dependencies if it matches the query, rather than (as it currently does) for every result.

[xlotlu]

@davidferguson did this get fixed in the meantime?

[dairiki]

No, I don't think this has been fixed. I'm not quite sure such a fix is possible.

It's a very subtle issue. It depends on how "correct" (vs "useful") one wants the dependency tracking to be. Note that, in general, the result sequence of a filtered query (or a limited query) can change due to changes in children which were outside the filtered set.

I've been trying to wrap my head around some of these issues while working on the lektor-index-pages plugin. There, I work-around some of this by having the filtered queries depend on a virtual source object which has a "checksum" which depends on the identities and order of the children in the filtered set.

I have written another plugin (which I have not yet published to github or PyPI, but could be convinced to do so, if there's interest) in an attempt to generalize this idea. It provides a jinja-global filter, limit_dependencies which can be applied to other (filtered or limited) queries. The filter computes the results of the query (while hacking things so as to prevent any depencies from being recorded) and returns a custom Query instance containing just the limited results. It also creates a virtual source object whose checksum depends on the identities and order of the query results and registers that virtual source as a dependency. It's a little hacky and goofy, but it appears to work.

[davidferguson]

No, it wasn't fixed, because I realised that it wasn't actually ever a bug - just me misunderstanding the dependency system (and I just forgot to close this issue). As @dairiki says, you do want all the pages of a query to be dependencies - not just filtered ones - because when those pages are updated, they might then meet the filter criteria.

@dairiki your idea sounds interesting - the site I'm working on right now has nearly 9000 content files, and the build times are over an hour now, much of which is spent on dependency tracking and updating the db. I did investigate getting round this by manually doing dependency tracking for the indexes (which does work, but is more hacky than your system!), but for now we're still using the default system.

[dairiki]

In case anyone is interested, I've just published lektor-limit-dependencies to PyPI.

The plugin defines a Jinja filter named limit_dependencies. Using the filter in the example in the original ticket like this:

{% for page in site.query('/people/').filter(F.filename.startswith('j'))|limit_dependencies %}
{{ page.name }}
{% endfor %}

"should" quasi-magically solve the so-many-dependencies issue.