Sphinx: No way for a directive to force document rebuilds

[webknjaz]

Describe the bug
I've built an extension that injects a subprocess output into the document. And by its nature, every re-run of sphinx-build should refresh its contents.

I know that there's a user-side way to do this, i.e. the end-user would have to _remember_ to pass -a or specify that document explicitly when running sphinx-build. But this is on very user-friendly and requires them to _know upfront_ that the cached doctrees that prevent document rebuilds will also prevent the directive re-invocation.

Expected behavior

I saw some prevention mechanisms in other places. Like, when an extension declares some config value, it can specify the rebuild= arg to make Sphinx force rebuilds on config changes. I'd like to have some mechanism to do the same for directives. Like specifying that if a document has the directive, it should always be rebuilt. Or maybe, re-run such directives to decide if the document should be taken from the doctree or be rebuilt.
Another thought is that it'd be interesting if a directive could declare that its own caching depends on whether certain files or directories have changed. In my case, this variant would be perfect because I invoke towncrier and towncrier basically takes a bunch of RST files and outputs them as one bigger RST (using Jinja2 internally tho).

Ref: https://github.com/sphinx-contrib/sphinxcontrib-towncrier/issues/1
Ref: https://stackoverflow.com/a/44758406/595220

[tk0miya]

You can do it with BuildEnvironment.note_reread(). see https://www.sphinx-doc.org/en/master/extdev/envapi.html#sphinx.environment.BuildEnvironment.note_reread.

[webknjaz]

Interesting, do you have any examples of how to use that? It doesn't seem like the docs cover that.

[webknjaz]

Are you suggesting that I need to subscribe for env-get-outdated and emit an extra file name?

[webknjaz]

@ewjoachim ^

[ewjoachim]

(I tried doing what you @webknjaz said with env-get-outdated but it didn't work).

From reading the doc, I'm really under the impression that properly using env-get-outdated and env-get-updated, plus maybe declaring our files somewhere, would be enough to benefit from the existing caching system, but I really can't wrap my mind around how to do this in practice.

[tk0miya]

Simply, please call the method on your directive.

class MyDirective(Directive):
    def run(self):
        self.state.document.settings.env.note_reread(filename)

You can do it more simply with SphinxDirective class.

from sphinx.util.docutils import SphinxDirective

class MyDirective(SphinxDirective):
    def run(self):
        self.env.note_reread(filename)

Additionally, you can use BuildEnvironment.note_dependency() to add a file to the dependency list of the source file. It is useful to re-build a document on changing the dependency files.

[ewjoachim]

Ah perfect! I think this could be made more apparent in the doc, and if you know where this should be added, I'll be happy to contribute a PR and add this info. But for now, our problem is probably solved !

[tk0miya]

I think these information are already documented in Sphinx's API doc. So I don't have an idea for the update. Please post a PR if you have idea.
Thanks,

[webknjaz]

Hey @tk0miya, this example is missing one corner-case. It assumes that all the dependency files already exist during the first directive run and the cache needs to only depend on those.
But in reality, we also need to depend on some files appearing in a certain folder. How can we implement this?

[webknjaz]

Aha! It looks like BuildEnvironment.note_reread() would add the current RST document that uses the directive to the rebuild_always list unconditionally... Well, this is suboptimal but is better than alternatives.

[webknjaz]

I'm curious if it's possible to attach own metadata to the build env (like recording what documents the directive got used in) so that it'd get picked and then use that in env-get-outdated or doctree-read...

[webknjaz]

It looks like sphinxcontrib.spelling achieves this by creating its own attribute on the instance on env. Also, my testing shows that it's technically possible to record something in env.metadata but it's unclear if that's a good idea...

[tk0miya]

Unfortunately, there is no official way to store own attributes to the env object. But, as you said, some extensions create their attribute to the env and it works fine.

[webknjaz]

@tk0miya re: "it works fine"

It is painful to declare "out-of-contract" dynamic attrs and then check that with mypy. So I suggest coming up with an official API to address this.