Pants: Option values read from @files don't invalidate when the file changes

[benjyw]

Any* option's value can be read from a file, by using @path/to/file as the value.

A subsystem instance that has such an option is not invalidated when the contents of the file change, so changing the file and then rerunning pants doesn't apply the new value, until you restart pantsd.

• Unless it sets fromfile=False.

[stuhood]

So, I think that a plan here is:

• ~Remove the dependency from rust bootstrapping on BinaryUtil, which should allow options parsing to depend on the native code.~
• Introduce a "bootstrap" Scheduler instance which will be responsible for constructing the bootstrap options that are used to create the "real" Scheduler.
• Refactor the relevant portion of options parsing (namely, anything that interacts with files) into @rules, initially breaking out of the sandbox with absolute file reads.
• Add an absolute file read instrinsic, and use that to finally resolve this.

[Eric-Arellano]

I like it. V2-ification of the options system.

However, this is a really big undertaking, so we should double check that this is a 2.0 priority. cc @benjyw. Is this only a broken window when pantsd is used?

[benjyw]

It is, but it's a big one.

We could however just not support fromfile options in 2.0 and reintroduce them in 2.1?

[Eric-Arellano]

We could however just not support fromfile options in 2.0 and reintroduce them in 2.1?

To confirm, something like --isort-config works okay, then, right? This is solely an issue with fromfile.

[stuhood]

Yea... that and possibly the file_option and dir_option types breaking out of the sandbox?

It has a slight impact on performance, because we re-parse all config files for each run, and validate all option values rather than having them memoized... the engine would make that easy.

[stuhood]

FWIW, the @file feature isn't even documented: https://www.pantsbuild.org/docs/options , so maybe we don't need to do anything here until closer to release.