Pants: Non-target specs

[benjyw]

Today, it is strongly assumed that command-line address specs refer to existing targets.

However in some contexts it may make more sense for them to refer to files/directories in general, regardless of the existence of BUILD files in those directories.

For example, to run a console_rule that generates BUILD files, you'll want to specify where the code that should be buildgenned is. You could do this with options:

./pants buildgen --source=src/python/

But it's a lot more natural to do something like:

./pants buildgen src/python/

And more generally it seems like being able to reference files and dirs, not targets, might be a more natural idiom in various contexts where target information (e.g., dependencies) is not required. E.g., cloc.

The v2 engine makes this quite feasible: You have to ask to have Specs turned into BuildFileAddresses anyway, and you could just as easily ask to turn them into paths. In fact, I have a prototype of this working, but it's a little hacky, as an input spec get parsed into SingleAddressSpec, which is a type assumed to represent a target, and in this case it doesn't.

This issue is for discussing potential solutions to the need outlined above.

Two possible approaches that come to mind:

• Slightly expand the specs concept to allow representing paths/files as well as target addresses.

• Greatly expand the specs concept to a general-purpose PositionalArgs product that only gets parsed as address specs if the console_rule asks for it to be, but could equally be parsed as anything else. A URL, say, where that made sense.

[cosmicexplorer]

I love it!!!

And more generally it seems like being able to reference files and dirs, not targets, might be a more natural idiom in various contexts where target information (e.g., dependencies) is not required. E.g., cloc.

This is super intriguing and it took me a few minutes but I've decided I really like this!

In fact, I have a prototype of this working, but it's a little hacky, as an input spec get parsed into SingleAddressSpec, which is a type assumed to represent a target, and in this case it doesn't.

As an aside, I found the code in #8542 to make the --query argument parsing process in #7350 much easier to express via an extension of the UnionRule concept. Would love to know whether it seems like #8542 could be used to help reduce hackiness here.

Greatly expand the specs concept to a general-purpose PositionalArgs product

I personally do not consider this a much greater expansion of the specs concept, to me it seems like the implementation might be much more readable, which feels preferable when introducing new CLI syntax. It also feels like it might be easier to allow deprecation of spec interpretations if we have a single explicit place we parse for them in.

[stuhood]

Would also be good to align this with --owner-of, which operates on files. When we were implementing it, there was discussion of whether it should be Spec syntax rather than an option, but we leaned toward an option for a reason that I think is relevant here as well: which target would "own" a directory?

Slightly expand the specs concept to allow representing paths/files as well as target addresses.

So, AFAIK: there is no reason why a @console_rule couldn't (today) take Specs in its arguments and do anything it wanted to them. Specs parsing does not confirm whether something is a file or directory (IIRC? and if it does, it doesn't need to).

Adding a @console_rule that did anything other than expect a target at some location would be a big step. But if we are willing to take the step, we could push --owner-of down into the syntax pretty easily, I think. Just need to convince @illicitonion =)

[benjyw]

Would also be good to align this with --owner-of, which operates on files. When we were implementing it, there was discussion of whether it should be Spec syntax rather than an option, but we leaned toward an option for a reason that I think is relevant here as well: which target would "own" a directory?

Slightly expand the specs concept to allow representing paths/files as well as target addresses.

So, AFAIK: there is no reason why a @console_rule couldn't (today) take Specs in its arguments and do anything it wanted to them. Specs parsing does not confirm whether something is a file or directory (IIRC? and if it does, it doesn't need to).

Indeed, this is how my proof-of-concept works. The only minor hackiness is that the Spec in question is a SingleAddressSpec, which implies that we already think it's an address. But we can fish the path out of that spec and treat it as not-an-address and it works fine. I'd just rather be a little more explicit in the types.

[benjyw]

Turns out that the ArgSplitter decides that something is a spec and not a goal based on it "looking like a spec" (containing /, : or being a top-level dir). But it doesn't really have to use that heuristic, since it knows about all the scopes, so it effectively knows about the goals.

[benjyw]

Hmm, this is not so simple. Basically, today we would know that in

./pants knowngoal unknowngoal

unknowngoal is a goal, because it doesn't "look like" a spec. If we allow arbitrary positional args that don't have to be spec-like, then we will assume that unknowngoal is such an arg. The knowngoal will then fail on a bogus arg, but that's a different error message than the one we show today, which is "no such goal", which is probably more useful.

[benjyw]

So I guess for now we don't support arbitrary positional args, just those that "look like it" according to some heuristic. That's probably fine for a long time.

[stuhood]

So I guess for now we don't support arbitrary positional args, just those that "look like it" according to some heuristic. That's probably fine for a long time.

Honestly, completely arbitrary positional args would be very confusing.

I think of this ticket as opening the door to "files and directories may also be Specs", but going further than that would be a huge decision anyway.

[benjyw]

True, but I'm thinking maybe things like URLs might make sense as specs. Referencing remote code repos directly is a coherent thing to do in a build tool.

[benjyw]

That would be really easy to add though.

[stuhood]

True, but I'm thinking maybe things like URLs might make sense as specs. Referencing remote code repos directly is a coherent thing to do in a build tool.

It is, but we have a kickass options system (thanks!). And I don't think that completely arbitrary computation in a @console_rule is a desirable goal. Also, explicit passthrough args are a thing, and should probably stay that way.

[Eric-Arellano]

FYI: I'll be doing some prework the next few days to get us closer to this feature. To clarify the scope from the discussion above, we are aiming for this scope:

• Continue supporting specifying targets, globs, and target shorthand (src/python/pants/util is shorthand for src/python/pants/util:util)
• Add support for specifying files and directories, e.g. foo.py, src/python, and likely src/python/**/*.py.
• Rules will request if they want files/directories or targets/globs (or both) through their signature.

We are _not_ aiming for this:

• Allow arbitrary positional args. The grammar should stay ./pants [options] [goals] [shorthand-options] [specs].
• Change how BUILD files work. sources should still require explicit files and dependencies should require targets.

I'll explore the --owner-of discussion some more. I'm not very familiar with how that all works.

[stuhood]

That all sounds great: thanks. I'd love to see --owner-of deprecated in favor of this feature. cc @illicitonion

[benjyw]

I'd like to mention that the fact that this should be straightforward to do without impacting existing rules is further evidence of the design strength of the engine.