Pip: Refactor the resolver to take a list of parsed requirements

[pfmoore]

What's the problem this feature will solve?
Currently, the (legacy) resolver takes a list of InstallRequirement objects, which are in an odd state compared to the later parts of pip's code. They contain a bunch of config information (option flags, etc) but don't necessarily refer to an actual distribution file (they can be something like pip >= 20.0). The resolver puts these requirements into a RequirementSet and then mutates them as it works through the resolution, preparing them, etc.

The problem here is that it means that the InstallRequirement object has an uncertain status - it is mostly (in particular, in later parts of the processing) expected to contain a link to a distribution, but early on it can be a "requirement specifier style" object that has no actual distribution behind it (and so, for example, can't be prepared).

Describe the solution you'd like
Make the resolver take ParsedRequirement objects, which are what the argument parsing code produces and which don't have any role in "post resolution" code. This will more completely decouple the classes used by "pre-resolver" code and "post-resolver" code.

The new resolver could then take ParsedRequirement objects as the basis of the resolvelib Requirement objects and InstallRequirement objects as the basis of the resolvelib Candidate objects, and not have to deal with the dual nature of InstallRequirement.

Alternative Solutions
The current prototype of the new resolver tries to work with InstallRequirement objects for both phases, but it is hitting errors which appear to be a result of being unable to correctly manage the state changes of InstallRequirement objects.

Additional context
Longer term, it might be useful to completely remove the state where an InstallRequirement doesn't have a populated link. But this is likely to be superseded by a move to the "project" model. In fact, the proposed change here would possibly be useful as a first step towards the "project" model (although that isn't an explicit goal of this refactoring).

[pfmoore]

Ping @uranusjr @pradyunsg - also see this comment on Zulip.

Also, @chrahunt do you have any comments - I feel like I'm re-inventing thoughts you probably had when you proposed the previous set of refactorings, so I'd appreciate your input here.

[pfmoore]

Reviewing the code further, this may actually not be as good an idea as I'd hoped. Parsed requirements are only generated as part of requirement file processing at the moment. Requirements specified on the command line don't go via that route at the moment, so that would need to be amended as well as the resolver calling code.

In addition, it's no longer clear to me that the problems with using InstallRequirement objects as input for the new resolver code are as fundamental as they initially seemed. It may be possible to fix them while still having InstallRequirement objects as inputs.

[chrahunt]

Needing some refactoring doesn't make this a bad idea. Aligning all inputs to be of the same type that is independent of the resolver implementation is a fundamental step that unlocks our ability to trade out InstallRequirement for something more robust and less stateful to back the Candidate objects.

[pfmoore]

Sorry, you're correct. What I should have said was (1) that I don't now believe that this refactoring is a blocker for the new resolver implementation and (2) that it's a bigger piece of work than I'd initially assumed. We can (if my latest experiments are correct) complete the implementation of the new resolver using InstallRequirement if necessary.

This refactoring is on the critical path for moving away from InstallRequirement, but what I was intending to say is that it's not on the critical path for the new resolver implementation. (Unless, of course, further weird behaviour of InstallRequirement crawls out of the woodwork 🙄)