Pants: Deprecate interpreter constraint ORing.

[jsirois]

AKA, move to [python-setup] interpreter_constraints and PythonTarget.compatibility being str arguments and not lists of strings. The list of strings (which are ORed) is an un-needed feature that diverges from the the python ecosystem way of spelling "works with python 2 and 3" which seems to be the only real use of the OR feature. Wheels themselves simply specify a full range and subtract out unsupported interior spans with != and * (https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires). Supporting OR provides shorter interpreter constraint strings, but at the cost of supporting a new concept.

So, the proposal is uses like so:

python_library(
  compatibility=['CPython>=2.7,<3', 'CPython>=3.6.*']
)

Port to the packaging ecosystem form of:

python_library(
  compatibility='CPython>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*'
)

We jump through some hoops to support the OR facility in #9265 and we could simplify that code and Pex itself if we deprecate and remove the feature.

[benjyw]

This is great. We may not be able to automate fixing BUILD files, but we can reliably detect all uses of this and provide a sensible deprecation warning, and let users figure out how to find&replace.

[benjyw]

Am I right in thinking that this works today?

python_library(
  compatibility=['CPython>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*']
)

?

So essentially we can already move to this, and the thing that needs a deprecation cycle is to switch the field type from list of strings to string?

[jsirois]

Am I right in thinking that this works today?
...
So essentially we can already move to this, and the thing that needs a deprecation cycle is to switch the field type from list of strings to string?

Yes, exactly.

[hrfuller]

I think it's a good idea to stay in line with python standards. My only concern is our ability to update all our build files internally fast enough. An automated build file fix would be ideal, but sounds hard. We have about ~125 uses internally that we would need to resolve. At least one maybe two releases as the deprecation cycle would be ideal. In lieu of an automated fix some instructions here, or even better in a comment above the deprecation about how to convert the values would help.

[hrfuller]

@nsaechao I got the number of uses from rg -I "compatibility\s?=\s?\[" | wc -l Does that cover the twitter internal usage or are there other places that would miss?

[jsirois]

@hrfuller the number of uses isn't all that interesting, just the number of unique uses. My guess is that would be 1-3. If so, we hand translate the 1-3 ORs into 1-3 ranges + exclusions and then run 1 big sed or equivalent.

[jsirois]

In other words, a BUILD file fix automated for a given repo is easy, it's only the general case, where you need to come up with code that maps a disjunction into a range with holes poked out with != that is harder. If we can come up with the mapping from disjunction to range with holes poked out by hand, this is a pretty easy one-off script.

[hrfuller]

Sure, I see your point. I'm satisfied that this won't be hard to fix for us, given a suitable deprecation schedule.

[hrfuller]

It would still be great to hear from @nsaechao because he led our py3 migration, and may have some knowledge that I don't about our interpreter constraint situation.

[jsirois]

@nsaechao in absence of your feedback; given @hrfuller's positive feedback given reasonable deprecation leeway, I'm going to proceed with this this week.

[hrfuller]

Sounds good, thanks @jsirois

[Eric-Arellano]

I started to implement and realized we'd be losing functionality: ORing across interpreter classes, e.g. PyPy OR CPython for the same code. Afaict, there is no way to express that via a single pkg_resources.Requirement() string.

Stu wondered if we need to support this. For a pex_binary, we can already use platforms to express that something works with either interpreter class. The platforms override the interpreter constraints.

But, that wouldn't work for python_library (and python_tests). Example: a team wants to experiment with PyPy, while rest of org uses CPython. They need to convert some core util code to work with PyPy, while still working with CPython for everyone else. They need to be able to set that common python_library to PyPy OR CPython.*

*Technically, they could create two targets for the same code. But this breaks dependency inference for imports of that code, and it doesn't scale well generally to have lots of targets.

[jsirois]

We need not lose functionality (I suspect this is unused functionality, but granted it might be useful):

$ python -mpex setuptools
Python 3.8.6 (default, Sep 30 2020, 04:00:38) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from pkg_resources import Requirement
>>> req = Requirement.parse("dummy >=3.6,<4 ; platform_python_implementation == 'CPython' or platform_python_implementation == 'PyPy'")
>>> req.marker
<Marker('platform_python_implementation == "CPython" or platform_python_implementation == "PyPy"')>