Pants: Pants 2.4.0 "Could not find a compatible interpreter" using pyenv (works in 2.3.0)

[sureshjoshi]

Just tried updating to Pants 2.4.0 - however (with no code changes on my side), Pants 2.4.0 will successfully install via ./pants version, but it will fail to run ./pants test :: - while 2.3.0 works just fine installing and running tests.

Tried clearing all the caches, .pids, .pants.d, etc and doing all the normal stuff. I was going to skip to 2.5.0dev2 with the Python 3.9 support, but decided to go to stable 2.4.0 first, before jumping to 2.5.x

Workflow is:

rm -r ~/.cache/pants
rm -r ./pants.d
rm -r ./pids
./pants version
./pants test ::

Environment is:

macOS Big Sur (11.2.3)

Python 3.7.9 (default, Jan 22 2021, 16:36:49) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
(Installed via pyenv)

Python 3.9.4 (default, Apr  5 2021, 01:50:46) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
(Installed via brew)

pants.toml:

[GLOBAL]
pants_version = "2.4.0"
pantsd = true
pants_ignore_use_gitignore = false
backend_packages = [
    "pants.backend.codegen.protobuf.python",
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.python.mixed_interpreter_constraints",
    "pants.backend.python.typecheck.mypy",
]
build_patterns = ["BUILD.pants"]

[source]
# The Python source root is the repo root. See https://www.pantsbuild.org/docs/source-roots.
#root_patterns = ["/"]
marker_filenames = ["BUILD.pants"]

[python-setup]
interpreter_constraints = [">=3.9"]
requirement_constraints = "requirements.lock.txt"

[coverage-py]
report = ["html", "console"]

[pytest]
version = "pytest>=6.2"
args=["-vv"]
pytest_plugins.add = [
    "pytest-asyncio>=0.14",
    "pytest-icdiff>=0.5",
]

[test]
extra_env_vars = [
    "APP_MODE=development",
]

On running ./pants version:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122  100   122    0     0   4692      0 --:--:-- --:--:-- --:--:--  4692
100   281  100   281    0     0   3267      0 --:--:-- --:--:-- --:--:--  3267
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3626k  100 3626k    0     0  7602k      0 --:--:-- --:--:-- --:--:-- 11.4M
Using base prefix '/Users/sj/.pyenv/versions/3.7.9'
New python executable in /Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.qogtiL/install/bin/python3.7
Also creating executable in /Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.qogtiL/install/bin/python
Installing setuptools, pip, wheel...
done.
Collecting pip
  Using cached https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-21.0.1
Collecting pantsbuild.pants==2.4.0
  Using cached pantsbuild.pants-2.4.0-cp37-cp37m-macosx_10_11_x86_64.whl (6.9 MB)
Collecting humbug==0.1.9
  Using cached humbug-0.1.9-py3-none-any.whl (8.5 kB)
Collecting typing-extensions==3.7.4.3
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting fasteners==0.15.0
  Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting packaging==20.9
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting ansicolors==1.1.8
  Using cached ansicolors-1.1.8-py2.py3-none-any.whl (13 kB)
Collecting setproctitle==1.2
  Using cached setproctitle-1.2-cp37-cp37m-macosx_11_0_x86_64.whl
Collecting psutil==5.7.0
  Using cached psutil-5.7.0.tar.gz (449 kB)
Collecting PyYAML<5.5,>=5.4
  Using cached PyYAML-5.4.1-cp37-cp37m-macosx_10_9_x86_64.whl (249 kB)
Collecting pex==2.1.35
  Using cached pex-2.1.35-py2.py3-none-any.whl (2.6 MB)
Collecting setuptools<54.0,>=50.3.0
  Using cached setuptools-53.1.0-py3-none-any.whl (784 kB)
Collecting toml==0.10.2
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting monotonic>=0.1
  Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)
Collecting bugout
  Using cached bugout-0.1.11-py3-none-any.whl (13 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pydantic>=1.6
  Using cached pydantic-1.8.1-cp37-cp37m-macosx_10_9_x86_64.whl (2.6 MB)
Collecting requests
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Building wheels for collected packages: psutil
  Building wheel for psutil (setup.py) ... done
  Created wheel for psutil: filename=psutil-5.7.0-cp37-cp37m-macosx_11_1_x86_64.whl size=227384 sha256=eac1450fcba82bfdb7fc9459a67f22c22ede29a2c900ebc5e00b5ded6f42489c
  Stored in directory: /Users/sj/Library/Caches/pip/wheels/b6/e7/50/aee9cc966163d74430f13f208171dee22f11efa4a4a826661c
Successfully built psutil
Installing collected packages: urllib3, typing-extensions, idna, chardet, certifi, requests, pydantic, six, pyparsing, monotonic, bugout, toml, setuptools, setproctitle, PyYAML, psutil, pex, packaging, humbug, fasteners, ansicolors, pantsbuild.pants
  Attempting uninstall: setuptools
    Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
      Successfully uninstalled setuptools-40.8.0
Successfully installed PyYAML-5.4.1 ansicolors-1.1.8 bugout-0.1.11 certifi-2020.12.5 chardet-4.0.0 fasteners-0.15 humbug-0.1.9 idna-2.10 monotonic-1.6 packaging-20.9 pantsbuild.pants-2.4.0 pex-2.1.35 psutil-5.7.0 pydantic-1.8.1 pyparsing-2.4.7 requests-2.25.1 setproctitle-1.2 setuptools-53.1.0 six-1.15.0 toml-0.10.2 typing-extensions-3.7.4.3 urllib3-1.26.4
New virtual environment successfully created at /Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37.
2.4.0

On running ./pants test ::

00:55:34.75 [INFO] initialization options changed: reinitializing scheduler...
00:55:35.92 [INFO] scheduler initialized.
00:55:38.53 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 229, in _run_inner
    return self._perform_run(goals)
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 168, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 136, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 494, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.core.goals.test.enrich_test_result (services/trainmounter/trainmounter/models_test.py:../tests)
  in pants.backend.python.goals.pytest_runner.run_python_test (services/trainmounter/trainmounter/models_test.py:../tests)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.engine.internals.graph.transitive_targets
  in pants.engine.internals.graph.resolve_targets (services/trainmounter/trainmounter/models_test.py:../tests)
  in pants.engine.internals.graph.resolve_unexpanded_targets (services/trainmounter/trainmounter/models_test.py:../tests)
  in pants.engine.internals.graph.resolve_dependencies (services/trainmounter/trainmounter/models_test.py:../tests)
  in pants.backend.python.dependency_inference.rules.infer_python_dependencies_via_imports (services/trainmounter/trainmounter/models_test.py:../tests)
  in pants.backend.python.dependency_inference.import_parser.parse_python_imports
  in pants.backend.python.util_rules.pex.find_interpreter (CPython==3.7.*)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/sj/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 255, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython==3.7.*' failed with exit code 102.
stdout:

stderr:
Could not find a compatible interpreter.

Examined the following working interpreters:
1.)           /usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/bin/python3.9 CPython==3.9.4
2.) /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python CPython==2.7.16
3.)                                                                                    /usr/bin/python3 CPython==3.8.2

Skipped the following broken interpreters:
1.) /Users/sj/.pyenv/shims/python3.7:
pyenv: python3.7: command not found

The `python3.7' command exists in these Python versions:
  3.7.9

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
2.) /Users/sj/.pyenv/shims/python3.7m:
pyenv: python3.7m: command not found

The `python3.7m' command exists in these Python versions:
  3.7.9

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
3.) /Users/sj/.pyenv/shims/python3.8:
pyenv: python3.8: command not found

The `python3.8' command exists in these Python versions:
  3.8.8

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

(See https://github.com/pantsbuild/pex/issues/1027 for a list of known breaks and workarounds.)

No working interpreter compatible with the requested constraints was found:
  Version matches CPython==3.7.*

[sureshjoshi]

Also, to narrow down exactly when this became a problem...

2.4.0.dev0 -> Works
2.4.0.dev1 -> Fails

[Eric-Arellano]

Thank you for the bug report and especially for narrowing it down to dev0 - dev1, that is extremely helpful!

Can you please try setting this in pants.toml, using 2.4.0.dev1, and see if this fixes the issue? If so, that will confirm that the regression was from upgrading Pex from 2.1.32 to 2.1.33:

[download-pex-bin]
version = "v2.1.32"
known_versions = [
   "v2.1.32|darwin|9cea86ae4338943ab347a5e2b3a71e3c27b9835cdd5c115c52d1a328fdb61160|3071734",
   "v2.1.32|linux|9cea86ae4338943ab347a5e2b3a71e3c27b9835cdd5c115c52d1a328fdb61160|3071734",
]

Also, without that setting and on 2.4.0.dev1, can you please try pyenv global 3.7.9 and see if that fixes things? Note that you can do something like pyenv global 3.7.9 3.8.4 3.9.2.

[sureshjoshi]

Updating the pants.toml did NOT solve the problem, however, setting pyenv global 3.7.9 DID solve the problem (on 2.4.0). Typically I set pyenv global I set to system, and use pyenv local for the version I want specifically to run.

[Eric-Arellano]

Hm, thanks for the update. @jsirois is this change expected that previously you did not have to do pyenv global for an interpreter to be used, and now you do? I see this Pyenv related PR, but that is not yet released: https://github.com/pantsbuild/pex/pull/1325.

These are the commits between 2.4.0.dev0 and 2.4.0.dev1:

3e7e97a26 (tag: release_2.4.0.dev1) Prepare 2.4.0.dev1 (#11653)
9141cd386 [internal] allow Toolchain plugin to request restriected access token for Github Actions PR builds (#11621)
73e97c2cb Require explicit environment usage (#11641)
96be55e11 Bump to version 0.7.0 of toolchain.pants.plugin to pull in explicit environment usage to unblock #11641. (#11654)
dfda4444c Remove WRAPPER_REQUIREMENTS and WRAPPER_SRCPATH (#11645)
9c23f2ba0 Prepare 2.3.0rc2 (#11652)
ea64517a8 Pytest and pylint entry points are console scripts. (#11649)
824a15daa Upgrade to Pex 2.1.33. (#11647)
e7e594cc9 Added support for Pytest config files (#11624)
16040c9a3 Remove references to PANTS_DEV env var (#11644)
eed7d2e77 Simplify pants initialization process and remove `pants_exe.py` (#11643)
542dd68b3 Wrap `./pants help` based on actual terminal width (#11378)
bd81ccaa9 Fix references to the deprecated pants log location. (#11636)
8a94bb4e9 Ensure that ancestor files at the buildroot are found. (#11632)
690aab634 [internal] Remove Google RBE leftovers. (#11630)
26103f431 Two enhancements to tailor. (#11625)
361cd0b87 Handle case where there where no observation histograms during the pants run. (#11628)
9dfb710f7 [internal] Rename pants CI config file to not be specific to travis (#11627)
d44d43ce1 [internal] House development venvs in home. (#11616)
2f298b247 Stabilize git references for indicatif/console changes. (#11623)
97cbd57e8 Add Python tool support for console scripts. (#11620)
e23352af7 [internal] Upgrade toolchain plugin to 0.6.0 (#11615)
eb94001df [internal] Increase gaps in rust tests to reduce flakiness. (#11617)
87f22614a Thread-local stdio handling to support concurrent pantsd clients (#11536)

The only thing I can possibly imagine changing interpreter selection is the Pex upgrade, and that's weird that rolling back to Pex 2.1.32 did not fix this.

[jsirois]

Pex has never handled shims properly. With https://github.com/pantsbuild/pex/pull/1325 it now does; so this may require upgrading Pants and doing some work on the Pants side. Shims are tricky since their activation via pyenv shell relies on the $PYENV_VERSION env var (we can plumb this), and then files for the other modes. Of the files, the global in $HOME can be made to work again by plumbing the $HOME env var if it's not already. For pyenv local though, things get harder. We can really only support the .python-version files in the repo and not any higher up the ancestor directory path. Gathering these files and placing in the sandbox will require a bit of rule work.

[jsirois]

@sureshjoshi it sounds like you have a not-so-attractive workaround with pyenv global. It'll take a bit of work to get this fixed up proper, so I hope that workaround can tide you over a few days. It's important to get this working since pyenv is useful and lots of folks use shims and pyenv local.

[sureshjoshi]

Thanks @Eric-Arellano and @jsirois

Yeah, I just filed the bug to point out the problem and give some details on it while I had the env setup.

I can stay on 2.3.0 or upgrade straight to 2.5.0dev2 and remove pyenv entirely and stick with Python 3.9 everywhere (I only installed pyenv because my apps work on py3.9, but Pants was py3.7/3.8).

So, this isn't remotely a blocker for me :)

[Eric-Arellano]

@jsirois thank you for the explanation. Given all the rule work required, a cherry-pick to 2.4.x sounds risky and we likely want to target 2.5.x for the better Pyenv support. Changing interpreter discovery that profoundly in a patch release sounds like it could be breaking.

[jsirois]

Agreed - this will be far beyond a cherry-pick.

[sureshjoshi]

As a note, this problem exists in 2.4.1 as well.

[sureshjoshi]

Not sure if this ticket needs to remain open - I think just upgrading to 2.5.0 fixes everything (for me at least, I've upgraded without problems).

[Eric-Arellano]

Great! Indeed, fixed in 2.5+