Pants: 2.3.0rc0 - ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2

[sureshjoshi]

Hi all,

I just upgraded from 2.3.0.dev3 to 2.3.0rc0 and running ./pants version wouldn't finish without an error. I can successfully compile and test with 2.3.0.dev1 and 2.3.0.dev3 (didn't test dev2) - but rc0 fails before it finishes installing/setting up.

Note: It DOES work on Mac, but fails on Debian.

Mac Environment (works)

macOS Big Sur (11.2.1)

Python 3.7.9 (default, Jan 22 2021, 16:36:49) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Python 3.9.1 (default, Jan 22 2021, 16:49:02) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Linux Environment (fails)

Debian GNU/Linux 10 (buster)
Linux 4.19.0-10-amd64

Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux

Python 3.9.1 (default, Feb 3 2021, 20:42:26)
[GCC 8.3.0] on linux

Stacktrace:

./pants version

10:33:48.67 [INFO] initializing pantsd...
10:33:49.60 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.7.3)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2.
stdout:

stderr:
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: syntax error near unexpected token `('
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: `    venv_dir = os.path.abspath(os.path.dirname(__file__))'

Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 154, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 124, in prepare
    self._initialize(options_fingerprint, options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 99, in _initialize
    raise e
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 85, in _initialize
    options_bootstrapper, raise_=True
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/options_initializer.py", line 108, in build_config_and_options
    build_config = _initialize_build_configuration(self._plugin_resolver, options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/options_initializer.py", line 43, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/plugin_resolver.py", line 109, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/plugin_resolver.py", line 130, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [self._interpreter_constraints])[
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 592, in product_request
    self._raise_on_error([t for _, t in throws])
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.7.3)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2.
stdout:

stderr:
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: syntax error near unexpected token `('
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: `    venv_dir = os.path.abspath(os.path.dirname(__file__))'

[stuhood]

Sorry for the trouble!

Would you mind attaching the contents of the /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8 directory as a zip file? cc @jsirois

[Eric-Arellano]

Thanks @sureshjoshi! I encountered this yesterday when trying to upgrade a repository using Circle CI, but upgrading one with Travis worked. I haven't been able to isolate why it works in some places yet, and that's helpful to know you're hitting it on Debian.

I had been waiting to announce the release because of this. We'll get out a new release asap once we figure out what's going on.

[sureshjoshi]

@stuhood Attached

pex.zip

pants.toml - created an empty project, no special pip resolvers (that fails too though), no other fields.

[GLOBAL]
pants_version = "2.3.0rc0"

@Eric-Arellano
No worries, there is no particular reason for me to use rc0 specifically.

I'm using 2.3.x because of the venv additions in the pex tool, but every now and again, I just upgrade to the latest dev versions to keep an eye on regressions that might burn me when I have less free time :)

dev3 works perfectly fine for me - so just wanted to post this issue as a flag for anyone else who might run into it if they're doing what I am too.

[jsirois]

@sureshjoshi / @Eric-Arellano the relevant file in the zip is the top level pex file. In order to debug this I'll either need:

1. You to manually try to run that file using the same python 3.7 interpreter Pants picked / edit, repeat to figure out the cause of the syntax error
2. More details on the vm / docker image. The exact Linux distribution and how you installed the relevant Python 3.7 if it wasn't pre-installed. This should allow me to repro and debug.

[sureshjoshi]

@jsirois

I'll play around with option 1 in the evening when I have some time, not sure how useful I'll be though.

Re 2:

Parallels installed Debian (10.7) - system python 3.7.3, pyenv installed 3.9.1 (which I guess isn't used yet, so doesn't matter)

>> hostnamectl
   Static hostname: debian-gnu-linux-10-vm
         Icon name: computer-vm
           Chassis: vm
        Machine ID: d86f5054f02e804a80c10d6e62349e7f
           Boot ID: 101a0a3078bc4becb84aaccdeed49437
    Virtualization: kvm
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-10-amd64
      Architecture: x86-64

>> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

>> cat /etc/debian_version
10.7

>> apt list --installed | grep python
dh-python/stable,now 3.20190308 all [installed,automatic]
libpython-stdlib/stable,now 2.7.16-1 amd64 [installed,automatic]
libpython2-stdlib/stable,now 2.7.16-1 amd64 [installed,automatic]
libpython2.7-minimal/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
libpython2.7-stdlib/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
libpython2.7/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
libpython3-dev/stable,now 3.7.3-1 amd64 [installed,automatic]
libpython3-stdlib/stable,now 3.7.3-1 amd64 [installed,automatic]
libpython3.7-dev/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
libpython3.7-minimal/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
libpython3.7-stdlib/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
libpython3.7/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python-apt-common/stable,now 1.8.4.3 all [installed,automatic]
python-crypto/stable,now 2.6.1-9+b1 amd64 [installed,automatic]
python-dnspython/stable,now 1.16.0-1 all [installed,auto-removable]
python-gpg/stable,now 1.12.0-6 amd64 [installed,automatic]
python-ldb/stable,now 2:1.5.1+really1.4.6-3 amd64 [installed,automatic]
python-minimal/stable,now 2.7.16-1 amd64 [installed]
python-pip-whl/stable,now 18.1-5 all [installed,automatic]
python-samba/stable,now 2:4.9.5+dfsg-5+deb10u1 amd64 [installed,automatic]
python-talloc/stable,now 2.1.14-2 amd64 [installed,automatic]
python-tdb/stable,now 1.3.16-2+b1 amd64 [installed,automatic]
python2-minimal/stable,now 2.7.16-1 amd64 [installed,automatic]
python2.7-minimal/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
python2.7/stable,now 2.7.16-2+deb10u1 amd64 [installed]
python2/stable,now 2.7.16-1 amd64 [installed,automatic]
python3-apt/stable,now 1.8.4.3 amd64 [installed,automatic]
python3-asn1crypto/stable,now 0.24.0-1 all [installed,automatic]
python3-bcrypt/stable,now 3.1.6-1 amd64 [installed,auto-removable]
python3-brlapi/stable,now 5.6-10+deb10u1 amd64 [installed,automatic]
python3-cached-property/stable,now 1.5.1-3 all [installed,auto-removable]
python3-cairo/stable,now 1.16.2-1+b1 amd64 [installed,automatic]
python3-certifi/stable,now 2018.8.24-1 all [installed,automatic]
python3-cffi-backend/stable,now 1.12.2-1 amd64 [installed,automatic]
python3-chardet/stable,now 3.0.4-3 all [installed,automatic]
python3-crypto/stable,now 2.6.1-9+b1 amd64 [installed,automatic]
python3-cryptography/stable,now 2.6.1-3+deb10u2 amd64 [installed,automatic]
python3-cups/stable,now 1.9.73-2+b1 amd64 [installed,automatic]
python3-cupshelpers/stable,now 1.5.11-4 all [installed,automatic]
python3-dbus/stable,now 1.2.8-3 amd64 [installed,automatic]
python3-debconf/stable,now 1.5.71 all [installed,automatic]
python3-debian/stable,now 0.1.35 all [installed,automatic]
python3-debianbts/stable,now 2.8.2 all [installed,automatic]
python3-dev/stable,now 3.7.3-1 amd64 [installed,automatic]
python3-distro-info/stable,now 0.21 all [installed,automatic]
python3-distro/stable,now 1.3.0-1 all [installed,automatic]
python3-distutils/stable,now 3.7.3-1 all [installed,automatic]
python3-docker/stable,now 3.4.1-4 all [installed,auto-removable]
python3-dockerpty/stable,now 0.4.1-1 all [installed,auto-removable]
python3-dockerpycreds/stable,now 0.3.0-1 all [installed,auto-removable]
python3-docopt/stable,now 0.6.2-2 all [installed,auto-removable]
python3-entrypoints/stable,now 0.3-1 all [installed,automatic]
python3-gi-cairo/stable,now 3.30.4-1 amd64 [installed,automatic]
python3-gi/stable,now 3.30.4-1 amd64 [installed,automatic]
python3-gst-1.0/stable,now 1.14.4-1+b1 amd64 [installed,automatic]
python3-httplib2/stable,now 0.11.3-2 all [installed,automatic]
python3-idna/stable,now 2.6-1 all [installed,automatic]
python3-jinja2/stable,now 2.10-2 all [installed,auto-removable]
python3-jmespath/stable,now 0.9.4-1 all [installed,auto-removable]
python3-jsonschema/stable,now 2.6.0-4 all [installed,auto-removable]
python3-kerberos/stable,now 1.1.14-2 amd64 [installed,auto-removable]
python3-keyring/stable,now 17.1.1-1 all [installed,automatic]
python3-keyrings.alt/stable,now 3.1.1-1 all [installed,automatic]
python3-lib2to3/stable,now 3.7.3-1 all [installed,automatic]
python3-libcloud/stable,now 2.4.0-1 all [installed,auto-removable]
python3-lockfile/stable,now 1:0.12.2-2 all [installed,auto-removable]
python3-louis/stable,now 3.8.0-2 all [installed,automatic]
python3-mako/stable,now 1.0.7+ds1-1 all [installed,automatic]
python3-markupsafe/stable,now 1.1.0-1 amd64 [installed,automatic]
python3-minimal/stable,now 3.7.3-1 amd64 [installed,automatic]
python3-nacl/stable,now 1.3.0-2 amd64 [installed,auto-removable]
python3-netaddr/stable,now 0.7.19-1 all [installed,auto-removable]
python3-ntlm-auth/stable,now 1.1.0-1 all [installed,auto-removable]
python3-paramiko/stable,now 2.4.2-0.1 all [installed,auto-removable]
python3-pip/stable,now 18.1-5 all [installed]
python3-pkg-resources/stable,now 40.8.0-1 all [installed,automatic]
python3-pyasn1/stable,now 0.4.2-3 all [installed,auto-removable]
python3-pyatspi/stable,now 2.30.0+dfsg-3 all [installed,automatic]
python3-pycurl/stable,now 7.43.0.2-0.1 amd64 [installed,automatic]
python3-pysimplesoap/stable,now 1.16.2-1 all [installed,automatic]
python3-reportbug/stable,now 7.5.3~deb10u1 all [installed]
python3-requests-kerberos/stable,now 0.11.0-2 all [installed,auto-removable]
python3-requests-ntlm/stable,now 1.1.0-1 all [installed,auto-removable]
python3-requests/stable,now 2.21.0-1 all [installed,automatic]
python3-secretstorage/stable,now 2.3.1-2 all [installed,automatic]
python3-selinux/stable,now 2.8-1+b1 amd64 [installed,auto-removable]
python3-setuptools/stable,now 40.8.0-1 all [installed,automatic]
python3-simplejson/stable,now 3.16.0-1 amd64 [installed,auto-removable]
python3-six/stable,now 1.12.0-1 all [installed,automatic]
python3-smbc/stable,now 1.0.15.6-1+b2 amd64 [installed,automatic]
python3-software-properties/stable,now 0.96.20.2-2 all [installed,automatic]
python3-speechd/stable,now 0.9.0-5+deb10u1 all [installed,automatic]
python3-texttable/stable,now 1.6.0-1 all [installed,auto-removable]
python3-uno/stable,now 1:6.1.5-3+deb10u6 amd64 [installed,automatic]
python3-urllib3/stable,now 1.24.1-1 all [installed,automatic]
python3-venv/stable,now 3.7.3-1 amd64 [installed]
python3-websocket/stable,now 0.53.0-1 all [installed,auto-removable]
python3-wheel/stable,now 0.32.3-2 all [installed,automatic]
python3-winrm/stable,now 0.3.0-2 all [installed,auto-removable]
python3-xdg/stable,now 0.25-5 all [installed,automatic]
python3-xmltodict/stable,now 0.11.0-2 all [installed,auto-removable]
python3-yaml/stable,now 3.13-2 amd64 [installed,auto-removable]
python3.7-dev/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3.7-minimal/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3.7-venv/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3.7/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3/stable,now 3.7.3-1 amd64 [installed]
python/stable,now 2.7.16-1 amd64 [installed]
unit-python3.7/stable,now 1.21.0-1~buster amd64 [installed,upgradable to: 1.22.0-1~buster]

[jsirois]

I don't repro:

$ docker run --rm -it -v $HOME/Downloads:/downloads debian:10.7 bash -c '
apt -qq update && apt -qq install --yes python3.7-dev python3.7-venv unzip
unzip -qd triage /downloads/pex.zip
triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python --version
triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex -c "import os, site; print(os.linesep.join(site.getsitepackages()))"
'
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
The following additional packages will be installed:
  bzip2 ca-certificates file libc-dev-bin libc6-dev libexpat1 libexpat1-dev libmagic-mgc libmagic1 libmpdec2 libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib libreadline7 libsqlite3-0 libssl1.1 linux-libc-dev manpages manpages-dev mime-support openssl
  python-pip-whl python3.7 python3.7-minimal readline-common xz-utils
Suggested packages:
  bzip2-doc glibc-doc man-browser python3.7-doc binutils binfmt-support readline-doc zip
The following NEW packages will be installed:
  bzip2 ca-certificates file libc-dev-bin libc6-dev libexpat1 libexpat1-dev libmagic-mgc libmagic1 libmpdec2 libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib libreadline7 libsqlite3-0 libssl1.1 linux-libc-dev manpages manpages-dev mime-support openssl
  python-pip-whl python3.7 python3.7-dev python3.7-minimal python3.7-venv readline-common unzip xz-utils
0 upgraded, 30 newly installed, 0 to remove and 7 not upgraded.
Need to get 68.9 MB of archives.
After this operation, 161 MB of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libssl1.1:amd64.
(Reading database ... 6677 files and directories currently installed.)
Preparing to unpack .../00-libssl1.1_1.1.1d-0+deb10u5_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1d-0+deb10u5) ...
Selecting previously unselected package libpython3.7-minimal:amd64.
Preparing to unpack .../01-libpython3.7-minimal_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7-minimal:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../02-libexpat1_2.2.6-2+deb10u1_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.6-2+deb10u1) ...
Selecting previously unselected package python3.7-minimal.
Preparing to unpack .../03-python3.7-minimal_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7-minimal (3.7.3-2+deb10u2) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../04-readline-common_7.0-5_all.deb ...
Unpacking readline-common (7.0-5) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../05-bzip2_1.0.6-9.2~deb10u1_amd64.deb ...
Unpacking bzip2 (1.0.6-9.2~deb10u1) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../06-libmagic-mgc_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking libmagic-mgc (1:5.35-4+deb10u2) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../07-libmagic1_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.35-4+deb10u2) ...
Selecting previously unselected package file.
Preparing to unpack .../08-file_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking file (1:5.35-4+deb10u2) ...
Selecting previously unselected package manpages.
Preparing to unpack .../09-manpages_4.16-2_all.deb ...
Unpacking manpages (4.16-2) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../10-mime-support_3.62_all.deb ...
Unpacking mime-support (3.62) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../11-xz-utils_5.2.4-1_amd64.deb ...
Unpacking xz-utils (5.2.4-1) ...
Selecting previously unselected package openssl.
Preparing to unpack .../12-openssl_1.1.1d-0+deb10u5_amd64.deb ...
Unpacking openssl (1.1.1d-0+deb10u5) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../13-ca-certificates_20200601~deb10u2_all.deb ...
Unpacking ca-certificates (20200601~deb10u2) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../14-libc-dev-bin_2.28-10_amd64.deb ...
Unpacking libc-dev-bin (2.28-10) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../15-linux-libc-dev_4.19.171-2_amd64.deb ...
Unpacking linux-libc-dev:amd64 (4.19.171-2) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../16-libc6-dev_2.28-10_amd64.deb ...
Unpacking libc6-dev:amd64 (2.28-10) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../17-libexpat1-dev_2.2.6-2+deb10u1_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.2.6-2+deb10u1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../18-libmpdec2_2.4.2-2_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-2) ...
Selecting previously unselected package libreadline7:amd64.
Preparing to unpack .../19-libreadline7_7.0-5_amd64.deb ...
Unpacking libreadline7:amd64 (7.0-5) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../20-libsqlite3-0_3.27.2-3+deb10u1_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.27.2-3+deb10u1) ...
Selecting previously unselected package libpython3.7-stdlib:amd64.
Preparing to unpack .../21-libpython3.7-stdlib_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7-stdlib:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package libpython3.7:amd64.
Preparing to unpack .../22-libpython3.7_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package libpython3.7-dev:amd64.
Preparing to unpack .../23-libpython3.7-dev_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7-dev:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package manpages-dev.
Preparing to unpack .../24-manpages-dev_4.16-2_all.deb ...
Unpacking manpages-dev (4.16-2) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../25-python-pip-whl_18.1-5_all.deb ...
Unpacking python-pip-whl (18.1-5) ...
Selecting previously unselected package python3.7.
Preparing to unpack .../26-python3.7_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7 (3.7.3-2+deb10u2) ...
Selecting previously unselected package python3.7-dev.
Preparing to unpack .../27-python3.7-dev_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7-dev (3.7.3-2+deb10u2) ...
Selecting previously unselected package python3.7-venv.
Preparing to unpack .../28-python3.7-venv_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7-venv (3.7.3-2+deb10u2) ...
Selecting previously unselected package unzip.
Preparing to unpack .../29-unzip_6.0-23+deb10u2_amd64.deb ...
Unpacking unzip (6.0-23+deb10u2) ...
Setting up libexpat1:amd64 (2.2.6-2+deb10u1) ...
Setting up mime-support (3.62) ...
Setting up libmagic-mgc (1:5.35-4+deb10u2) ...
Setting up manpages (4.16-2) ...
Setting up libssl1.1:amd64 (1.1.1d-0+deb10u5) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.
28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up unzip (6.0-23+deb10u2) ...
Setting up libsqlite3-0:amd64 (3.27.2-3+deb10u1) ...
Setting up libmagic1:amd64 (1:5.35-4+deb10u2) ...
Setting up linux-libc-dev:amd64 (4.19.171-2) ...
Setting up file (1:5.35-4+deb10u2) ...
Setting up bzip2 (1.0.6-9.2~deb10u1) ...
Setting up libpython3.7-minimal:amd64 (3.7.3-2+deb10u2) ...
Setting up xz-utils (5.2.4-1) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up python3.7-minimal (3.7.3-2+deb10u2) ...
Setting up libmpdec2:amd64 (2.4.2-2) ...
Setting up libc-dev-bin (2.28-10) ...
Setting up openssl (1.1.1d-0+deb10u5) ...
Setting up readline-common (7.0-5) ...
Setting up libreadline7:amd64 (7.0-5) ...
Setting up manpages-dev (4.16-2) ...
Setting up libpython3.7-stdlib:amd64 (3.7.3-2+deb10u2) ...
Setting up libpython3.7:amd64 (3.7.3-2+deb10u2) ...
Setting up ca-certificates (20200601~deb10u2) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.
28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Updating certificates in /etc/ssl/certs...
137 added, 0 removed; done.
Setting up libc6-dev:amd64 (2.28-10) ...
Setting up python-pip-whl (18.1-5) ...
Setting up python3.7 (3.7.3-2+deb10u2) ...
Setting up python3.7-venv (3.7.3-2+deb10u2) ...
Setting up libexpat1-dev:amd64 (2.2.6-2+deb10u1) ...
Setting up libpython3.7-dev:amd64 (3.7.3-2+deb10u2) ...
Setting up python3.7-dev (3.7.3-2+deb10u2) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for ca-certificates (20200601~deb10u2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Python 3.7.3
Re-execing from /triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/site-packages
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/local/lib/python3.7/dist-packages
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3/dist-packages
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/dist-packages

Is there anything odd with the parallels setup @sureshjoshi ? In particular any mounts that leak out into macOS land or something like that?

[sureshjoshi]

So strange.

My VM is isolated, and works fine on 2.3.0.dev1 and 2.3.0.dev3 as mentioned above, specifically rc0 giving me a hassle. I've tried wiping all the caches I can think of as well.

I'll try creating a fresh VM and seeing what happens there in the eve - but as far as I can imagine, not sure why the other two dev versions would work.

[jsirois]

Ah, so in the never ending - read closer dummy department - I read closer and got less dumb: "syntax error near unexpected token `('" - That's bash speaking and not Python AFAICT. Ruminating on that...

[jsirois]

As to the other two dev versions @sureshjoshi , plugin resolution - where this is failing - is new between the last dev release and rc0.

[sureshjoshi]

Ah, I was wondering if it was actually something between pex 2.1.30 and 2.1.31 - because I vaguely recall messing with something the other day where I was changing pex versions (in a docker container somewhere, unrelated to this problem) and seeing something similar.

But, sounds like we're far outside my wheelhouse. I'm available to run tests on my environ as needed

[sureshjoshi]

@jsirois Weird question... Could we be exceeding the shebang line limit?

It's 155 characters in my example, I put a symlink to my home dir, used that instead, and running ./pex now works

Symlink:

ln -s /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python3.7 ~/pppython

In __main__.py:

From:

#!/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python3.7 -sE

To:

#! /home/parallels/pppython -sE

./pants version succeeds

[jsirois]

I'm ignorant of shebang length limits. I do know ecrpytfs encrypted directories have ~140 character limits for ... path components IIRC. But do you have that setup for home dirs even? And then, even if so, I'm pretty sure that's a path component (file name) length limit, not a total path length limit.

I'll poke around though to learn about any shebang specific restrictions...

[jsirois]

Yeah - no dice on straight up debian 10.7 with no encrypted home dir:

$ docker run --rm -it -v $HOME/Downloads:/downloads debian:10.7 bash -c '
apt -qq update && apt -qq install --yes python3.7-dev python3.7-venv unzip
mkdir -p /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943
unzip -qd /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943 /downloads/pex.zip
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python --version
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex -c "import os, site; print(os.linesep.join(site.getsitepackages()))"
'
...
Python 3.7.3
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/site-packages
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/local/lib/python3.7/dist-packages
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3/dist-packages
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/dist-packages

[sureshjoshi]

FWIW: This shebang thing seems to be a known Python issue after looking for it

https://www.in-ulm.de/~mascheck/various/shebang/
https://github.com/pypa/pip/issues/1773
https://github.com/pypa/virtualenv/issues/596
https://github.com/pypa/virtualenv/issues/997

It also looks like the latest pex re-wrote a lot of code around this functionality with the venvs __main__.py file. Not sure if it's related - since the shebang at the top is the same, but there seems to be code to re-exec if python isn't in use: https://github.com/pantsbuild/pex/compare/v2.1.30...v2.1.31#diff-44e8fd52cd3a63a2d3f2fc516418c2aa55e8ee96f6eb82b2186765a8b853922f

Is there some way that code be added to create a pex_root symlink to the appropriate venv? That could chomp the length in half probably.

As I'm not familiar with the codebase, I'm not really sure why the venvs folders are so nested (some pex hashes I guess along the way) - so I don't know if there are expected to be multiple venvs in the same folders which would necessitate some clever symlink naming in my example (which could just be a pain to do and maintain).

What really sucks is that this appears to be an OS-specific version, and if the Linux distro in use increases the specified length, this is basically a non-issue.

[jsirois]

Ah yes. Thanks. Not a known Python issue - just a known shebang issue regardless of interpreter. I'll fork a Pex issue, fix over there and upgrade Pants, etc.

Thanks again for digging with me @sureshjoshi .

[jsirois]

And to stomp out the last variable: This explain _exactly_ why my docker repro didn't repro since that uses the host kernel which is my Arch Linux kernel:

$ grep BINPRM_BUF_SIZE /usr/include/linux/binfmts.h 
#define BINPRM_BUF_SIZE 256

[jsirois]

https://github.com/pantsbuild/pex/issues/1252

[jsirois]

I'll spin up a 2.3.0rc1 as soon as #11604 lands with the Pex upgrade.

[jsirois]

And to finally repro:

$ grep BINPRM_BUF_SIZE /usr/include/linux/binfmts.h
#define BINPRM_BUF_SIZE 256
$ head -1 script.py | wc -c
272
$ cat script.py 
#!/tmp/CBgxyrSClhbciHwNdHUGtUvBgZliL73duGVtv8Wmv6d1KfJdjHsJfucoENe5B2eRwiLhO4CXon4bUJHwQOSKiZYOkbhs9JJlZgd2qTmdcRhWqkbsGPjbnpp92B4S5R97DQL3kMwo5b750FIsImpmFgX4dHdXyFjotXUvW3m2lZ4IQ4k8UReZwVutsCKhgWb8gpyt5nQKV1ABOPZ8KzhFbpsEpi3ttYcEwGMhMHKtKrg0zCR4a00vNb1fcr/bin/python3.9

echo $SHELL
echo $BASH
echo $BASH_VERSION
echo bob

print("Hello World!")

$ ./script.py 
/bin/bash
/bin/bash
5.1.4(1)-release
bob
./script.py: line 8: syntax error near unexpected token `"Hello World!"'
./script.py: line 8: `print("Hello World!")'

Which is _disturbing_. Instead of - say E2BIG - it looks like bash is being run instead.

That disturbing behavior is just bash though and not Linux - phew:

$ python -c 'import subprocess; subprocess.check_call(["/tmp/script.py"])'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/tmp/script.py'

The reason @sureshjoshi saw the error he did which is the disturbing category error above is that we use bash as an intermediary for running venvs and thus buy this bash bad behavior:
https://github.com/pantsbuild/pants/blob/9dd433dddd814b1a15a7573b6fac7f119d295d0a/src/python/pants/backend/python/util_rules/pex.py#L688-L704

Bash MAN page snip:

COMMAND EXECUTION
       After a command has been split into words, if it results in a simple command and an optional list of arguments, the following actions are taken.

       If the command name contains no slashes, the shell attempts to locate it.  If there exists a shell function by that name, that function is invoked as described above in FUNCTIONS.  If the name does not match a function, the shell searches for it  in  the  list  of  shell
       builtins.  If a match is found, that builtin is invoked.

       If  the  name is neither a shell function nor a builtin, and contains no slashes, bash searches each element of the PATH for a directory containing an executable file by that name.  Bash uses a hash table to remember the full pathnames of executable files (see hash under
       SHELL BUILTIN COMMANDS below).  A full search of the directories in PATH is performed only if the command is not found in the hash table.  If the search is unsuccessful, the shell searches for a defined shell function named command_not_found_handle.  If that function ex‐
       ists,  it  is invoked in a separate execution environment with the original command and the original command's arguments as its arguments, and the function's exit status becomes the exit status of that subshell.  If that function is not defined, the shell prints an error
       message and returns an exit status of 127.

       If the search is successful, or if the command name contains one or more slashes, the shell executes the named program in a separate execution environment.  Argument 0 is set to the name given, and the remaining arguments to the command are set to the arguments given, if
       any.

       If  this  execution fails because the file is not in executable format, and the file is not a directory, it is assumed to be a shell script, a file containing shell commands.  A subshell is spawned to execute it.  This subshell reinitializes itself, so that the effect is
       as if a new shell had been invoked to handle the script, with the exception that the locations of commands remembered by the parent (see hash below under SHELL BUILTIN COMMANDS) are retained by the child.

       If the program is a file beginning with #!, the remainder of the first line specifies an interpreter for the program.  The shell executes the specified interpreter on operating systems that do not handle this executable format themselves.  The  arguments  to  the  inter‐
       preter consist of a single optional argument following the interpreter name on the first line of the program, followed by the name of the program, followed by the command arguments, if any.

[jsirois]

Now live in 2.3.0rc1: https://pypi.org/project/pantsbuild.pants/2.3.0rc1/

[sureshjoshi]

@jsirois Thanks! Can confirm this works for me now.

I should also note that in plugin_resolver.py, the following code meant I completely bypassed the venv workflow (and thereby, bypassed the shebang problem) in rc1:

if not requirements:
    return ResolvedPluginDistributions()

To sanity check, I commented that code out, and can confirm that I'm now getting the named_caches/pex_root/venvs/short/{8ByteHash} path and that also succeeds.