Pants: build-checker step fails: site-packages as dist.location is ambiguous

[RemiTorracinta]

I posted this on slack, but have since gathered more detail so I figured I would post here.

I'm trying to upgrade to pants version 1.26, from 1.22.

When running any lint task, I get this error:

/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/pex_builder.py", line 300, in add_distribution
    dist_name = dist_name or os.path.basename(dist.location)
Exception message: 'NoneType' object has no attribute 'location'

(full stacktrace at the bottom of this issue)

I used pdb to take a look, and it seems the problem is in this snippet:

https://github.com/pantsbuild/pants/blob/release_1.26.0/contrib/python/src/python/pants/contrib/python/checks/tasks/checkstyle/checkstyle.py#L188-L195

For me, the working set resolves pantsbuild.pants.contrib.python.checks.checker==1.26 to the following dists:

pantsbuild.pants.contrib.python.checks.checker 1.26.0 (/Users/remi/.cache/pants/plugins/pantsbuild.pants.contrib.python.checks.checker-1.26.0-py2.py3-none-any.whl-install)
six 1.15.0 (/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.jdiS1p/install/lib/python3.7/site-packages)
pyflakes 2.1.1 (/Users/remi/.cache/pants/plugins/pyflakes-2.1.1-py2.py3-none-any.whl-install)
pycodestyle 2.4.0 (/Users/remi/.cache/pants/plugins/pycodestyle-2.4.0-py2.py3-none-any.whl-install)

The problem lies with six, whose dist.location is ~/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.jdiS1p/install/lib/python3.7/site-packages. When that location is passed to pex_builder.add_dist_location, pex_builder cannot work out what distribution actually needs to be added.

I assume the reason six is in site-packages is that it's also a transitive dependency of pants code (not just checker code). six is also in the ~/.cache/pants/plugins directory as a .whl-install in my sys.path, but I guess it's not chosen as the dist.location, probably because it's later in the list.

However, I'm not sure why I'm hitting this but presumably other Pants users haven't had this issue. Maybe something is different about my environment? Or maybe my interpretation of the problem is wrong?

By the way, I'm reproducing the problem on both linux and mac, with ~/.cache, .pants.d, and .local_artifact_cache cleaned. Any "quick fix" would be super helpful.

Here is the full stack-trace:

➜  base git:(remi-upgrade-pants-1.26) $ ./pants lint company/project/database/:db_connector_test
/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/bin/pants:8: DeprecationWarning: DEPRECATED: option 'glob_expansion_failure' in global scope will be removed in version 1.27.0.dev0.
  If you currently set `--glob-expansion-failure=error`, instead set `--files-not-found-behavior=error`.
  sys.exit(main())
/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/bin/pants:8: DeprecationWarning: DEPRECATED: option 'glob_expansion_failure' in global scope will be removed in version 1.27.0.dev0.
  If you currently set `--glob-expansion-failure=error`, instead set `--files-not-found-behavior=error`.
  sys.exit(main())
/Users/remi/.cache/pants/plugins/astroid-1.6.6-py2.py3-none-any.whl-install/astroid/interpreter/_import/spec.py:6: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/bin/pants:8: DeprecationWarning: DEPRECATED: option 'glob_expansion_failure' in global scope will be removed in version 1.27.0.dev0.
  If you currently set `--glob-expansion-failure=error`, instead set `--files-not-found-behavior=error`.
  sys.exit(main())
22:01:31 [WARN] /Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/bin/pants:8: DeprecationWarning: DEPRECATED: option 'glob_expansion_failure' in global scope will be removed in version 1.27.0.dev0.
  If you currently set `--glob-expansion-failure=error`, instead set `--files-not-found-behavior=error`.
  sys.exit(main())
22:01:31 [WARN] /Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py:126: DeprecationWarning: DEPRECATED: Defaulting to `--build-file-imports=warn` will be removed in version 1.27.0.dev0.
  Import statements should be avoided in BUILD files because they can easily break Pants caching and lead to stale results. The default behavior will change from warning to erroring in 1.27.0.dev0, and the option will be removed in 1.29.0.dev0.
To prepare for this change, please explicitly set the option `--build-file-imports=warn` or `--build-file-imports=error` (we recommend using `error`).
If you still need to keep the functionality you have from import statements, consider rewriting your code into a Pants plugin: https://www.pantsbuild.org/howto_plugin.html
  native, options_bootstrapper, build_config
22:01:31 [WARN] /Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/legacy/parser.py:180: DeprecationWarning: DEPRECATED: Using `globs`, `rglobs`, and `zglobs` will be removed in version 1.27.0.dev0.
  Using deprecated `globs` in company/db/project/BUILD.pants at line 33. Instead, use a list of files and globs, like `sources=['f1.py', '*.java']`. Specify excludes by putting an `!` at the start of the value, like `!ignore.py`.
We recommend using our migration script by running `curl -L -o fix_deprecated_globs_usage.py 'https://git.io/JvOKD' && chmod +x fix_deprecated_globs_usage.py && ./fix_deprecated_globs_usage.py company/db/project`
  self.check_for_deprecated_globs_usage(token_str, filepath, lineno)
22:01:31 [WARN] /Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/legacy/parser.py:180: DeprecationWarning: DEPRECATED: Using `globs`, `rglobs`, and `zglobs` will be removed in version 1.27.0.dev0.
  Using deprecated `globs` in company/db/grader/BUILD.pants at line 37. Instead, use a list of files and globs, like `sources=['f1.py', '*.java']`. Specify excludes by putting an `!` at the start of the value, like `!ignore.py`.
We recommend using our migration script by running `curl -L -o fix_deprecated_globs_usage.py 'https://git.io/JvOKD' && chmod +x fix_deprecated_globs_usage.py && ./fix_deprecated_globs_usage.py company/db/grader`
  self.check_for_deprecated_globs_usage(token_str, filepath, lineno)
18:01:29 00:00 [main]
               (To run a reporting server: ./pants server)
18:01:31 00:02   [setup]
18:01:31 00:02     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> unpack-wheels -> deferred-sources -> native-compile -> link -> gen -> pyprep -> lint
18:01:31 00:02   [bootstrap]
18:01:31 00:02     [substitute-aliased-targets]
18:01:31 00:02     [jar-dependency-management]
18:01:31 00:02     [bootstrap-jvm-tools]
18:01:31 00:02     [provide-tools-jar]
18:01:31 00:02   [imports]
18:01:31 00:02     [ivy-imports]
18:01:31 00:02   [unpack-jars]
18:01:31 00:02     [unpack-jars]
18:01:31 00:02   [unpack-wheels]
18:01:31 00:02     [unpack-wheels]
18:01:31 00:02   [deferred-sources]
18:01:31 00:02     [deferred-sources]
18:01:31 00:02   [native-compile]
18:01:31 00:02     [conan-prep]
18:01:31 00:02     [conan-fetch]
18:01:31 00:02     [c-for-ctypes]
18:01:32 00:03     [cpp-for-ctypes]
18:01:32 00:03   [link]
18:01:32 00:03     [shared-libraries]
18:01:32 00:03   [gen]
18:01:32 00:03     [antlr-java]
18:01:32 00:03     [antlr-py]
18:01:32 00:03     [jaxb]
18:01:32 00:03     [protoc]
18:01:32 00:03     [ragel]
18:01:32 00:03     [thrift-java]
18:01:32 00:03     [thrift-py]
18:01:32 00:03     [py-thrift-namespace-clash-check]
18:01:32 00:03     [grpcio-prep]
18:01:32 00:03     [grpcio-run]
18:01:32 00:03     [wire]
18:01:32 00:03   [pyprep]
18:01:32 00:03     [interpreter]
18:01:32 00:03     [build-local-dists]
18:01:32 00:03     [requirements]
18:01:32 00:03     [sources]
18:01:32 00:03   [lint]
18:01:32 00:03     [scalafix]
18:01:32 00:03     [scalafmt]
18:01:32 00:03     [scalastyle]
18:01:32 00:03     [checkstyle]
18:01:32 00:03     [jvm-dep-check]
18:01:32 00:03     [mypy]
18:01:32 00:03       [create_mypy_pex]
18:01:32 00:03       [check]
Success: no issues found in 9 source files
18:01:33 00:04     [javascriptstyle]
18:01:33 00:04     [python-eval]22:01:33 [WARN] /Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/round_engine.py:47: DeprecationWarning: DEPRECATED: `python-eval` defaulting to being used will be removed in version 1.27.0.dev0.
  `python-eval` is scheduled to be removed in Pants 1.29.0.dev0. The Python linter landscape has changed since we first created this tool - there are now popular linters that dramatically improve upon this one, such as MyPy and Pylint. Pants currently provides a wrapper around MyPy and will soon add Pylint. (To install MyPy, add `pantsbuild.pants.contrib.mypy==%(pants_version)s` to your `plugins` list.)
To prepare, set `skip = true` in your `pants.toml` under the section `python-eval`. If you still need to use this tool, set `skip = false`. In Pants 1.27.0.dev0, the default will change from `skip = false` to `skip = true`, and in Pants 1.29.0.dev0, the module will be removed.
  elif task.skip_execution:
18:01:33 00:04     [pythonstyle]
                   Invalidated 1 target.
18:01:34 00:05       [build-checker]
               Waiting for background workers to finish.
18:01:34 00:05   [complete]
               FAILURE
timestamp: 2020-06-02T18:01:34.270126
Exception caught: (builtins.AttributeError)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_loader.py", line 94, in main
    PantsLoader.run()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_loader.py", line 90, in run
    cls.load_and_execute(entrypoint)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_loader.py", line 83, in load_and_execute
    entrypoint_main()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_exe.py", line 32, in main
    PantsRunner(start_time=start_time).run()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_runner.py", line 115, in run
    return runner.run()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 293, in run
    self._run()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 389, in _run
    goal_runner_result = self._maybe_run_v1()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 330, in _maybe_run_v1
    self._exiter,
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/goal_runner.py", line 230, in run
    return self._run_goals()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/goal_runner.py", line 201, in _run_goals
    result = self._execute_engine()
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/goal_runner.py", line 189, in _execute_engine
    result = engine.execute(self._context, self._goals)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/legacy_engine.py", line 21, in execute
    self.attempt(context, goals)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/round_engine.py", line 252, in attempt
    goal_executor.attempt(explain)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/round_engine.py", line 50, in attempt
    task.execute()
  File "/Users/remi/.cache/pants/plugins/pantsbuild.pants.contrib.python.checks-1.26.0-py36.py37-none-any.whl-install/pants/contrib/python/checks/tasks/checkstyle/checkstyle.py", line 284, in execute
    failure_count += self.checkstyle(interpreter, sources)
  File "/Users/remi/.cache/pants/plugins/pantsbuild.pants.contrib.python.checks-1.26.0-py36.py37-none-any.whl-install/pants/contrib/python/checks/tasks/checkstyle/checkstyle.py", line 217, in checkstyle
    checker = self.checker_pex(interpreter)
  File "/Users/remi/.cache/pants/plugins/pantsbuild.pants.contrib.python.checks-1.26.0-py36.py37-none-any.whl-install/pants/contrib/python/checks/tasks/checkstyle/checkstyle.py", line 195, in checker_pex
    pex_builder.add_dist_location(dist.location)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/python/pex_build_util.py", line 576, in add_dist_location
    self._builder.add_dist_location(location)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/pex_builder.py", line 337, in add_dist_location
    self.add_distribution(dist, dist_name=name)
  File "/Users/remi/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/pex_builder.py", line 300, in add_distribution
    dist_name = dist_name or os.path.basename(dist.location)
Exception message: 'NoneType' object has no attribute 'location'

[stuhood]

Thank you for the report!

Re: six: when you say "just site-packages", do you mean your machine's global site-packages, or something inside the virtualenv created by the pants script? The latter would be normal and expected... the former would be problematic. Pants/PEX try to scrub out global site-packages, but in some cases that can fail... if you see anything that isn't related to pants' own virtualenv in sys.path, that could be an issue.

It may not be relevant here, but we would generally recommend bumping one version at a time (even if you only sanity check rather than commit each of them), because it is the best way to see all deprecations. I'm not aware of any deprecations affecting this area, but in this case bumping version at a time might help us to determine where this was first introduced.

It's definitely not satisfying, but as a workaround to unblock yourself to focus on the rest of the upgrade, disabling the pants.contrib.python.checks backend temporarily would allow you to make progress.

[RemiTorracinta]

Re: six: when you say "just site-packages", do you mean your machine's global site-packages, or something inside the virtualenv created by the pants script?

No, sorry, it's not a problem with global site-packages. There are two sixes available on sys.path

• ~/.cache/pants/plugins/six-1.15.0-py2.py3-none-any.whl-install
• ~/.cache/pants/setup/bootstrap-Darw in-x86_64/pants.jdiS1p/install/lib/python3.7/site-packages/six-1.15.0.dist-info

I think one is available for pants code itself, and one is available because six is a dependency of pantsbuild.pants.contrib.python.checks.checker==1.26

When the checker tries to build the cheker pex, the dist.location it passes is just the bootstrap site-packages folder (~/.cache/pants/setup/bootstrap-Darw in-x86_64/pants.jdiS1p/install/lib/python3.7/site-packages).

https://github.com/pantsbuild/pants/blob/release_1.26.0/contrib/python/src/python/pants/contrib/python/checks/tasks/checkstyle/checkstyle.py#L195

As a result, pex_builder can't work out what dist the path is referring to:
https://github.com/pantsbuild/pex/blob/v2.1.7/pex/pex_builder.py#L336

[stuhood]

Got it... I think that that makes sense. It's still somewhat baffling that others haven't seen this, but this "resolve from the python path" code has caused some issues in the past: sorry for the trouble.

There is a bit of context here that affects how deeply we will investigate this though: 1.28.x deprecates the contrib/python package (and the 1.30.x series removes it) in favor of recommending that folks use the v2 Python backend for linting and formatting: see https://pants.readme.io/docs/python-linters-and-formatters.

So... while I would still recommend bumping version-at-a-time, if you're prepared to bump a bit further and try out the new backend, I'd recommend disabling that plugin, getting on a version that supports the new linters (1.28.0 is the current stable version, but 1.29.0rc1 is also maturing), and see how that goes.

cc @Eric-Arellano who's been driving this improvement.

[RemiTorracinta]

Yes, still unsure what's different for me than others.

Two things that might make the issue hard to reproduce:

• The bug does not occur when PANTS_DEV is enabled, as that results in a different execution path: https://github.com/pantsbuild/pants/blob/release_1.26.0/contrib/python/src/python/pants/contrib/python/checks/tasks/checkstyle/checkstyle.py#L169
• The bug does not occur if DistributionNotFound or PEXBuilder.InvalidDistribution are raised somewhere, which I observed when trying to copy the checker plugin to my local repo.

By the way, I proved to myself that the sys.path order is affecting things,
added import sys; sys.path.reverse() after the for loop here, https://github.com/pantsbuild/pants/blob/release_1.26.0/src/python/pants/init/plugin_resolver.py#L68 , and the issue is resolved (though of course that's not the "real" fix.)

By the way, I tested on major versions 1.22 through 1.25 and did not see the issue.

[RemiTorracinta]

FYI: as a workaround I plan to copy the plugin to our repository, and eventually drop it when we get onto v2.

[stuhood]

Thank you: very sorry for the trouble. The volume of changes in 1.26.x was pretty high, and tapers off a bit in 1.27.x and 1.28.x... overall the UI changes have been settling down as v2 has matured.

[Eric-Arellano]

Thank you Remi! Sorry for the trouble.

FYI you can start using the V2 linters even without using V2 overall. See https://pants.readme.io/v1.27/docs/python-linters-and-formatters for the 1.27.x docs on how to activate it. Leave off pants.backend.python from backend_packages2.

Overall, v2 Flake8 is much better as the Python checks plugin misses a bunch of important checks, and the v2 implementation works with plugins. Should also be faster.

If you have any Python 2 code left, and can upgrade to 1.29.x, you'll be able to run ./pants lint ::, and Pants will run the Python 2 targets separately from Python 3 :)

[Eric-Arellano]

(Closing as answered, even though the answer is not very satisfying. Please feel free to reopen if the workaround stops working.)

[RemiTorracinta]

I didn't know that about mixing in v2 linters! That's very helpful, thanks.

[Eric-Arellano]

Yay, glad that helps! One of the huge v2 benefits is that we now support 6 linters/formatters, rather than only isort and the Checks plugin. And those linters will all run in parallel :)

(Check out the recently added page https://pants.readme.io/docs/pants-v1-vs-v2 for more info on v1 vs. v2 and how to upgrade)