Pip: Isolated builds when both pyproject.toml and setup.py co-exist

[adithyabsk]

Environment

• pip version: 20.1.1
• Python version: 3.7.7
• OS: macOS 10.14.6 (18G103)

Description
The maintainers of flake8 are suggesting that the adoption of pyproject.toml will not move forward until pip's behavior is amended. This seems to be due to the backend build system changing in the presence of a pyproject.toml file. Popular projects such as black have centralized their configuration using pyproject.toml and hence there may be many systems with both setup.py and a pyproject.toml configuration files during the transition period to pyproject.toml for build specifications.

I wanted to open a dialogue here to try and understand better understand pip's behavior in this regard and the maintainer's claim of "forced isolated builds with no recourse".

How to Reproduce
I was able to reproduce their issue with pip using the latest version.

$ pip install six
$ pip --version
pip 20.1.1 from /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip (python 3.7)
$ pip freeze
six==1.15.0
$ cat setup.py
import setuptools
import six

setuptools.setup(
    name="pip_test", # Replace with your own username
    version="0.0.1",
    author="Example Author",
    author_email="[email protected]",
    description="A small example package",
    long_description="long_description",
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)
$ pip install .
Processing /Users/adithyabalaji/Coding/pip_test
Building wheels for collected packages: pip-test
  Building wheel for pip-test (setup.py) ... done
  Created wheel for pip-test: filename=pip_test-0.0.1-py3-none-any.whl size=1563 sha256=71b54964c94d7ebbc065f2a3afb2cb05ba23ae23974a58e1eefbeb91c8c8e493
  Stored in directory: /private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-ephem-wheel-cache-uv97sy5f/wheels/5c/62/db/42908c304e6d9828375edcd5c4347c2d13df469ca005a1e575
Successfully built pip-test
Installing collected packages: pip-test
Successfully installed pip-test-0.0.1
$ touch pyproject.toml
$ pip install .
Processing /Users/adithyabalaji/Coding/pip_test
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/bin/python /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/tmpw33op9k3
       cwd: /private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-req-build-scma8bnc
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 148, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 128, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 250, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 143, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 2, in <module>
      import six
  ModuleNotFoundError: No module named 'six'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/bin/python /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/tmpw33op9k3 Check the logs for full command output.
$ pip install . --no-use-pep517
Processing /Users/adithyabalaji/Coding/pip_test
Building wheels for collected packages: pip-test
  Building wheel for pip-test (setup.py) ... done
  Created wheel for pip-test: filename=pip_test-0.0.1-py3-none-any.whl size=1563 sha256=91a191eac2dcb7cc85c10c27d1cb603bc1ba98c1091cc0f177dfd644e18b87dc
  Stored in directory: /private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-ephem-wheel-cache-o2hrr93v/wheels/5c/62/db/42908c304e6d9828375edcd5c4347c2d13df469ca005a1e575
Successfully built pip-test
Installing collected packages: pip-test
  Attempting uninstall: pip-test
    Found existing installation: pip-test 0.0.1
    Uninstalling pip-test-0.0.1:
      Successfully uninstalled pip-test-0.0.1
Successfully installed pip-test-0.0.1

[adithyabsk]

Tagging the aforementioned maintainer of flake8 here in case they have anything to add to the discussion here.

@asottile

[McSinyx]

Hi, IIUC isolated build will get triggered when pyproject.toml exists, and I don't understand how the linked thread is relevant (which to my understanding, it's about flake8 supporting configuration from the TOML file, not on the build of the project). With build isolation, aside from the fallback setuptools and wheel, you'll need to add six to the build requirements in this case.

[adithyabsk]

@McSinyx I do not want to speak on behalf of the flake8 maintainers but my understanding of their concern is that if pyproject.toml is supported, some projects could become broken due to the pip installation backend changing. As mentioned in https://github.com/pypa/pip/issues/2381, I am not aware of a way to specify build-time requirements in setup.py without switching to pyproject.toml as the build system.

My (unfounded, data here would be good) claim would be that users who have setup.py scripts that depend on non-isolated builds (ie their setup.py script has build-time dependencies) are few and far between and could easily be pointed to an FAQ page as to how to address that issue (migrating to pyproject.toml). But, I can see the burden that might pose to developers since flake8 is such a widely used package.

[McSinyx]

I think there's some misunderstanding here (either from my side or your side):

• The flake8's ticket is about delay support flake8 configuration (which usually resides in tox.ini, setup.cfg, etc.), not delay the usage of isolation of the build process of flake8 itself.
• I think (I could be wrong though) that the final goal of Python packaging is to have build-isolation for every package (i.e. PEP 517/518). The isolation can be used by pip regardless pyproject.toml exists or not. I don't understand how flake8 is relevant here since it's for linting, not building and thus is not relevant to the PEPs anyhow. It is entirely possible to have build isolation and have flake8 running, e.g. pip itself.

It is possible that you're confused about the meaning of build-isolation. If one's

setup.py script has build-time dependencies

perse should have the dependency specified in pyproject.toml. The mentioned PEPs are to solve such situation, so that a front-end (like pip) may be aware of the back-end needed (setuptools, wheel and six in your case) and install them in a virtual environment during build, regardless of the outer environment. To my understanding

setup.py scripts that depend on non-isolatedable builds

are obsolete stuff build via e.g. GNU make and should be converted to or wrapped by Python tooling somehow.

[adithyabsk]

there's some misunderstanding here

It's likely on my end. I've read the above-linked issue thread in the flake8 repo multiple times and I'm still not fully following. Maybe you can help me figure out what I am missing.

The flake8's ticket is about delay support flake8 configuration [...] not delay the usage of isolation of the build process of flake8 itself.

I think we both agree here. It's a long thread over there, but my understanding of the argument against supporting pyproject.toml in flake8 is due to the default behavior of pip when a pyproject.toml exists in a repository.

I don't understand how flake8 is relevant here since it's for linting, not building and thus is not relevant to the PEPs anyhow.

flake8 is not _specifically_ relevant here. The reason I linked their thread was to provide context for my question. The key reason cited against supporting pyproject.toml as a flake8 configuration option was due to the backend switching behavior of pip:

Anthony Sottile · 6 months ago
If you want this, I'd suggest lobbying for the following:

• pip to change its behaviour so mere presence of the file does not change functionality
  
  
  
  • this probably involves a PEP, or at least a discussion on the various pypa mailing lists
  
  

Based on this:

the final goal of Python packaging is to have build-isolation for every package... [scripts that depend on non-isolatable builds] are obsolete stuff build via e.g. GNU make

I guess my question would then be, is the old pip build backend on deprecation track? If it is, there would be a stronger argument to suggest the "forced isolated builds" are not a concern since that would be the direction of the Python packaging ecosystem, and solutions are provided in the form of PEP 517/518.

[pfmoore]

I agree with @McSinyx here. The example setup.py provided is incorrect, in the sense that it will not work in an environment where six isn't present, but there is no indication of this fact in the project metadata.

To demonstrate this, create a new virtualenv without six installed, and with just the given setup.py (no pyproject.toml). pip install . will fail there, too. This is not about pyproject.toml, instead it's about projects having build requirements that are not available in machine readable form.

The fix for a project like this is to amend the documentation that says "you must manually install six before installing this project" to also say "... and you need to provide the --no-build-isolation flag to pip". Or better, just add six to pyproject.toml and then you need neither bit of documentation 🙂

[uranusjr]

Title and top post edited to fix typos.

I also removed #6163 from the title since it describes the use case of a setup.py importing a module from the to-be-installed-distribution, which is a supported (although discouraged) usage, while the setup.py in this case is referencing a module that is not a part of the to-be-installed distribution, which has a well-defined way to declare (listing it in pyproject.toml).

[pganssle]

My (unfounded, data here would be good) claim would be that users who have setup.py scripts that depend on non-isolated builds (ie their setup.py script has build-time dependencies) are few and far between and could easily be pointed to an FAQ page as to how to address that issue (migrating to pyproject.toml). But, I can see the burden that might pose to developers since flake8 is such a widely used package.

To be 100% clear here, because I don't think anyone has clarified it — pyproject.toml is not a replacement for setup.py, you don't have to do any migration. If you have a setup.py you should also have a pyproject.toml that describes that your build backend in setuptools. In almost all cases, even if the build-system table is unspecified in pyproject.toml, pip will still work just fine.

[pfmoore]

Also, this is not about pyproject.toml at all, but about build isolation. It's easy to confuse the two, because (for better or worse) pip uses the presence of pyproject.toml as a signal that projects had opted into newer standards, and were therefore prepared for build isolation. The unexpected speed with which tools like black and flake8 switched to storing configuration in pyproject.toml made that assumption seem a bit optimistic in hindsight.

Better documentation and blogs/articles/tutorials explaining how all of this hangs together would be very welcome. I don't know if anyone with the right combination of familiarity with the subject, good writing skills, and access to a suitable audience, would be able to help here.

[adithyabsk]

Hi @McSinyx @pfmoore @uranusjr and @pganssle thank you for taking the time to follow up on my question. I've been going back and forth with a flake8 developer via email (@asottile) based on your feedback. I asked if it was okay for me to share his concerns with this thread and he gave me the go-ahead.

Here is an (abbreviated) summary of the exchange:

---

On Jun 13, 2020, at 9:52 PM, @adithyabsk wrote:
[...]

> I know it was mentioned in the thread that the presence of pyproject.toml forces isolated builds with no opt-out but my understanding is that you can use the “—no-use-pep517” to opt-out of the new pip build backend. I know that toml language support still isn’t in the core python lang, another concern that was raised. But, I was hoping this new information might persuade you to re-open the thread for discussion.

On Jun 22, 2020, at 12:08 AM, @asottile wrote:
yes, the problem is that the presence of the file fundamentally changes how pip installs things. --no-use-pep517 is a cop-out solution at best, pip install X should just work. additionally the file ends up bundled in distributions (since setuptools includes it by default) making --no-use-pep517 even more of a non-solution (since you cannot predict which of your dependencies use / don't use isolated builds).

> because it causes pip to have strange behaviours, I can't in-earnest support pyproject.toml in flake8 and deprecate all of the other configuration approaches. and adding yet another way to configure flake8 when the configuration story is pretty terrible is not going to happen.

On Sun, Jun 21, 2020 at 9:17 PM @adithyabsk wrote:
Okay, that’s totally fair.

I can't in-earnest support pyproject.toml in flake8 and deprecate all of the other configuration approaches

I cannot speak to the other implementation but the PR that I had opened, was only around 100 lines of code to support pyproject.toml in addition to the other configuration approaches. Regardless, I understand your frustration with additional code to support yet another “standard” for configuration which increases the burden on the flake8 developers.

because it causes pip to have strange behaviours

> With that said, would you willing to post your concerns to the thread that I started in pypa/pip - or - would you be okay if I shared your concerns with that thread? Since, I do not maintain nearly as many projects you do, I don’t have personal experience with strange pip behaviors that you mention.

On Jun 22, 2020, at 12:21 AM, @asottile wrote:
sure feel free to share

my thoughts on the sensible things pip should do is either:

• don't treat empty pyproject as isolated
• default isolated build on regardless of pyproject

[pfmoore]

I'm not sure where this discussion is going. We've covered pip's existing behaviour. Is there a specific request for a change in pip that we can consider here? The current behaviour is a compromise (between encouraging projects to specify build dependencies in pyproject.toml and not breaking projects who aren't ready to change how they are configured). The two proposals above:

• don't treat empty pyproject as isolated
• default isolated build on regardless of pyproject

are basically the two extremes that pip is compromising between. So which are you proposing we move to?

Note that as far as I can see, there's been no explanation given of why a project that changes to add flake8 configuration to pyproject.toml can't, at the same time add their build dependencies and get the whole transition done at once. Or conversely why, if they don't want to use pyproject.toml to specify their build dependencies yet, they can't just hold off putting flake8 config in there? "We didn't know that making this one change would also affect this other area" is a fair point, but it'll get caught in testing, and it's easy to fix (revert or add the dependencies, and you're done).

I'm genuinely baffled as to why people feel pip has to change, rather than projects make a simple build config update - so please, if there is a good reason, can someone explain it?

[bluemellophone]

Thank you all in this thread for pushing this discussion and development of pip forward. In regards to Black using project.toml files for configuration, it is a known issue that the original maintainer has refused to fix. For anybody else who has run into this issue with building wheels and setup.py throwing ModuleNotFound errors due to a Black configuration file, I'd suggest pivoting away from Black and using this instead: https://github.com/odwyersoftware/brunette. Luckily, Flake8 also allows setup.cfg files, so a great alternative for our project is to pivot to configuring Flake8 and Brunette in a single file and deprecating the toml file.

This fix works with pip 20.1.1

[pganssle]

Probably a better fix would be to just explicitly add a [build-system] table to your pyproject.toml file, and report any issues you have with it?

Eventually the behavior that you are switching away from black to avoid will become the default, and you'll have no recourse but to pin to an older pip. It would be better to opt in today and get your problems fixed before it causes problems.

[bluemellophone]

We will migrate to pyproject.toml once the standard is better documented, discussions and bugs around policies like this one have settled, and setup.py is officially deprecated. Our workflows rely on editable installs of local packages for development and, as far as I know, the new pyproject.toml + setup.cfg doesn't allow easily. We may eventually go back to Black and its configuration via toml but switching to Brunette for the interim has more than one upside for us.

Thanks for following up about there being multiple paths, though, but we may peg pip depending on how stable we want our immediate CI infrastructure to behave under sporadic development under COVID.

[pganssle]

We can close this issue now, right? There's nothing actionable, and it mainly seems like it's based on misunderstandings, which I'm not sure we can clarify better than we already have - both here and in many, many other venues.

[bluemellophone]

I would like to quote @pfmoore above:

"We didn't know that making this one change would also affect this other area" is a fair point, but it'll get caught in testing, and it's easy to fix (revert or add the dependencies, and you're done).

This will get caught in testing, but it breaks existing behavior and leaves the developer with not much in terms of hints on where to turn. For example, we use scikit-build to marshal our built C libraries from CMake. This is the error log when we do pip install -e .

Obtaining file:///Users/<username>/code/<package>
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/<username>/virtualenv/<env3.6>/bin/python3.6 /Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/tmp27lw61jr
       cwd: /Users/<username>/code/<package>
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 147, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 249, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 252, in <module>
      import skbuild
  ModuleNotFoundError: No module named 'skbuild'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/<username>/virtualenv/<env3.6>/bin/python3.6 /Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/tmp27lw61jr Check the logs for full command output.

If I delete the pyproject.toml file as recommended by Black for configuration (https://github.com/psf/black/blob/master/pyproject.toml), this magically works. There isn't much to indicate that isolated builds or PEP 517/518 are related. At the very least, if pip determines that a build error is related to this change, a simple message would help. I'm not the only one who is confused or out of date: https://news.ycombinator.com/item?id=22746762 or https://www.reddit.com/r/learnpython/comments/biq99o/what_is_the_current_best_practice_for_specifying/.

I don't mind the change _per se_, I mind that the change is causing breaking behavior but that the developer is not provided a clear error message to prompt a migration.

Edit: Since it's not listed explicitly, here would be a correct pyproject.toml file that would be consistent with black, flake8, etc.

[tool.black]
line-length = 88
target-version = ['py36', 'py37', 'py38']
'''

[build-system]
requires = ["setuptools>=41.0", "setuptools-scm", "wheel"]
build-backend = "setuptools.build_meta"

[pfmoore]

At the very least, if pip determines that a build error is related to this change, a simple message would help.

As I already noted, better docs or other information is welcome. In particular, if anyone wants to submit a PR providing a better message covering this situation, it would be very much appreciated. However, I hope you understand that the pip developers aren't deliberately trying to obscure the issue here - the reason there's no such message at the moment is because it's hard to detect this situation, so anyone contemplating writing a PR should appreciate that.

Anyway, as @pganssle noted, there isn't really anything actionable here, so I'm going to close the issue now.

[bluemellophone]

Fair enough, stay safe!