Poetry: Path dependencies with setup.py cannot be installed if setup.py imports `numpy` (or more general, imports any site-packages)

[hyliu1989]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10
• Poetry version: 1.1.4

Issue

I have searched in here and found #2638 is similar but not the same problem as I am facing. I have also tried a possible solution of [build-system].requires and consulted its documentation (as specified in #744) but no success. Here is a snapshot of the error message:

C:\Users\XXX\Desktop\test-module>poetry update -vvv
Updating dependencies
Resolving dependencies...

  PackageInfoError

  Unable to determine package info for path: C:\Users\XXX\Desktop\test-module\lib\my-dev-module

  Fallback egg_info generation failed.

  Command C:\Users\XXX\AppData\Local\Temp\tmpg5w9jw0j\.venv\Scripts\python.exe setup.py egg_info errored with the following return code 1, and output:
  Traceback (most recent call last):
    File "setup.py", line 11, in <module>
      import numpy
  ModuleNotFoundError: No module named 'numpy'

  at ~\.poetry\lib\poetry\inspection\info.py:503 in _pep517_metadata
      499│                     venv.run("python", "setup.py", "egg_info")
      500│                     return cls.from_metadata(path)
      501│                 except EnvCommandError as fbe:
      502│                     raise PackageInfoError(
    → 503│                         path, "Fallback egg_info generation failed.", fbe
      504│                     )
      505│                 finally:
      506│                     os.chdir(cwd.as_posix())
      507│

I will reuse its structure and show how the problem is reproduced.

The project structure is something like

test-module/
  + pyproject.toml
  + testmodule/
      + __init__.py
  + lib/
      + my-dev-module/
          + setup.py
          + mydevmodule/
              + __init__.py

The toml file is

[tool.poetry]
name = "testmodule"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.8"
numpy = "^1.19.1"
mydevmodule = {path = "./lib/my-dev-module", develop = true}

[build-system]
requires = ["numpy>=1.19.1,<1.20.0","poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

And ./lib/my-dev-module/setup.py is like

from distutils.core import setup
import numpy

if __name__ == "__main__":
    setup(
        name="mydevmodule",
        version="0.2",
        description="",
        author="me",
    )

The setup() in setup.py is a placeholder and the script does not get executed to setup(). It erred out at line 2 when importing numpy. In my actual application, module numpy is used to provide its C library for non-Python extension.

My question is: why does it fail at importing numpy? Shouldn't specifying [build-system].requires fix this? If not, I got that to fix this problem requires a [build-system].requires and a pyproject.toml in mydevmodule but that is not an ideal solution. Is there any other ways to specify the installation time requirements of a path-dependency module? Note that I have put numpy in both the dependencies and build-system requirement of the top-level project testmodule.

[sinoroc]

@hyliu1989

Looks to me like you might need a ./lib/my-dev-module/pyproject.toml file such as:

[build-system]
build-backend = 'setuptools.build_meta'
requires = [
    'setuptools >= 43.0.0',
    'numpy',
]

Although it might get complicated for the _editable_ installation...

[hyliu1989]

Yes that could help, but it is less ideal.

This means that all the previously written setup.py ~containing~ _importing_ site-packages would not work. I am not sure if many people encounter this problem but we can wait and see if this issue is severe.

[sinoroc]

setup.py containing site-packages

@hyliu1989 I do not understand what you mean.

---

Edit

Oh, you mean setup.py files that try to import modules or packages from the current environment? No, since there is _build isolation_, things like that do not work by default any more. But adding a pyproject.toml file as I have shown is easy and straightforward. When it can not be added, then most likely _build isolation_ can be deactivated for that particular case.

[hyliu1989]

Are there a discussion and documentations about build isolation? I would like to know how to deactivate it.

[sinoroc]

Are there a discussion and documentations about build isolation? I would like to know how to deactivate it.

@hyliu1989 I do not have a self contained link to suggest, nothing that comes to mind. You might have to read a bit everywhere.

• maybe _PEP 517_ is interesting for some related background
• with _pip_ one can use _pip install --no-build-isolation_, to deactivate it
• I do not know how to deactivate build isolation for _poetry_

[finswimmer]

@hyliu1989 your case is exactly why PEP 517 was invented: The need to know which packages are needed to build the package before starting the build process. So the way that @sinoroc shows above is the correct solution.

[hyliu1989]

@finswimmer I got that to fix this problem requires a [build-system].requires and a pyproject.toml in mydevmodule but that is not an ideal solution. It is forcing the developers of mydevmodule to adopt pyproject.toml while it has a complicated but working setup.py and a `requirements.txt.

[sinoroc]

@hyliu1989 You can add the minimal pyproject.toml without changing anything else. No modification to setup.py. Developers of the project can still use python setup.py develop or whatever other _setuptools_ commands they need. There should be no drawbacks when done correctly. But the big advantage of making sure that _numpy_ is available when _pip_ (or _poetry_ or any other build frontend) need to build that project.

[hyliu1989]

@sinoroc Thanks! Now I remember seeing that poetry says setup.py exists and it is going to use that setup.py. Having a pyproject.toml with minimal specification will be a solution to my question.

A follow-up question is: will poetry support running an existing setup.py in the foreseeable future? Is this behavior something poetry will get rid of?

[sinoroc]

That is the beauty of it. _poetry_ nor _pip_ (the build front ends) do not need to support _setuptools_ (or any other build back ends) directly. As long as they all agree to the _PEP 517_ standard (which is declared in the [build-system] section of pyproject.toml), then it will all work just fine.

In those cases, no need for poetry (or pip) to execute the setup.py directly. Instead they call the function declared in the build-backend field which in the case of _setuptools_ is setuptools.build_meta. And that function most likely reads and/or executes the setup.py (but this is an implementation detail of the backend, that the frontends don't need to know about).

I hope it is clear, let me know if some things need more details.

[sinoroc]

@hyliu1989

Now that I re-read your question:

will poetry support running an existing setup.py in the foreseeable future? Is this behavior something poetry will get rid of?

I think I missed the point of it. So allow me to rectify...

To be honest, I do not know if _poetry_ will forever be able to install projects that have a setup.py and are not PEP517 conform (i.e. no pyproject.toml). I am not a maintainer, so it is not my place to talk on this topic.

My personal opinion is: as much as I would like to get rid of those, there is such a huge amount of older pre-PEP517 projects (i.e. setup.py only) still in use, that it would make sense to keep this compatibility for as long as possible.

[finswimmer]

To be honest, I do not know if poetry will forever be able to install projects that have a setup.py and are not PEP517 conform (i.e. no pyproject.toml). I am not a maintainer, so it is not my place to talk on this topic.

There are currently no plans to drop the support for non-pyproject.toml projects. Furthermore PEP517 says "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:__legacy__ backend)."

[zoj613]

is there a way to install a local project that is only contained in a __init__.py file? I keep getting:

$ poetry install
Installing dependencies from lock file

No dependencies to install or update

  ValueError

  pyhtnorm is not a package.

  at ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/masonry/utils/package_include.py:73 in check_elements
      69│                 self._package = root.name
      70│                 self._elements = sorted(list(root.glob("**/*")))
      71│ 
      72│                 if not self.is_stub_only() and not self.has_modules():
    → 73│                     raise ValueError("{} is not a package.".format(root.name))
      74│ 
      75│                 self._is_package = True
      76│             else:

Anytime I use poetry install. It will only go away if there is a <project name>.py file in the source tree. I dont like the idea of having an empty .py file just to get poetry to work.

[sinoroc]

@zoj613 This is not related to the rest of the current discussion, is it? I think your query is similar to this one, please read that discussion. And if it does not correspond to what you want then I would recommend you to open your own new ticket (or come to _poetry_'s Discord chat channel).