Poetry: Poetry 1.0.10 -> 1.1.4: Files added to local repository are not referenced

[gitpushdashf]

Let's say you have an index.html file you want to put along your Python code.

$ find .
pyproject.toml
package
package/foo.py

You do poetry install like that, before index.html exists.

Then you add it.

$ find .
pyproject.toml
package
package/foo.py
package/index.html

Now with 1.1.4, foo.py cannot see a index.html next to it. In 1.0.10 this worked fine. If you run poetry install after adding the index.html file, it installs nothing new and does not do what I intend.

If the index.html file exists before the very first poetry install, it seems to be fine.

I presume something with the way the virtual environments are made has been changed.

[sinoroc]

Do you mind sharing the pyproject.toml file?

[gitpushdashf]

I'm working on this. I tried to make a minimal test case but it was working how I expected.

Our project is pretty complicated and uses sub-packages or what have you. It's also not yet open source.

[gitpushdashf]

It looks like we're actually using include to get those files that are missing going from 1.0.10 -> 1.1.4. I am wondering if the include is necessary at all. Will keep testing.

[gitpushdashf]

Ok, the structure is like this.

packages/package/package/web.py
packages/package/package/web/

Web includes assets from another repositroy, HTML files and such, that don't need to be in the source repository but do need to be in the published PIP package.

It looks like we're .gitignoring packages/package/package/web/, so that's why the include is there for packages/package/package/web/, to override that.

Do you think something might have changed with include/exclude override priority?

[gitpushdashf]

Ok, without the .gitignore line it's still not being included.

[gitpushdashf]

From what I can tell, the virtualenv has a package.egg-link made and package/. package/ is not updated on poetry install.

In the Poetry 1.0.10-based virtualenv, it had only the package.egg-link file.

[sinoroc]

You can't have a web directory and a web.py file in the same directory, because you can't have both a something.web package and a something.web module. Not sure if it's the cause of your issue, though.

I am not following what exactly is the structure of the project and it would be helpful to see the pyrpoject.toml or at least some parts of it.

[finswimmer]

Now with 1.1.4, foo.py cannot see a index.html next to it.

Could you provide a code example on how foo.py tries to see index.html?

[gitpushdashf]

web/ and web.py together worked fine. I did try without that, though.

I think web/__init__.py and web.py would conflict, however.

Basically like this:

import os

with os.path.join(os.path.dirname(__file__), "web", "index.html") as fp:
    print(fp.read())

[sinoroc]

I think web/__init__.py and web.py would conflict, however.

Yes, of course, that is what I mean. I thought it was obvious. This can not work.

For reading static files you need importlib.resources (or similar). Never rely on __file__ for such things. But again, you can not have a package and a module of the same name. I still do not understand what is the structure of your project and how the pyproject.toml file looks like.

[gitpushdashf]

Thank you for letting me know about importlib.resources. I'm adjusting the code to use it.

However, web still does not show up in importlib.resources.contents("package")

I'll see if I can make a repository that shows the same structure and hopefully reproduces the behavior.

[gitpushdashf]

I've opened a MR internally to convert our stuff to use importlib.resources -- thank you again for the tip.

I also have a minimal, working testcase.

https://github.com/gitpushdashf/issue3330/

Please see test.sh.

Thank you!

[sinoroc]

https://github.com/gitpushdashf/issue3330/

I do not understand the content of that repository. It seems like there are 2 projects, one inside the other. Why? Is this complicated structure with 2 projects relevant to the issue we are trying to solve here?

Anyway, I believe what you are missing is a package initializer packages/subpackage/subpackage/someresources/__init__.py.

[gitpushdashf]

This is basically just:

poetry new issue3330
mkdir issue3330/packages
cd issue3330/packages
poetry new subpackage

I've never had to have __init__.py for resources before. In 1.0.10 this worked fine.

someresources works because it already exists when you do poetry install, even without __init__.py.

https://github.com/gitpushdashf/issue3330/blob/master/test.sh#L9

However, if I mkdir somemoreresources and touch a file under that directory, it never shows up even after another poetry install. poetry install only installs resources the first time. If you add more resources after the first poetry install, for a subpackage, they are never added.

[sinoroc]

If there is no __init__.py, then it is not a Python package (it might be a namespace package, maybe but that is something else), and so importlib.resources might not be able to find it. It might be that somehow sometimes it works, but it is unreliable. Maybe it does work in newer versions of importlib_resources. I lost track of things. Anyway long story short: my recommendation is to always place an __init__.py to be on the safe side. Some reading:

• https://stackoverflow.com/a/58941536/
• https://gitlab.com/python-devs/importlib_resources/-/issues/58
• https://github.com/wimglenn/resources-example

---

I will look further into the "files added after the first install (build?) are not found (not actually installed?)".

[gitpushdashf]

Thank you for the information. Even with a __init__.py, I still get this behavior.

$ poetry run python3 packages/subpackage/subpackage/__init__.py
__init__.py
__pycache__
someresources
$ mkdir packages/subpackage/subpackage/evenmoreresources
$ touch packages/subpackage/subpackage/evenmoreresources/__init__.py
$ touch packages/subpackage/subpackage/evenmoreresources/index.html
$ poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: issue3330 (0.1.0)
$ poetry run python3 packages/subpackage/subpackage/__init__.py
__init__.py
__pycache__
someresources
$

[sinoroc]

As the console output shows: "_No dependencies to install or update_". So the new files are not installed. You would need to poetry run python -m pip uninstall subpackage, or bump the version number for subpackage.

But wait...

Read this: https://python-poetry.org/docs/dependency-specification/#path-dependencies

Before _poetry 1.1_ directory path dependencies were installed in editable mode by default. You should set the develop attribute explicitly, to make sure the behavior is the same for all poetry versions.

That is what changed for you between _1.0.10_ and _1.1.4_. I guess you relied on the _editable_ installation of _path_ dependencies. But now _path_ dependencies are not installed as _editable_ by default any more. So you would need something like that in your pyproject.toml:

[tool.poetry.dependencies]
python = "^3.8"
subpackage = {path = "packages/subpackage", develop = true}

[gitpushdashf]

That might just be it, thank you so much!

It does look like poetry lock doesn't update develop changes quite right, or so it was from my limited testing.

Will confirm if this is indeed working how I hope it will.

[gitpushdashf]

It works!!!

Actually, the issue I noticed was in converting an existing virtualenv from non-develop to develop (or I guess vice versa). Which, admittedly could be pretty dicey.

I'll close this out. Thank you so much for your help.

[sinoroc]

I'm glad! :)