Poetry: [FAQ]: The tox answer should recommend `poetry install --no-root` instead of `poetry install`

[thejohnfreeman]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

In the FAQ, there is a recommendation for how to use tox with Poetry. It suggests this tox.ini:

[tox]
isolated_build = true
envlist = py27, py36

[testenv]
whitelist_externals = poetry
commands =
    poetry install -v
    poetry run pytest tests/

With this configuration, when a user runs tox:

1. tox packages the project with Poetry.
2. tox creates a virtual environment.
3. tox installs the package it built, but not its dependencies (inst-nodeps).
4. tox calls poetry install, which installs the project dependencies, and then installs the project, as editable, overwriting what tox installed in step 3.
5. tox calls poetry run pytest tests which tests against the package as it exists in the working directory, not as it was packaged by Poetry.

To fix this, the recommended commands should be:

commands =
    poetry install --no-root -v
    poetry run pytest tests/

[sinoroc]

I believe _poetry_ should be left out of tox.ini entirely. This is the pattern I would recommend:

tox.ini

[tox]
envlist = py{36,37,38}-django{22,30}
isolated_build = True

[testenv]
# ...
deps =
    django22: Django==2.2
    django30: Django==3.0
extras =
    test
commands =
    pytest

pyproject.toml

[tool.poetry]
# ...

[tool.poetry.dependencies]
python = "^3.6"
django = "^2.2 || ^3.0"
#
pytest = { version = "^5.2", optional = true }

[tool.poetry.extras]
test = ["pytest"]

[build-system]
# ...

[Alexspayne]

I believe _poetry_ should be left out of tox.ini entirely. This is the pattern I would recommend:

tox.ini

[tox]
envlist = py{36,37,38}-django{22,30}
isolated_build = True

[testenv]
# ...
deps =
    django22: Django==2.2
    django30: Django==3.0
extras =
    test
commands =
    pytest

pyproject.toml

[tool.poetry]
# ...

[tool.poetry.dependencies]
python = "^3.6"
django = "^2.2 || ^3.0"
#
pytest = { version = "^5.2", optional = true }

[tool.poetry.extras]
test = ["pytest"]

[build-system]
# ...

You will have maintain duplicate lists of your dependencies if you do that, and you might not get the same version of the dependencies in the tox environment as you would in production because it wouldn't be honoring the poetry.lock file.

[sinoroc]

In this workflow scenario the dev-dependencies section is explicitly left out in favor of the test extra in order to avoid having to maintain two lists of development dependencies. The _run-time_ dependencies are listed as usual.

The _tox_ tool is meant to test a library or application against multiple combinations of different versions of the Python interpreter and different versions of the dependencies. It is explicitly not intended to test just one combination. If it is your case, you probably don't need _tox_ to begin with. Unless you are both the developer/maintainer and sole user of the project (application or library) and you can allow yourself to exactly pin the version numbers of all dependencies, there is little control over which versions of the other packages are installed in the environment your project runs in.

I believe it is the responsibility of the user (integrator, system administrator) to pin the versions of the dependencies and thoroughly test this exact combination. Not the responsibility of the maintainers of the project. This is the kind of workflow scenario this suggestion is targeted at.

A scenario where the solution currently presented in the FAQ could make sense (everything is pinned in poetry.lock) could be in the case of an application distributed as a whole with all dependencies included, such as _pex_, _shiv_, _zipapp_, or anything similar. But there is no need for _tox_ to test such a static environment.

[thejohnfreeman]

I'm totally misunderstanding something then. How can I use tox or another tool to verify that poetry build packages my library correctly?

[sinoroc]

I'm totally misunderstanding something then. How can I use tox or another tool to verify that poetry build packages my library correctly?

Well, _tox_ does exactly that. The first thing _tox_ does is package a _source distribution_ (_sdist_) of the project. Then it creates a _virtual environment_ and installs the _sdist_ in it. So _tox_ executes the commands against the project as it is installed from a distribution of it, and not as an _editable_ or _development_ version of it (like poetry install does).

When isolated_build is set, _tox_ knows to check for the build-system section of pyproject.toml in order to build the _sdist_. I believe it is safe to assume that executing the poetry.masonry.api function that is set as value for build-backend leads to the same result as executing poetry build.

To also run some tests against the project as installed from its _wheel_ distribution, one could look into the _tox-wheel_ plugin or something similar.

Additionally I would also recommend doing something like the following to check the distributions with _twine_:

[testenv:package]
skip_install = True
deps =
    twine
commands =
    python3 -m poetry build
    python3 -m twine check dist/*

And it also goes without _tox_:

$ poetry add --dev twine
$ poetry build
$ poetry run twine check dist/*

[igor47]

A scenario where the solution currently presented in the FAQ could make sense (everything is pinned in poetry.lock) could be in the case of an application distributed as a whole with all dependencies included, such as _pex_, _shiv_, _zipapp_, or anything similar. But there is no need for _tox_ to test such a static environment.

i maintain some applications distributed as a whole, with all dependencies included (if by distributed, you mean packaged into docker containers to be served to users). i still like using tox, because you can specify multiple test commands. i tell other developers of the application to just run tox, and they get a run of pytest, flake8, mypy, black, etc... without tox, they would need to run each of these separately. by having developers as well as CI just invoke tox, i get consistency in testing between local and CI environments.

so, i believe that there is still utility in using tox, even when what i want tox to do is to install just the things in my poetry.lock. given that tox with poetry seems to ignore poetry.lock, i think this comment from @sinoroc might be incorrect at least in my use case:

I believe poetry should be left out of tox.ini entirely.

[sinoroc]

i believe that there is still utility in using tox

@igor47 My point is: Do use tox, but don't call _poetry_ from inside tox.ini. The reason is that otherwise _poetry_ cancels some of the work that _tox_ does.

even when what i want tox to do is to install just the things in my poetry.lock. given that tox with poetry seems to ignore poetry.lock

The main reason for using _tox_ is to test one's project against a wide range of combinations of dependencies and Python interpreters. So testing against the one and only combination registered in poetry.lock seems to me like it goes against this philosophy.

i maintain some applications distributed as a whole, with all dependencies included (if by distributed, you mean packaged into docker containers to be served to users). i still like using tox, because you can specify multiple test commands. i tell other developers of the application to just run tox, and they get a run of pytest, flake8, mypy, black, etc... without tox, they would need to run each of these separately. by having developers as well as CI just invoke tox, i get consistency in testing between local and CI environments.

Now I am open to admit that for some projects it might be the sensible thing to test only against one version of Python interpreter (in that case, the one in the _Docker_ container) and the combination of dependencies registered in poetry.lock, and _tox_ could help do such a thing, but I would still argue that in such a case it's not the right tool for the job. Maybe something like _pyinvoke_ for example might be better suited to such a scenario (poetry run pyinvoke format lint test).

[schinckel]

Tox is not just useful for testing combinations of different versions of packages or python interpreters.

We use tox for automation: seperate tox environments are used to have things like Robot Framework tests run, tests of JS, pg_tap tests for testing database functions. Also, having tox environments for flake8, isort and friends (which can finish much faster than the full test suite) means that those can report failures much faster than waiting for all tests to run.

It's also very useful for managing dependencies between test environments (ie, runs the "clean" environment before anything else, and the "code-coverage" collector environment after all other environments).

[sinoroc]

Tox is not just useful for testing combinations of different versions of packages or python interpreters.

[...]

@schinckel Those are all legitimate use cases for _tox_ indeed, but... What would that mean for the current topic? What would be your recommendation? What should be written in the FAQ? In short, which tool should handle the creation of the virtual environments, the installation of the project, of its dependencies? Should it be _tox_ or _poetry_?

[jlumbroso]

Thanks to everyone for the conversation.

I agree with @sinoroc and others on the principle, that the whole goal of tox (and similar tools) is "_What would happen in similar configurations I don't have exact control over?_" In that sense, as package creators, we can choose to use poetry, but there is no guarantee that will be used by the end user.

However, the goal of poetry the way I understand it is precisely to abstract all these config tweaking/hacking problems away and to provide A Good Solution to all common package managing issues—in part just to make it easier for package creators to do the right thing and adhere to best practices. For instance, that's why poetry provides the pytest stub in new projects. Not everybody needs to have testing, or even use pytest ❤️, but it's a good choice and the default makes it easier.

In that sense, I agree with @thejohnfreeman, @Alexspayne and @schinckel, that there is something sub-optimal about the current answers.

In another issue, @wesleykendall provided an excellent alternate solution that is: Automatically export dependencies to a common API that tox understands without poetry's help (i.e., the requirements.txt format). I added a suggestion that this export happen at commit-time rather than run/compile/build-time so that poetry is not a dependency just to export the deps list.

Other issues re: tox/poetry: https://github.com/python-poetry/poetry/issues/848, https://github.com/python-poetry/poetry/issues/1745, https://github.com/python-poetry/poetry/issues/1941.

[thejohnfreeman]

I left a thumbs up on @sinoroc's explanation, but never explicitly said that he changed my opinion on this subject, after I copied his suggested pattern with success. I was unaware that tox would build the package with Poetry via the [build-system] section of pyproject.toml. I thought it wasn't, and that that was the reason it had to be hacked in with the pattern presently suggested in the documentation. Maybe that documentation was written at a time before tox was made aware of [build-system], when the pattern it recommends was necessary.

[sinoroc]

I was unaware that tox would build the package with Poetry via the [build-system] section of pyproject.toml. I thought it wasn't, and that that was the reason it had to be hacked in with the pattern presently suggested in the documentation. Maybe that documentation was written at a time before tox was made aware of [build-system], when the pattern it recommends was necessary.

This is enabled by the isolated_build = true setting. This lets _tox_ know to look into pyproject.toml's build-system section, and delegates the build to the proper build backend, which in the case here is _poetry_ obviously. This was always (as far as I can remember at least), clearly the case in both _FAQ_'s (_poetry_ and _tox_), but then _poetry_'s FAQ muddies the water by recommending commands = poetry install, which is a step back: it destroys the work done by isolated_build.

This _isolated build_ feature is the new standard (look up _PEP 517_ and _PEP 518_), works also with _setuptools_, _flit_, etc.

The missing element here is about the _development dependencies_. How to let _tox_ know what the _dev dependencies_ are?

This can be solved by using the _extras_ (which is a standard, i.e. specified by a _PEP_), this pattern is quite common nowadays and is guaranteed to work across many different tools (not only _tox_, and _poetry_).

On the other hand, generating a requirements.txt file (not a standard, there is no _PEP_ specification) dynamically, feels unnecessary to me. And _poetry_'s own dev-dependencies field is not a standard either, so _tox_ doesn't know how to handle it.

But again, if one wants to go all-in with _poetry_ and use all its features (even the non standard ones), feel free. But it means that you will have to work harder to make it cooperate with other tools that do follow standards. Which could be fine, maybe there isn't even need for _tox_ to begin with. Just running poetry run pytest could be perfectly good, no need for more.

So my point is: in my opinion _poetry_'s FAQ on the topic of _tox_ is misleading and shows a largely sub-optimal pattern.

---

I have been thinking for a while about writing a plugin for _tox_ that would offer _tox_ a way to automatically pick up _poetry_'s _dev dependencies_. Maybe that would put this issue to rest. Or maybe there's already such a plugin? If you know of one, let us know.

[fredrikaverpil]

I'm hoping to be able to specify dependency groups in my CI once this is available: https://github.com/python-poetry/poetry/issues/1644 (planned for poetry 1.2)

But until then, what _is_ the recommended usage of poetry with tox when you need to install dev dependencies for e.g. linting/pytest or the other usual CI suspects?

I've noticed that poetry's pyproject.toml itself differs from what the FAQ says on the build system declaration:
https://github.com/python-poetry/poetry/blob/master/pyproject.toml#L86
So right now I'm using that in conjunction with "poetry install" prior to my tox commands.

I'm also contemplating whether I should try to use tox-current-env right now too, as all my tox envs end up identical in terms of dependencies but takes ages to build. And no I cannot use poetry's extras...

Ping @abn @sdispater can someone please explain what is the current recommendation?

[jlumbroso]

@sinoroc

This can be solved by using the extras (which is a standard, i.e. specified by a PEP), this pattern is quite common nowadays and is guaranteed to work across many different tools (not only tox, and poetry).

On the other hand, generating a requirements.txt file (not a standard, there is no PEP specification) dynamically, feels unnecessary to me. And poetry's own dev-dependencies field is not a standard either, so tox doesn't know how to handle it.

Generating a requirements.txt feels unnecessary to you—and using an "extras" to specify dev dependencies when there is another more specific mechanism designed explicitly to do so (i.e., dev-dependencies, literally), that's what feels redundant/unnecessary to me.

I have been thinking for a while about writing a plugin for tox that would offer tox a way to automatically pick up poetry's dev dependencies. Maybe that would put this issue to rest. Or maybe there's already such a plugin? If you know of one, let us know.

I knew of no such plugin, however some due-diligence Googling led me to discover a very unusual effort:
https://github.com/tkukushkin/tox-poetry
https://pypi.org/project/tox-poetry/

There are some things that are unusual: The PyPi repo link is pointing to a different repo. The test coverage is 0%. The project seems to have been forked (a bit sloppily) from tox-pipenv-install (probably a good initial starter code).

I think it would be an excellent idea to have a reliable plugin.

[jlumbroso]

@fredrikaverpil

I'm hoping to be able to specify dependency groups in my CI once this is available: #1644 (planned for poetry 1.2)

But until then, what _is_ the recommended usage of poetry with tox when you need to install dev dependencies for e.g. linting/pytest or the other usual CI suspects? [...] And no I cannot use poetry's extras...

I agree that using extras is not ideal.

So here's what I have done in my most recent package, comma (Python CSV for human!! 😇):

• I have my dev-dependencies specified in pyproject.toml
• I have my CI automatically update requirements.txt and requirements-dev.txt on any commit that changes dependencies
• [ Currently I use GitHub Actions as CI, and a big great shoutout to @abatilo for the super-convenient and foolproof [actions-poetry](https://github.com/abatilo/actions-poetry) ]
• Then I have configured my tox.ini to simply be:
  ```ini
  [tox]
  isolated_build = true
  envlist = py{36,37,38,39}

[testenv]
deps = -rrequirements-dev.txt
commands =
    pytest
```

This works wonderfully and allowed me to expand the compatibility of my package to versions different than my own (Py 3.8 right now). What I like about this solution is that I still only specify everything in one place, pyproject.toml, the CI pipeline robustly takes care of everything else.

My next steps are:

• take advantage of the fact that you can embed the legacy tox.ini file description within pyproject.toml to reduce the number of configuration files;
• have tox be run on commit (right now, the regression tests built on commit only test Py 3.8).

[ If you find this useful, I would appreciate [a ⭐ on my package 😅](https://github.com/jlumbroso/comma/). ]

[thejohnfreeman]

One of the reasons I'm excited about PEP 518 (pyproject.toml) is that I don't have to keep a requirements.txt (much less two of them) in my codebase, confusing newcomers and adding an operational burden (CI job just to keep it synchronized).

Extras has good support across Python tools, so I have switched to using it instead of [dev-dependencies], a Poetry-specific extension not found in PEP 518 (or any other PEP to my knowledge).

[jlumbroso]

Good points @thejohnfreeman.

Although I agree with you, I think the practice of having requirements-dev.txt widespread in our community enough that GitHub automatically parses that file to provide metadata about a repository, which I think is important for many reasons—for instance it helps us track usage, it helps us track old dependencies, security vulnerabilities, etc..

But personally I agree it makes more sense to use an existing mechanism for specifying an additional subset of dependencies. It would be nice if there was some PEP just to say "the best way to provide development dependencies is an 'extra' called 'dev'" or something like that. I wonder what the process would be for that to be suggested. Any ideas?

[sinoroc]

It would be nice if there was some PEP just to say "the best way to provide development dependencies is an 'extra' called 'dev'" or something like that.

Agreed. From my point of view it should be a convention (a recommendation), not a specification (a rule).

I wonder what the process would be for that to be suggested. Any ideas?

Maybe. _PyPA_ is in the process of writing new _PEPs_, one of them in particular _PEP 621_ proposes to specify the way project's metadata should be stored in pyproject.toml files. The discussion is a bit on pause since a few days, but still open, so it's probably the right time to make such suggestions.

You might want to read these two threads first:

• https://discuss.python.org/t/pep-621-storing-project-metadata-in-pyproject-toml/4513
• https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599

And then open a new thread on that forum to make your suggestion. But no guarantee: it might be considered out of scope for a _PEP_.

[sinoroc]

I have been thinking for a while about writing a plugin for _tox_ that would offer _tox_ a way to automatically pick up _poetry_'s _dev dependencies_. Maybe that would put this issue to rest.

I published on _PyPI_ a plugin for _Tox_ to instruct it to install _Poetry_'s development dependencies in the test environments: tox-poetry-dev-dependencies. It's just a proof of concept. I didn't test much, only with a very simple project.

[jlumbroso]

Very exciting @sinoroc — can't wait to try it out!!

[thejohnfreeman]

I'd rather move away from Poetry [tool.poetry.dev-dependencies] and toward extras (or dependency groups when they become available). Right now, I structure my projects with at least 3 extras:

• test: dependencies necessary to run the tests: pytest, hypothesis, ...
• docs: dependencies necessary to build the documentation: sphinx, sphinx_rtd_theme
• dev: dependencies used only when writing code: yapf, sphinx-autobuild, python-language-server, ...

Tox only needs to install test. Read The Docs only needs to install docs. My development environment installs everything. If Poetry is going to install any extra dependencies by default, it should let me pick a set of extras.

[sinoroc]

I'd rather move away from Poetry [tool.poetry.dev-dependencies] and toward extras (or dependency groups when they become available). Right now, I structure my projects with at least 3 extras _[...]_

I also prefer _extras_ for my projects, (see the very code of tox-poetry-dev-dependencies for example). If I am not mistaken, in your case one of the reasons might be that you want to split your _development dependencies_ over multiple categories (lint, test, and doc for example), while (for now) _Poetry_ only offers one single group (dev-dependencies). But for some projects, one category might be enough. Poetry's _dependency groups_ if they are released might indeed be a better fit for your use cases, but most likely that would still require a plugin to integrate them with _Tox_. So here we are, there is a plugin now for those projects that use Poetry's dev-dependencies, and when/if the time comes I guess it could be extended to support Poetry's dependency groups as well.

[layday]

One issue with using extras for development is that they make part of the distribution. What this means in practice is that you cannot install development dependencies from source (VCS or URL), otherwise your package cannot be published on PyPI.

[sinoroc]

One issue with using extras for development is that they make part of the distribution. What this means in practice is that you cannot install development dependencies from source (VCS or URL), otherwise your package cannot be published on PyPI.

I am not entirely sure I understood.

What does it have to do with _PyPI_? Usually _Tox_ installs (the project being tested) from the _source distribution_ it just built itself, and not from _PyPI_. Am I missing something?

Would one of the following suggestions work for your use cases?

---

A.

pyproject.toml

# ...
[tool.poetry.dependencies]
mypy = {git = "https://github.com/python/mypy.git", optional = true}
pytest = {version = "*", optional = true}

[tool.poetry.extras]
dev_test = ["mypy", "pytest"]
# ...

tox.ini

# ...
[testenv]
extras =
    dev_test
# ...

---

B.

pyproject.toml

# ...
[tool.poetry.dependencies]
mypy = {version = "*", optional = true}
pytest = {version = "*", optional = true}

[tool.poetry.extras]
dev_test = ["mypy", "pytest"]
# ...

tox.ini

# ...
[testenv]
deps =
    mypy @ git+https://github.com/python/mypy.git
extras =
    dev_test
# ...

---

Thinking about it, it could be that _PyPI_ would reject version A. Or maybe _PyPI_ would accept it, and then _pip_ would fail or (most likely) simply ignore the URL for _mypy_. I do not know of the top of my head.

[layday]

Version A will be rejected by pip when installing, either from a wheel or from source, and by PyPI when uploading a wheel:

$ twine upload --repository testpypi dist/sampleproject_direct_url_reference-2.0.0-py3-none-any.whl
[...]
HTTPError: 400 Bad Request from https://test.pypi.org/legacy/
Invalid value for requires_dist. Error: Can't have direct dependency: "mypy @ git+https://github.com/python/mypy.git ; extra == 'dev_test'"

[sinoroc]

Version A will be rejected by pip when installing, either from a wheel or from source, and by PyPI when uploading a wheel:

Thanks for the confirmation that in version A the upload to _PyPI_ fails.

Although, I must say, that for me _pip_ was able to install version A from its wheel (local file system, not _PyPI_) and from its source directory (again local file system, obviously).

Would version B work for you? It requires specifying some of the development dependencies twice, but only those that require a direct URL. That would be the best compromise I can think of right now.

[layday]

I personally use Nox, but version B would work. Just a little gotcha to bear in mind. I can't imagine it crops up often.

[DustinMoriarty]

All of this seems like a work around to me. Is there something that needs to change on the tox side to make tox set up poetry environments correctly? In theory, it seems that the tox.ini should look the same for a poetry package as it would for a package using setup.py. I don't put "pip install" in a tox.ini for a setuptools project. If there are things that need to be fixed on the tox side, perhaps we can focus our efforts there.

Sometimes, I have been able to get this to work, but other times it has not. I have yet to chase down all the different system configuration effects to understand why it does not always work. It seems that the goal should be that this would be all that would be needed.

[tox]
envlist = py36,py37,py38
isolated_build = True

[testenv]
deps = pytest
commands = pytest