Poetry: running inside tox uses unvendorized dependencies

[gaetano-guerriero]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian testing

• Poetry version: 0.12.11

Issue

I'm trying to use poetry inside tox following instructions in #222.

I think that when installing the local package inside virtualenv created by tox a new poetry is used using dependencies from surrounding virtualenv.

If the project has dependency conflicting with poetry dependencies the installation fails.

My conflicting dependency is jsonschema, jsonschema ^2.6 used by project is not compatible with jsonschema 3 used by poetry.

pyproject.toml

[tool.poetry]
name = "foo"
version = "0.1.0"
authors = "me"
[tool.poetry.dependencies]
python = "^2.7"
jsonschema = "^2.6"

[tool.poetry.dev-dependencies]
tox = "^3.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

tox.ini

[tox]
envlist = py27
skipdist = True

[testenv]
whitelist_externals =
    poetry
skip_install = true
commands =
    poetry install -vvv

$ poetry run tox -r 
py27 recreate: /tmp/foo/.tox/py27
py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,foo==0.1.0
py27 run-test-pre: PYTHONHASHSEED='2696339194'
py27 runtests: commands[0] | poetry install -vvv
Using virtualenv: /tmp/foo/.tox/py27
Installing dependencies from lock file


Package operations: 9 installs, 0 updates, 0 removals

  - Installing filelock (3.0.10)
  - Installing functools32 (3.2.3-2)
  - Installing pluggy (0.8.1)
  - Installing py (1.7.0)
  - Installing six (1.12.0)
  - Installing toml (0.10.0)
  - Installing virtualenv (16.3.0)
  - Installing jsonschema (2.6.0)
  - Installing tox (3.7.0)
  - Installing foo (0.1.0)

[EnvCommandError]                                                                                                       
Command ['/tmp/foo/.tox/py27/bin/pip', 'install', '-e', '/tmp/foo', '--no-deps'] errored with the following output:                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be main  
tained after that date. A future version of pip will drop support for Python 2.7.                                                    
Obtaining file:///tmp/foo                                                                                                            
  Installing build dependencies: started                                                                                             
  Installing build dependencies: finished with status 'done'                                                                         
  Getting requirements to build wheel: started                                                                                       
  Getting requirements to build wheel: finished with status 'error'                                                                  
  Complete output from command /tmp/foo/.tox/py27/bin/python /tmp/foo/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep51  
7/_in_process.py get_requires_for_build_wheel /tmp/tmpJlWvBQ:                                                                        
  Traceback (most recent call last):                                                                                                 
    File "/tmp/foo/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>             
      main()                                                                                                                         
    File "/tmp/foo/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main                 
      json_out['return_val'] = hook(**hook_input['kwargs'])                                                                          
    File "/tmp/foo/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_bui  
ld_wheel                                                                                                                             
      return hook(config_settings)                                                                                                   
    File "/tmp/pip-build-env-p66lJi/overlay/lib/python2.7/site-packages/poetry/masonry/api.py", line 23, in get_requires_for_build_  
wheel                                                                                                                                
      poetry = Poetry.create(".")                                                                                                    
    File "/tmp/pip-build-env-p66lJi/overlay/lib/python2.7/site-packages/poetry/poetry.py", line 106, in create                       
      cls.check(local_config)                                                                                                        
    File "/tmp/pip-build-env-p66lJi/overlay/lib/python2.7/site-packages/poetry/poetry.py", line 226, in check                        
      validation_errors = validate_object(config, "poetry-schema")                                                                   
    File "/tmp/pip-build-env-p66lJi/overlay/lib/python2.7/site-packages/poetry/json/__init__.py", line 25, in validate_object        
      validator = jsonschema.Draft7Validator(schema)                                                                                 
  AttributeError: 'module' object has no attribute 'Draft7Validator'                                                                 

  ----------------------------------------                                                                                           
Command "/tmp/foo/.tox/py27/bin/python /tmp/foo/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_r  
equires_for_build_wheel /tmp/tmpJlWvBQ" failed with error code 1 in /tmp/foo                                                         

Exception trace:
 /home/u/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/u/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/u/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/u/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /home/u/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/u/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/u/.poetry/lib/poetry/console/commands/install.py in handle() at line 89
   self.env.run("pip", "install", "-e", str(setup.parent), "--no-deps")
 /home/u/.poetry/lib/poetry/utils/env.py in run() at line 492
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /home/u/.poetry/lib/poetry/utils/env.py in run() at line 354
   raise EnvCommandError(e)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

ERROR: InvocationError for command '/home/u/.poetry/bin/poetry install -vvv' (exited with code 1)
_______________________________________________________________ summary _______________________________________________________________
ERROR:   py27: commands failed

[XayOn]

I confirm this bug happens.

[Alexspayne]

This also happens to me. I have not found a workaround yet, it would be helpful if this was fixed so that I could use poetry as the build-system.

[sdispater]

You should use isolated builds as explained in the documentation: https://poetry.eustace.io/docs/faq/#is-tox-supported

[gaetano-guerriero]

Tried as doc suggests:

pyproject.yaml

as above

tox.ini

[tox]
envlist = py27
isolated_build = true
skipdist = true

[testenv]
whitelist_externals =
    poetry
commands =
    poetry install -vvv
    poetry python foo.py

Inside environment used to build local package jsonschema 2.6 is still installed and poetry uses it:

$ poetry run tox
.package create: /tmp/foo/.tox/.package
.package installdeps: poetry>=0.12
ERROR: invocation failed (exit code 1), logfile: /tmp/foo/.tox/.package/log/.package-4.log
ERROR: actionid: .package
msg: perform-isolated-build
cmdargs: '/tmp/foo/.tox/.package/bin/python -c \'\nimport sys\nimport poetry.masonry.api\nbasename = poetry.masonry.api.build_sdist(\'"\'"\'/tmp/foo/.tox/dist\'"\'"\', { "--global-option": ["--formats=gztar"]})\nprint(basename)\''

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/tmp/foo/.tox/.package/lib/python2.7/site-packages/poetry/masonry/api.py", line 47, in build_sdist
    poetry = Poetry.create(".")
  File "/tmp/foo/.tox/.package/lib/python2.7/site-packages/poetry/poetry.py", line 106, in create
    cls.check(local_config)
  File "/tmp/foo/.tox/.package/lib/python2.7/site-packages/poetry/poetry.py", line 226, in check
    validation_errors = validate_object(config, "poetry-schema")
  File "/tmp/foo/.tox/.package/lib/python2.7/site-packages/poetry/json/__init__.py", line 25, in validate_object
    validator = jsonschema.Draft7Validator(schema)
AttributeError: 'module' object has no attribute 'Draft7Validator'

ERROR: FAIL could not package project - v = InvocationError('/tmp/foo/.tox/.package/bin/python -c \nimport sys\nimport poetry.masonry.api\nbasename = poetry.masonry.api.build_sdist(\'/tmp/foo/.tox/dist\', { "--global-option": ["--formats=gztar"]})\nprint(basename) (see /tmp/foo/.tox/.package/log/.package-4.log)', 1)

[wfscheper]

I believe the issue here is that poetry doesn't have a way to specify build dependencies separate from runtime dependencies. Here's the problematic code:

def get_requires_for_build_wheel(config_settings=None):
    """
    Returns a list of requirements for building, as strings
    """
    poetry = Poetry.create(".")

    main, _ = SdistBuilder.convert_dependencies(poetry.package, poetry.package.requires)

    return main

Ideally, poetry would either have a separate section for build dependencies, aladev-dependencies, or a way to flag a dependency as a build dep, ala allows-prerelease.

[aizelauna]

I have a very similar issue.
The strange thing is that I have configured three environments (py27, py36, py37) in _tox.ini_ file and only the 2.7 fails.
My exception log is slightly different:

$ tox
py27 inst-nodeps: /tmp/tools/.tox/.tmp/package/1/tools-0.1.0.tar.gz
ERROR: invocation failed (exit code 1), logfile: /tmp/tools/.tox/py27/log/py27-3.log
============================================= log start =============================================
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Processing ./.tox/.tmp/package/1/tools-0.1.0.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Complete output from command /tmp/tools/.tox/py27/bin/python /tmp/tools/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp7HehbM:
  ERROR: Traceback (most recent call last):
    File "/tmp/tools/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/tmp/tools/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tools/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-3THB7E/overlay/lib/python2.7/site-packages/poetry/masonry/api.py", line 23, in get_requires_for_build_wheel
      poetry = Poetry.create(".")
    File "/tmp/pip-build-env-3THB7E/overlay/lib/python2.7/site-packages/poetry/poetry.py", line 106, in create
      cls.check(local_config)
    File "/tmp/pip-build-env-3THB7E/overlay/lib/python2.7/site-packages/poetry/poetry.py", line 226, in check
      validation_errors = validate_object(config, "poetry-schema")
    File "/tmp/pip-build-env-3THB7E/overlay/lib/python2.7/site-packages/poetry/json/__init__.py", line 25, in validate_object
      validator = jsonschema.Draft7Validator(schema)
  AttributeError: 'module' object has no attribute 'Draft7Validator'
  ----------------------------------------
ERROR: Command "/tmp/tools/.tox/py27/bin/python /tmp/tools/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp7HehbM" failed with error code 1 in /tmp/pip-req-build-1SMhgE

============================================== log end ==============================================
______________________________________________ summary ______________________________________________
ERROR:   py27: InvocationError for command /tmp/tools/.tox/py27/bin/python -m pip install --no-deps -U .tox/.tmp/package/1/tools-0.1.0.tar.gz (exited with code 1)

With this log, it appears that pip uses poetry from a temporary folder /tmp/pip-build-env-3THB7E/overlay/lib/python2.7/site-packages/poetry and not the one installed with the preferred procedure.

[funkyfuture]

I think that when installing the local package inside virtualenv created by tox a new poetry is used using dependencies from surrounding virtualenv.

i tried upgrading jsonschema of the 'surrounding' virtual environment, but 2.6 will still be installed in the .tox/.package venv. the same is true when no virtual environment is active and jsonschema isn't available at all. it still yields this error.

i also specified this in the pyproject.toml, but with the same result:

[build-system]
requires = ["poetry>=0.12.16", "jsonschema>=3.0"]
build-backend = "poetry.masonry.api"

thus, i wouldn't be sure whether an 'outer' jsonschema is causing this. i'm currently motivated to dig into this, so any proposals where or what to dig for would be appreciated.

[funkyfuture]

for what it's worth, switching to nox and using an 'outside' poetry installation is not an option.

edited after i noticed that the wrong interpreter is used.

[jimmyharris]

I had a similar problem with a project based around docker-compose. Docker compose requires jsonschema < 3 in its current release, so I had to downgrade my in-envrionment poetry install to 0.11.5 (the last version released that did not have the higher jsonschema requirement. It makes me wonder if moving to the newest jsonschema for 0.12 was really a wise idea.

[pmav99]

@jimmyharris docker-compose, like poetry itself should be installed in a different python distribution than the one you are developing. That distribution might be the OS one, or a different virtualenv. I usually install docker-compose using pipx

[jimmyharris]

Generally this is true but I am actually building a project that extends docker compose programmatically and i can’t guarantee that my tools will be correctly installed my ci group.

[pikacic]

I just stumbled on this problem when trying to enable tox integration in my poetry managed project.

In my case the issue comes from the dependency on cleo = ^0.7.

The first failure is when, preparing the build venv, tox asks poetry which are the build dependencies of my project (in this case "cleo>=7.0,cleo<0.8", but poetry requires "cleo<0.7"), then tox upgrades the version of cleo in the same venv, and that makes the previously installed poetry unusable.

I tried to follow up on @wfscheper suggestion, by making get_requires_for_build_wheel to return an empty list (after all I only need poetry and its dependencies to build the wheel), but it's not enough, because now I get a failure in the testenv (before it was in the buildenv).
This time the error message in the log file is a bit more useful:

ERROR: Some build dependencies for file:///some-path/test-848/.tox/.tmp/package/1/test-848-0.1.0.tar.gz conflict with the backend dependencies: cleo==0.6.8 is incompatible with cleo<0.8.0,>=0.7.6.

So I deduce tox requires the build backend and the project to have compatible dependencies, even if we require isolated_builds = true.

I would suggest, at least, to clarify this point in the documentation.

[pikacic]

Actually I just realized it's not tox that imposes that build and runtime dependencies have to be compatible: it's pip itself.

I just tried to call pip install . on my project (I only have a pyproject.toml and no setup.py) and I got:

pip._internal.exceptions.InstallationError: Some build dependencies for file:///home/marco/Devel/workspace/nightlies-jenkins-scripts/lb-jenkins conflict with the backend dependencies: cleo==0.6.8 is incompatible with cleo<0.8.0,>=0.7.6.

This also makes my problem similar to #826 too.

[schinckel]

I believe that the suggested way of installing using poetry in tox does not install to an isolated environment: I'm not sure if this is what is causing this issue, but I think it exposes some issues about the way poetry works, that mean it cannot work with tox to ensure a fully isolated test run.

Instead of installing packages to the tox environment that is currently active, it installs to the same environment that is used in development. Whilst this works, it does not feel like it is the correct approach - the point of tox is that it is an isolated test environment, and thus should be an isolated virtual environment too.

It is possible to indicate to poetry that it should not _create_ a virtual environment: but this does not appear to change if poetry will _use_ a virtual environment - in my tox test runs it still installs to the same location (or, more specifically, indicates that it doesn't need to install).

Likewise, there is a setting that indicates that it should use an "in-project" virtualenv. However, this appears to not work either: it still installs to my "development" virtualenv.

But, setting config values from within a tox environment test run is not a satisfactory pathway when using poetry anyway: all poetry config ... commands write to the user's global poetry settings: there does not appear to be a way to have configuration that only applies to this current execution of poetry.

Pipenv avoids this problem by automatically installing to the current virtualenv if it detects one: but I think an alternative solution could be to allow setting which virtualenv should be used to install to.

[sinoroc]

Issues #1745 and #1941 might be related.

From my point of view having poetry install and poetry run in the _tox_ commands does not make much sense, since _tox_ already takes care of installing the project (via the _sdist_ built thanks to isolated_build = true) and its dependencies in a virtual environment. If somehow _tox_ had a a way to read and install the dev-dependencies instead of the _run-time_ dependencies then the integration would be much smoother. Maybe the development dependencies should simply be listed in an extra instead.

[finswimmer]

The original issue should not exists anymore. That's why I'm closing this thread.