poetry doesn't install extras

[devkral]

setup:
only a pytoml file with the new build system

to reproduce:
poetry install -E foo
or
poetry install --extras "foo"
or
pip install ".[foo]"

result:
the dependencies of the foo group won't be installed

I think the issue here is that the extras system stops to work if no setup.py bails out.

[bersace]

I confirm this with poetry 1.0.0a3.

[bersace]

I think there is a bug around option handling. cf. #1141

[paveldedik]

I had the same issue, but it seems to work correctly if I specify the extra dependencies in the tool.poetry.dependencies section instead of the dev-dependencies section which seems to be broken.

Can any of you confirm this behavior?

[dgilge]

In my case it sometimes works and sometimes not. They are listed in tool.poetry.dependencies.
Poetry 0.12.16 and 1.0.0a4.

[thedrow]

Same issue here with latest.

[mrezzamoradi]

I have this issue when installing from a password-protected private repo which does not have the package:

[[tool.poetry.source]]
name = "private"
url = "https://private.repo.com/simple"

[tool.poetry.dependencies]
python = "^3.7"
scikit-learn = {version = "0.19.0",extras = ["alldeps"]}

and then:

$ poetry update
Updating dependencies
Resolving dependencies... (18.2s)


Package operations: 5 installs, 0 updates, 0 removals

  - Installing scikit-learn (0.19.0)

[EnvCommandError]                                                                                                                                        
Command ['/home/reza/projects/virtualenvs/nava_37/bin/python', '-m', 'pip', 'install', '--no-deps', '--index-url', 'https://xxxx:[email protected]/simple',        
 'scikit-learn==0.19.0'] errored with the following return code 1, and output:                                                                                        
Looking in indexes: https://xxxx:****@private.repo.com/simple                                                                                                         
Collecting scikit-learn==0.19.0                                                                                                                                       
  Using cached https://files.pythonhosted.org/packages/c2/38/e3b0333e661ab411545583cb24940223917fe7ffc9c68a77730dce3b10b0/scikit-learn-0.19.0.tar.gz                  
Installing collected packages: scikit-learn                                                                                                                           
  Running setup.py install for scikit-learn: started                                                                                                                  
    Running setup.py install for scikit-learn: finished with status 'error'                                                                                           
    ERROR: Command errored out with exit status 1:                                                                                                                    
     command: /home/reza/projects/virtualenvs/nava_37/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pifc_oyo/scikit-learn/                
setup.py'"'"'; __file__='"'"'/tmp/pip-install-pifc_oyo/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\  
n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-s3itmoyt/install-record.txt --single-version-externa  
lly-managed --compile --install-headers /home/reza/projects/virtualenvs/nava_37/include/site/python3.7/scikit-learn                                                   
         cwd: /tmp/pip-install-pifc_oyo/scikit-learn/                                                                                                                 
    Complete output (19 lines):                                                                                                                                       
    Partial import of sklearn during the build process.                                                                                                               
    Traceback (most recent call last):                                                                                                                                
      File "/tmp/pip-install-pifc_oyo/scikit-learn/setup.py", line 168, in get_numpy_status                                                                           
        import numpy                                                                                                                                                  
    ModuleNotFoundError: No module named 'numpy'                                                                                                                      
    Traceback (most recent call last):                                                                                                                                
      File "/tmp/pip-install-pifc_oyo/scikit-learn/setup.py", line 148, in get_scipy_status                                                                           
        import scipy                                                                                                                                                  
    ModuleNotFoundError: No module named 'scipy'                                                                                                                      
    Traceback (most recent call last):                                                                                                                                
      File "<string>", line 1, in <module>                                                                                                                            
      File "/tmp/pip-install-pifc_oyo/scikit-learn/setup.py", line 269, in <module>                                                                                   
        setup_package()                                                                                                                                               
      File "/tmp/pip-install-pifc_oyo/scikit-learn/setup.py", line 249, in setup_package                                                                              
        .format(numpy_req_str, instructions))                                                                                                                         
    ImportError: Numerical Python (NumPy) is not installed.                                                                                                           
    scikit-learn requires NumPy >= 1.8.2.                                                                                                                             
    Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html                                                  

    ----------------------------------------                                                                                                                          
ERROR: Command errored out with exit status 1: /home/reza/projects/virtualenvs/nava_37/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/p                
ip-install-pifc_oyo/scikit-learn/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pifc_oyo/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file_  
_);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-s3itmoyt/install-re  
cord.txt --single-version-externally-managed --compile --install-headers /home/reza/projects/virtualenvs/nava_37/include/site/python3.7/scikit-learn Check the logs   
for full command output.                                                                                                                                              

update [--no-dev] [--dry-run] [--lock] [--] [<packages>]...

Note: the same pyproject.toml without [tool.poetry.source] section works fine.

Debug info

• pip: 19.2.1
• setup-tools: 41.0.1

Poetry

• Version: 0.12.17
• Python: 3.7.3

Virtualenv

• Python: 3.7.3
• Implementation: CPython
• Path: /home/reza/projects/virtualenvs/nava_37
• Valid: True

System

• Platform: linux
• OS: posix
• Python: /usr

[esciara]

@devkral

setup:
only a pytoml file with the new build system

Could you give a gist link to your pyproject.toml file?

~Because what I found out is that adding packages in extras using poetry add foo_package -E foo generates the following:~

[Code block removed]

~Which is not what is explained on the doc for extras, which should be:~

poetry install -E foo will work with the following:

# ~~ snip ~~

[tool.poetry.dependencies]
foo_package = {version = "^1.0", optional = true}

[tool.poetry.extras]
foo = ["foo_package"]

# ~~ snip ~~

Or am I mixing up things? (again...)

[bersace]

@esciara add -E allows to add a dependency and some extras of the mentionned dependency. It's not meant to manage tool.poetry.extras. I don't know if there is CLI for this.

[esciara]

Thanks @bersace . I don't quite understand the difference. Could you point to some references or the doc clarifying this?

[bersace]

@esciara poetry add X -E Y is equivalent to appending X[Y] in install_requires. That's all.

What you understood is editing setup(extras).

Maybe poetry add should have a --optional extra_name option for editing tools.poetry.extras

$ poetry add foo_package -E foo_extra --optional foo

This would add foo_package[foo_extra] as depency of mypackage[foo]. Do you get it ?

[esciara]

Aha! Yes got it.

Yes I was expecting that everything poetry uses within pyproject.toml could be modified through the cli rather than manually only. Does that make sense?

If this logic is followed, then yes, doing something like adding a --optional extra_name to poetry add.

Do you reckon it would be the most intuitive way to add to tools.poetry.extras?

And by the way, do you have an example where --optional would be used on its own, without any extra_name?

[bersace]

Maybe poetry add foo_package[foo_extras] -E myextra seems easier for us used to setuptools.

[bersace]

I don't know of usage of --optionnal out of extras.

[esciara]

anyone else? @sdispater ?

[M5oul]

I also thought the extra feature was for optional dependencies.
Yes, that would be great to be able to specify optional dependencies.
In my case, I want to be able to install just one dependency for a CI job, in order to have a faster job, without installing all dependencies.

[msiemens]

I had the same issue, but it seems to work correctly if I specify the extra dependencies in the tool.poetry.dependencies section instead of the dev-dependencies section which seems to be broken.

Can any of you confirm this behavior?

I have the same behavior. From a quick search it seems like this is by design, see https://github.com/sdispater/poetry/pull/606#issuecomment-437943927

[devkral]

it was the missing optional dependency.
Sry for the late answer.

[leosussan]

This seems like a really important addition, IMO. Users are forced to add it manually to an extras list & run poetry lock otherwise - and deploys dependent on that list are inevitably going to break as a result.

(quick: tip for folks reading this comment: implement pre-commit with a hook that runs poetry check or poetry lock on commit to prevent somebody from forgetting to run poetry lock)

For the purpose of matching the -E and --extras flags present in other parts of the library, and keeping poetry consistent with other parts of the ecosystem, @bersace's suggested syntax feels most appropriate!

[taidinh-tiki]

Is it true that this problem has not been fixed yet? :(

[bersace]

It looks like https://github.com/python-poetry/poetry/blob/master/poetry/installation/installer.py#L147-L159 does not pass extras to solver.

[matthijskooijman]

Is it me, or are there two issues being mixed up here? This was originally about "extras" not being installed, but most discussion seems to be about needing a CLI instead of manually addin "extras" to pyproject.toml (and to poetry.lock using poetry lock)? Maybe the latter should move to a new issue if there isn't already?

[matthijskooijman]

From https://github.com/python-poetry/poetry/issues/1145#issuecomment-526184709 of @esciara

And by the way, do you have an example where --optional would be used on its own, without any extra_name?

Isn't this how --optional works now, you can do git add some_package --optional without further arguments? Or am I misunderstanding the question?

I think --optional now means to not remove the package if it is installed, but not install it autmatically either? I wonder if reusing --optional to also specify extras-only dependencies might not only make things more confusing? See also #1076.

From https://github.com/python-poetry/poetry/issues/1145#issuecomment-526193403 of @bersace

Maybe poetry add foo_package[foo_extras] -E myextra seems easier for us used to setuptools.

Are you there proposing to repurpose poetry add -E from meaning "extras to apply to the given package" to mean "extras needed for this dependency to be installed" (e.g. from 1. to 2. as described in https://github.com/python-poetry/poetry/issues/1076#issuecomment-639569322)? Wouldn't that bring compatibility problems and maybe more confusion?

One way to somewhat sidestep this question of how to add a dependency into an "extras", would be to move towards a "dependency-group" terminology (as proposed in #1644), which might be more natural to reason about?

[tony]

@matthijskooijman

Is it me, or are there two issues being mixed up here? This was originally about "extras" not being installed, but most discussion seems to be about needing a CLI instead of manually addin "extras" to pyproject.toml (and to poetry.lock using poetry lock)? Maybe the latter should move to a new issue if there isn't already?

I'm not able to understand this issue at the moment. It's both a bug and feature thread. I still don't know if this is 1.0 working as designed, a bug, if they're workarounds, etc. Ideally we'd be collecting / cross-referencing bugs and reproductions

@finswimmer @sdispater Can we close this and make 2 new issues:

1. Bug reports / reproductions / workarounds for extras not being installed in 1.0
2. For discussing CLI aesthetics / features around extra/optional packages?

[abn]

Thanks for going through this @tony.

The original reported issue is can no longer be reproduced.

$ cat pyproject.toml 
[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
psycopg2-binary = {version = "^2.8.6", optional = true}

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[tool.poetry.extras]
pg = ["psycopg2-binary"]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
$ [email protected] install -E pg --no-dev
Creating virtualenv foobar in /tmp/foobar/.venv
Installing dependencies from lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing psycopg2-binary (2.8.6)
  - Installing foobar (0.1.0)

Regarding (2) I think this warrants its own issue to discuss the options. The current options will likely not be repurposed. However, the CLI is tday lacking management of project extras. This migh also include a new option to specify what project extra an optional dependency should be added to. Documentation fixes can also be assessed if the current wording is causing confusion.

[matthijskooijman]

Note that this also interacts with #1644, which is about extending the extras feature into a more generic groups feature. If that is indeed implemented, that would also be a good moment to add new CLI options to manage the groups/extras of the package itself, I think.

[abey79]

The original reported issue is can no longer be reproduced.

@abn I disagree. As suggested, I opened a new issue for this: #3348.