Poetry: Adding extras for dependencies

[bertjwregeer]

Is there a way to specify extras that should be installed from a dependency? For example, I'd like to specify:

poetry add sqlalchemy[postgresql]

Since my project requires sqlalchemy and postgresql, but the only hard dependency is sqlalchemy, so depending on psycopg2 directly doesn't make sense. psycopg2 being the package that is installed as a sqlalchemy extra when you ask for postgresql.

This way if sqlalchemy changes the postgresql extra to something else I'd automatically pick up the change.

[sdispater]

Currently this is not possible, unfortunately.

However, it will be possible in the next feature release. The add command is being improved to support adding path dependencies, git dependencies and dependencies with extras.

[sdispater]

This is now available in release 0.9.0!

[bertjwregeer]

@sdispater I am trying to use this, and have the following in pyproject.toml:

sqlalchemy = {version = "^1.2", extras = ["postgresql"]}

According to sqlalchemy's setup.py this should as an extra install psycopg2: https://github.com/zzzeek/sqlalchemy/blob/master/setup.py#L159

However poetry does not seem to even consider the extras, and does not install psycopg2:

```Riley:pyramidtesting xistence$ poetry show
attrs 18.1.0 Classes Without Boilerplate
cffi 1.11.5 Foreign Function Interface for Python calling C code.
colorama 0.3.9 Cross-platform colored terminal text.
coverage 4.5.1 Code coverage measurement for Python
cryptacular 1.4.1 A password hashing framework with bcrypt and pbkdf2.
funcsigs 1.0.2 Python function signatures from PEP362 for Python 2.6, 2.7 and 3.2+
hupper 1.2 Integrated process monitor for developing and reloading daemons.
ipaddress 1.0.22 IPv4/IPv6 manipulation library
mako 1.0.7 A super-fast templating language that borrows the best ideas from the existing templating languages.
markupsafe 1.0 Implements a XML/HTML/XHTML Markup safe string for Python
misaka 2.1.0 A CFFI binding for Hoedown, a markdown parsing library.
more-itertools 4.1.0 More routines for operating on iterables, beyond itertools
ordereddict 1.1 A drop-in substitute for Py2.7's new collections.OrderedDict that works in Python 2.4-2.6.
pastedeploy 1.5.2 Load, configure, and compose WSGI applications and servers
pbkdf2 1.3 PKCS#5 v2.0 PBKDF2 Module
plaster 1.0 A loader interface around multiple config file formats.
plaster-pastedeploy 0.5 A loader implementing the PasteDeploy syntax to be used by plaster.
pluggy 0.6.0 plugin and hook calling mechanisms for python
py 1.5.3 library with cross-python path, ini-parsing, io, code, log facilities
pycparser 2.18 C parser in Python
pygments 2.2.0 Pygments is a syntax highlighting package written in Python.
pyramid 1.9.2 The Pyramid Web Framework, a Pylons project
pyramid-authsanity 1.1.0 An auth policy for the Pyramid Web Framework with sane defaults.
pyramid-debugtoolbar 4.4 A package which provides an interactive HTML debugger for Pyramid application development
pyramid-mailer 0.15.1 Sendmail package for Pyramid
pyramid-mako 1.0.2 Mako template bindings for the Pyramid web framework
pyramid-retry 0.5 An execution policy for Pyramid that supports retrying requests after certain failure exceptions.
pyramid-services 1.1 A service layer abstraction for the Pyramid Web Framework.
pyramid-tm 2.2 A package which allows Pyramid requests to join the active transaction
pytest 3.5.1 pytest: simple powerful testing with Python
pytest-cov 2.5.1 Pytest plugin for measuring coverage.
repoze.lru 0.7 A tiny LRU cache implementation and decorator
repoze.sendmail 4.4.1 Repoze Sendmail
six 1.11.0 Python 2 and 3 compatibility utilities
sqlalchemy 1.2.7 Database Abstraction Library
transaction 2.2.1 Transaction management for Python
translationstring 1.3 Utility library for i18n relied on by various Repoze and Pyramid packages
venusian 1.1.0 A library for deferring decorator actions
waitress 1.1.0 Waitress WSGI server
webob 1.8.1 WSGI request and response object
zope.deprecation 4.3.0 Zope Deprecation Infrastructure
zope.interface 4.5.0 Interfaces for Python
zope.sqlalchemy 1.0 Minimal Zope/SQLAlchemy transaction integration

Changing `pyproject.toml` to:

sqlalchemy = {version = "^1.2", extras = ["thisdoesnotexist"]}

Shows that poetry recognises the change, but an update does nothing:

```Riley:pyramidtesting xistence$ poetry install
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

Nothing to install or update

Riley:pyramidtesting xistence$ poetry update
Updating dependencies
Resolving dependencies...

Nothing to install or update

Writing lock file

[sdispater]

@bertjwregeer The extras feature works fine but in your case there is a problem which prevents poetry from getting sqlalchemy's dependency information.

Here is the breakdown of what happens:

• First, poetry checks the PyPI JSON API (https://pypi.org/pypi/SQLAlchemy/json)
• Since it can't get dependencies (requires_dist is null), it proceeds with downloading and inspecting the release tarball (https://files.pythonhosted.org/packages/c1/c8/392fcd2d01534bc871c65cb964e0b39d59feb777e51649e6eaf00f6377b5/SQLAlchemy-1.2.7.tar.gz).
• First, it checks if it can get the dependency information from the PKGINFO file, which is not possible in this case.
• Then, it checks if there is a *.egg-info directory which contains a requires.txt file containing the dependency information (this is a fairly common case). But in this case it does not exist.
• At this point, poetry could execute the setup.py file but I don't feel confortable doing that just for the sake of getting dependencies since to me it introduces a security risk and will sometimes fail due to C extensions or missing dependencies referenced in setup.py.

[bertjwregeer]

It would be nice if there was some sort of warning about that, I just hadn't realised that it is due to missing packaging information.

I know it may go against part of the project, but it may make sense in that case to explicitly call pip install sqlalchemy[extra] when installing, vs just silently letting it go.

[sdispater]

I know it may go against part of the project, but it may make sense in that case to explicitly call pip install sqlalchemy[extra] when installing, vs just silently letting it go.

The thing with this is installing with extras that are not tracked can lead to conflicting or overridden dependencies, so I don't think this is a good idea since it would break the feature given by the lock mechanism.

I tested if executing the setup.py of sqlalchemy would work and it does but like I said I think this leads to indroducing a security risk that, to me, should not be necessary to get dependency metadata, because dependencies should only be metadata and the execution of a file should not be required.

[nickpresta]

@sdispater am I correct in that if the package provides a *.egg-info directory with a requires.txt with the following content (for example):

requests<3,>=2.19

[local_testing]
pytest
pytest-cov

[testing]
requests_mock<2,>=1.5

That poetry should be able to figure out that installing package[testing] should install requests_mock?

My pyproject.toml looks like this:

[tool.poetry.dependencies]
my_package = "^1"

[tool.poetry.dev-dependencies]
my_package = { version = "^1", extras = [ "testing" ] }

When I run a full poetry update followed by a poetry show --tree, I don't see the requests_mock package:

my_package 1.2.1 Summary here
└── requests >=2.19,<3
    ├── certifi >=2017.4.17
    ├── chardet >=3.0.2,<3.1.0
    ├── idna >=2.5,<2.8
    └── urllib3 >=1.21.1,<1.25

I would expect to see requests_mock:

$ poetry show requests_mock

[ValueError]
Package requests_mock not found

show [--no-dev] [-t|--tree] [-l|--latest] [-o|--outdated] [-a|--all] [--] [<package>]

Is this a bug? Is it this bug, and if so, can this be reopened?

Thanks!

[sdispater]

It should install requests_mock, yes.

What's the complete output of poetry debug:resolve -vvv?

[hixus]

I'm not also getting extras installed. Trying to convert requirements.txt apache-airflow[gcp_api,google_auth,vertica,slack,s3]==1.10.4 to poetry

[hixus]

I'm not also getting extras installed. Trying to convert requirements.txt apache-airflow[gcp_api,google_auth,vertica,slack,s3]==1.10.4 to poetry

Sorry, I was wrong. It did work after all.

[wgordon17]

Hey all, I ran poetry add fastapi[requests]. This resulted in the expected fastapi = {extras = ["requests"], version = "^0.54.1"} line in my pyproject.toml file, but I'm still not seeing requests getting installed. Is something wrong with my poetry install??

[gergelyk]

Shouldn't this be poetry add fastapi -E requests, @wgordon17 ?

[wgordon17]

Thanks @gergelyk, I just tried that. It had the same effect as before. It creates the expected entry in pyproject.toml, but doesn't _actually_ install the requests package.

[bertjwregeer]

I don't see an extra defined in https://github.com/tiangolo/fastapi/blob/master/pyproject.toml that is called requests.

There's: test, doc, dev, and all.

[wgordon17]

Heh, somehow I didn't realize that 🤦 Thanks @bertjwregeer, testing with another project, all worked as expected 👍🏻