Pip-tools: Transitive extras are not resolved

[vphilippon]

Transitive extras are not resolved.
If records[pandas] depends on tablib[pandas] which requires pandas, then pandas is not included in the requirements.

Environment Versions

1. OS Type: Any
2. Python version: Any
3. pip version: Any
4. pip-tools version: 1.10.1

Steps to replicate

requirements.in

records[pandas]

• pip-compile

Expected result

pandas is in the resulting requirements.txt

Actual result

pandas is not in the resulting requirements.txt

[atugushev]

Looks like pip's resolver still can't handle that case, see:

$ pip --version
pip 19.2.2 from /Users/albert/Projects/pip-tools/.venv/lib/python3.7/site-packages/pip (python 3.7)

$ pip install records[pandas]
Collecting records[pandas]
  Using cached https://files.pythonhosted.org/packages/ef/93/2467c761ea3729713ab97842a46cc125ad09d14a0a174cb637bee4983911/records-0.5.3-py2.py3-none-any.whl
Collecting SQLAlchemy; python_version >= "3.0" (from records[pandas])
  Downloading https://files.pythonhosted.org/packages/c2/75/6217c626fa22ad56ae5ccb1a36e7c4f17f5ca31543887e00179468d10464/SQLAlchemy-1.3.7.tar.gz (5.9MB)
     |████████████████████████████████| 5.9MB 4.1MB/s
Collecting docopt (from records[pandas])
  Using cached https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz
Collecting tablib>=0.11.4 (from records[pandas])
  Downloading https://files.pythonhosted.org/packages/7b/c7/cb74031b330cd94f3580926dc707d148b4ba9138449fc9f433cb79e640d8/tablib-0.13.0-py3-none-any.whl (75kB)
     |████████████████████████████████| 81kB 4.5MB/s
Collecting openpyxl<2.5.0 (from records[pandas])
  Using cached https://files.pythonhosted.org/packages/77/26/0bd1a39776f53b4f28e5bb1d26b3fcd99068584a7e1ddca4e09c0d5fd592/openpyxl-2.4.11.tar.gz
Collecting backports.csv (from tablib>=0.11.4->records[pandas])
  Using cached https://files.pythonhosted.org/packages/8e/26/a6bd68f13e0f38fbb643d6e497fc3462be83a0b6c4d43425c78bb51a7291/backports.csv-1.0.7-py2.py3-none-any.whl
Collecting xlrd (from tablib>=0.11.4->records[pandas])
  Using cached https://files.pythonhosted.org/packages/b0/16/63576a1a001752e34bf8ea62e367997530dc553b689356b9879339cf45a4/xlrd-1.2.0-py2.py3-none-any.whl
Collecting xlwt (from tablib>=0.11.4->records[pandas])
  Using cached https://files.pythonhosted.org/packages/44/48/def306413b25c3d01753603b1a222a011b8621aed27cd7f89cbc27e6b0f4/xlwt-1.3.0-py2.py3-none-any.whl
Collecting pyyaml (from tablib>=0.11.4->records[pandas])
Collecting odfpy (from tablib>=0.11.4->records[pandas])
  Using cached https://files.pythonhosted.org/packages/85/7d/8f6d1f2a4683be362b101c00232b4c3839e4e4a90e0945d8d43ec6aa671d/odfpy-1.4.0.tar.gz
Collecting jdcal (from openpyxl<2.5.0->records[pandas])
  Using cached https://files.pythonhosted.org/packages/f0/da/572cbc0bc582390480bbd7c4e93d14dc46079778ed915b505dc494b37c57/jdcal-1.4.1-py2.py3-none-any.whl
Collecting et_xmlfile (from openpyxl<2.5.0->records[pandas])
  Using cached https://files.pythonhosted.org/packages/22/28/a99c42aea746e18382ad9fb36f64c1c1f04216f41797f2f0fa567da11388/et_xmlfile-1.0.1.tar.gz
Collecting defusedxml (from odfpy->tablib>=0.11.4->records[pandas])
  Using cached https://files.pythonhosted.org/packages/06/74/9b387472866358ebc08732de3da6dc48e44b0aacd2ddaa5cb85ab7e986a2/defusedxml-0.6.0-py2.py3-none-any.whl
Building wheels for collected packages: SQLAlchemy, docopt, openpyxl, odfpy, et-xmlfile
  Building wheel for SQLAlchemy (setup.py) ... done
  Created wheel for SQLAlchemy: filename=SQLAlchemy-1.3.7-cp37-cp37m-macosx_10_14_x86_64.whl size=1145604 sha256=5397c866669ee8461c23ce93e38c11eaa3783f08e835c961d1663bfb374cef4f
  Stored in directory: /Users/albert/Library/Caches/pip/wheels/80/e0/68/3d11cc7209b5bd2c7d55cbb56c6bda843cc82f77c8387468ea
  Building wheel for docopt (setup.py) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=97b55e96cd59a303ecd1b5c514f1c9953c4d92085e57d8feeaa7382f955c0da2
  Stored in directory: /Users/albert/Library/Caches/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e
  Building wheel for openpyxl (setup.py) ... done
  Created wheel for openpyxl: filename=openpyxl-2.4.11-py2.py3-none-any.whl size=222820 sha256=98c8b31114291ed8ab8fc6e2f95173fec10993b41733296a19aed0ef76dce57d
  Stored in directory: /Users/albert/Library/Caches/pip/wheels/59/44/27/63b211425501ad51d197ff8ed00e9e469e38b9e516cb69b1c2
  Building wheel for odfpy (setup.py) ... done
  Created wheel for odfpy: filename=odfpy-1.4.0-py2.py3-none-any.whl size=137219 sha256=a423bce5e06e8dca85dfb13d4f3ec22dbb02004f4e2cf438d661a9c026b18f5c
  Stored in directory: /Users/albert/Library/Caches/pip/wheels/06/2d/19/f5a4eed468fecff295ff8ac49e5dd5fb22d7ffc7ff072deabf
  Building wheel for et-xmlfile (setup.py) ... done
  Created wheel for et-xmlfile: filename=et_xmlfile-1.0.1-cp37-none-any.whl size=8914 sha256=9308fef4af95b86f18f79ac5ddf4e859bb9193aee7048373ec345f0374772e6b
  Stored in directory: /Users/albert/Library/Caches/pip/wheels/2a/77/35/0da0965a057698121fc7d8c5a7a9955cdbfb3cc4e2423cad39
Successfully built SQLAlchemy docopt openpyxl odfpy et-xmlfile
Installing collected packages: SQLAlchemy, docopt, backports.csv, xlrd, jdcal, et-xmlfile, openpyxl, xlwt, pyyaml, defusedxml, odfpy, tablib, records
Successfully installed SQLAlchemy-1.3.7 backports.csv-1.0.7 defusedxml-0.6.0 docopt-0.6.2 et-xmlfile-1.0.1 jdcal-1.4.1 odfpy-1.4.0 openpyxl-2.4.11 pyyaml-5.1.2 records-0.5.3 tablib-0.13.0 xlrd-1.2.0 xlwt-1.3.0

As you can see there is no installed pandas.

Refs https://github.com/pypa/pip/issues/6239.

[atugushev]

The pip's crew is working on a dependency resolver, so let's wait until it'll be ready https://github.com/pypa/pip/issues/988.

[AndydeCleyre]

Just checking in to note that pip master (20.0.2-601-g97f63905) now can do:

$ pip install --unstable-feature=resolver 'records[pandas]'
$ pip freeze
docopt==0.6.2
et-xmlfile==1.0.1
jdcal @ https://files.pythonhosted.org/packages/f0/da/572cbc0bc582390480bbd7c4e93d14dc46079778ed915b505dc494b37c57/jdcal-1.4.1-py2.py3-none-any.whl#sha256=1abf1305fce18b4e8aa248cf8fe0c56ce2032392bc64bbd61b5dff2a19ec8bba
numpy @ https://files.pythonhosted.org/packages/2e/2f/5d2f9eb8ea6702966e31ca8a1f8515b34c240699fca389c6009fec919d56/numpy-1.18.2-cp38-cp38-manylinux1_x86_64.whl#sha256=82847f2765835c8e5308f136bc34018d09b49037ec23ecc42b246424c767056b
openpyxl==2.4.11
pandas @ https://files.pythonhosted.org/packages/f5/10/40688389f5e234bde06aa84e6f3ccf5beea6269f57e2bef67866d3b43268/pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl#sha256=387dc7b3c0424327fe3218f81e05fc27832772a5dffbed385013161be58df90b
python-dateutil @ https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl#sha256=75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a
pytz @ https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl#sha256=1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d
records @ https://files.pythonhosted.org/packages/ef/93/2467c761ea3729713ab97842a46cc125ad09d14a0a174cb637bee4983911/records-0.5.3-py2.py3-none-any.whl#sha256=cdbacf52c61b4a3bc10fef1286a24a63ae95255a2e7b4e8ccb1e1f96737231ed
six @ https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl#sha256=8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c
SQLAlchemy @ https://files.pythonhosted.org/packages/2b/1a/65673e3d242dd557576c538b72e8ae85224d698cd08b7b60e822d203fab4/SQLAlchemy-1.3.16-cp38-cp38-manylinux2010_x86_64.whl#sha256=5b1bf3c2c2dca738235ce08079783ef04f1a7fc5b21cf24adaae77f2da4e73c3
tablib @ https://files.pythonhosted.org/packages/7b/68/6ebc32ee79945feac078713f3fe3dc280f328d10d178674dc1227ade8d01/tablib-1.1.0-py3-none-any.whl#sha256=80f6c3453431cedf1125f23d16b3d96b92b426495714ebf0b4dede1fa75b447d

[atugushev]

See also https://github.com/pradyunsg/zazo/issues/15#issuecomment-615864064

[webknjaz]

Is this still relevant?

[graingert]

@webknjaz yes! this is now fixed so we can use https://github.com/dask/s3fs/pull/362/files