Pipenv: Pipfile needs a way to manually specify package installation order

Created on 28 May 2018 · 10Comments · Source: pypa/pipenv

GDAL is a package that is used by many GIS projects. During installation, GDAL's python bindings are compiled using the headers from the existing numpy version. So you need to install numpy before GDAL, else numpy bindings will not work.

This seems to be a use case for pipenv install --sequential, but I am not able to control the order of the sequential install.

NB: you need to have libgdal-dev on Ubuntu 16 and CPLUS_INCLUDE_PATH=/usr/include/gdal and C_INCLUDE_PATH=/usr/include/gdal to reproduce this.

pipenv install numpy gdal==1.11.2 --sequential works as expected.

Then, pipenv --rm, pipenv install --deploy --sequential gives me the wrong sequence: numpy is installed after gdal, and my numpy bindings do not work.

The sequence seems to origin from the Pipfile.lock, which is alphabetic. gdal is before numpy in the alphabet, so at least installation fails in a reproducible way.

How should I approach this issue?

Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
numpy = "*"
gdal = "==1.11.2"

[dev-packages]

[requires]
python_version = "2.7"

Pipfile.lock:

{
    "_meta": {
        "hash": {
            "sha256": "42b5387796abc5d9542f6213df851d564959e44cd1c1a0789fc9dc5e0e5f5d0f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "gdal": {
            "hashes": [
                "sha256:7827197836690815b0142412a49a97c6c6a18b21a83ece8cd6d8ec66127f10df"
            ],
            "index": "pypi",
            "version": "==1.11.2"
        },
        "numpy": {
            "hashes": [
                "sha256:0074d42e2cc333800bd09996223d40ec52e3b1ec0a5cab05dacc09b662c4c1ae",
                "sha256:034717bfef517858abc79324820a702dc6cd063effb9baab86533e8a78670689",
                "sha256:0db6301324d0568089663ef2701ad90ebac0e975742c97460e89366692bd0563",
                "sha256:1864d005b2eb7598063e35c320787d87730d864f40d6410f768fe4ea20672016",
                "sha256:46ce8323ca9384814c7645298b8b627b7d04ce97d6948ef02da357b2389d6972",
                "sha256:510863d606c932b41d2209e4de6157ab3fdf52001d3e4ad351103176d33c4b8b",
                "sha256:560e23a12e7599be8e8b67621396c5bc687fd54b48b890adbc71bc5a67333f86",
                "sha256:57dc6c22d59054542600fce6fae2d1189b9c50bafc1aab32e55f7efcc84a6c46",
                "sha256:760550fdf9d8ec7da9c4402a4afe6e25c0f184ae132011676298a6b636660b45",
                "sha256:8670067685051b49d1f2f66e396488064299fefca199c7c80b6ba0c639fedc98",
                "sha256:9016692c7d390f9d378fc88b7a799dc9caa7eb938163dda5276d3f3d6f75debf",
                "sha256:98ff275f1b5907490d26b30b6ff111ecf2de0254f0ab08833d8fe61aa2068a00",
                "sha256:9ccf4d5c9139b1e985db915039baa0610a7e4a45090580065f8d8cb801b7422f",
                "sha256:a8dbab311d4259de5eeaa5b4e83f5f8545e4808f9144e84c0f424a6ee55a7b98",
                "sha256:aaef1bea636b6e552bbc5dae0ada87d4f6046359daaa97a05a013b0169620f27",
                "sha256:b8987e30d9a0eb6635df9705a75cf8c4a2835590244baecf210163343bc65176",
                "sha256:c3fe23df6fe0898e788581753da453f877350058c5982e85a8972feeecb15309",
                "sha256:c5eb7254cfc4bd7a4330ad7e1f65b98343836865338c57b0e25c661e41d5cfd9",
                "sha256:c80fcf9b38c7f4df666150069b04abbd2fe42ae640703a6e1f128cda83b552b7",
                "sha256:e33baf50f2f6b7153ddb973601a11df852697fba4c08b34a5e0f39f66f8120e1",
                "sha256:e8578a62a8eaf552b95d62f630bb5dd071243ba1302bbff3e55ac48588508736",
                "sha256:f22b3206f1c561dd9110b93d144c6aaa4a9a354e3b07ad36030df3ea92c5bb5b",
                "sha256:f39afab5769b3aaa786634b94b4a23ef3c150bdda044e8a32a3fc16ddafe803b"
            ],
            "index": "pypi",
            "version": "==1.14.3"
        }
    },
    "develop": {}
}

Behavior Change Discussion Type

Source

caspervdw

👍11 ❤1

Most helpful comment

(I've just stumbled across this same problem, thanks for the pointer to pygdal)

The way I see it, not all projects can be relied on to write their setup.py scripts in the "right" way. Therefore, it would be really nice to be able to override the dependencies of any package in the Pipfile. Somehow we would need to specify that, for the current application, gdal depends on numpy. Would something like this be possible?

[packages]
gdal = { setup_requires = ["numpy"] }

simonk52 on 4 Jun 2018

👍6

All 10 comments

Oh that’s an interesting one!

Perhaps sequential installs should always start with Pipfile ordering and should always install top level packages first? /cc @uranusjr — do you think this impacts people who depend on things like Cython in their installation ordering?

techalchemy on 28 May 2018

I just need numpy before gdal. For larger projects, the ideal solution would be specifying just that instead of the slower sequential install.

Is there a way to extend extras=numpy for this use case?

caspervdw on 28 May 2018

It is probably best for package maintainers to have appropriate extras_require flags for those usages (e.g. gdal[numpy] to require numpy before gdal; I think we account for those already). But yes it would be nice if we can default to ordering in Pipfile, following the “not worst than the predecessor” (predecessor here being requirements.txt) rule.

uranusjr on 28 May 2018

@uranusjr That would be the ideal solution indeed. It seems that https://pypi.org/project/pygdal/ may solve the issue for me; they properly specify numpy as a dependency.

Apart from that, I would like to be able to control the --sequential order. To be complete: installation order now is alphabetic, and editable / VCS packages are installed first (but @techalchemy said earlier that a PR addressing the VCS issue is up already)

caspervdw on 30 May 2018

(I've just stumbled across this same problem, thanks for the pointer to pygdal)

[packages]
gdal = { setup_requires = ["numpy"] }

simonk52 on 4 Jun 2018

👍6

Another use-case:
On Windows, python-ldap is rather tricky to build and they don't publish wheels to PyPi.
You need to be able to say that your wheel (e.g. downloaded from https://www.lfd.uci.edu/~gohlke/pythonlibs/) should be installed before anything that requires python-ldap.

OrangeDog on 7 Jun 2018

I assume this is not yet solved and when you run pipenv install it is still random ?

Eimis on 18 Aug 2019

It is not random, but there is no way to customise ordering (I assume this is what you actually intend to ask).

uranusjr on 18 Aug 2019

It is not random, but there is no way to customise ordering (I assume this is what you actually intend to ask).

Yes exactly. Thanks for the response, it's clear!

Eimis on 18 Aug 2019

This is not a proper solution, but it might be acceptable for some until (if) this feature is implemented.

pipenv install --dev appears to install dev-packages before the default packages, so after editing the Pipfile by placing the numpy dependency in dev-packages, and GDAL in packages, then pipenv install --dev --sequential seems to install numpy before GDAL and avoids the author's original error.