Pipenv: Pipenv fails to install latest version of pyarrow on linux

Created on 13 Apr 2018 · 18Comments · Source: pypa/pipenv

Pipenv fails to install and sync latest version of pyarrow on linux. As far as I could tell this is due to a difference between their versioning of the latest release:

linux, windows: 0.9.0 (https://pypi.python.org/pypi/pyarrow)
macOS: 0.9.0.post1 (https://pypi.python.org/pypi/pyarrow/0.9.0.post1)

When I attempt to install pyarrow on a linux, pipenv returns the following:

And on a macOS, it works:

Howerver if I specify the version to: pyarrow==0.9.0
On linux, it works:

And on macOS, it does not:

I know that this is a edge case, and according to here, it should be fixed when the pyarrow team releases a working macOS wheel of the 0.9.0 version. However pip can correctly identify and install each OS specific version. So, I, personally, think this should work on pipenv too.

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/home/vasv/.local/bin/python'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
3.6.3: /home/vasv/.local/bin/python
2.7.14: /usr/bin/python
2.7.14: /usr/bin/python2
3.6.3: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-38-generic',
 'platform_system': 'Linux',
 'platform_version': '#43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018',
 'python_full_version': '3.6.3',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

LS_COLORS
LESSCLOSE
LANG
GDM_LANG
HISTIGNORE
DISPLAY
COLORTERM
NO_AT_BRIDGE
XDG_VTNR
SSH_AUTH_SOCK
MANDATORY_PATH
S_COLORS
XDG_SESSION_ID
XDG_GREETER_DATA_DIR
USER
DESKTOP_SESSION
DEFAULTS_PATH
QT_QPA_PLATFORMTHEME
PWD
HOME
SSH_AGENT_PID
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
XDG_SESSION_DESKTOP
TERM
SHELL
XDG_SEAT_PATH
SHLVL
XDG_SEAT
LANGUAGE
WINDOWID
GDMSESSION
TERMINFO
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_SESSION_PATH
XDG_CONFIG_DIRS
PATH
PS1
PKG_CONFIG_PATH
PYTHON
GCC_COLORS
LESSOPEN
_
OLDPWD
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/vasv/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/vasv/Workspace/antenna_protocol

Contents of Pipfile ('/home/vasv/Workspace/antenna_protocol/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyarrow = ">=0.9.0,<0.9.1"
logzero = "*"
async-timeout = "*"
aioreactive = "*"
uvloop = {version = "*", sys_platform = "!= 'win32'"}

[dev-packages]
pipenv = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/home/vasv/Workspace/antenna_protocol/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "840ae8c8bc18ece383055eb1462561e4490412e84042a0bc25d5feb13b09f693"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aioreactive": {
            "hashes": [
                "sha256:201de56d7bc12854dda45a7e289dce49c99f7d3a50296feb48a401992f37b678",
                "sha256:6c31397f2be441fdcf951f74298ddb7bf64670b8e3b34cd0f537b02016eb40a9"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "async-timeout": {
            "hashes": [
                "sha256:00cff4d2dce744607335cba84e9929c3165632da2d27970dbc55802a0c7873d0",
                "sha256:9093db5b8ddbe4b8f6885d1a6e0ad84ae3155464cbf6877c387605244c285f3c"
            ],
            "index": "pypi",
            "version": "==2.0.1"
        },
        "futures": {
            "hashes": [
                "sha256:9ec02aa7d674acb8618afb127e27fde7fc68994c0437ad759fa094a574adb265",
                "sha256:ec0a6cb848cc212002b9828c3e34c675e0c9ff6741dc445cab6fdd4e1085d1f1"
            ],
            "version": "==3.2.0"
        },
        "logzero": {
            "hashes": [
                "sha256:34fa1e2e436dfa9f37e5ff8750e932bafe0c5abbb42e1f669e4cf5ce1f179142",
                "sha256:818072e4fcb53a3f6fb4114a92f920e1135fe6f47bffd9dc2b6c4d10eedacf27"
            ],
            "index": "pypi",
            "version": "==1.5.0"
        },
        "numpy": {
            "hashes": [
                "sha256:0739146eaf4985962f07c62f7133aca89f3a600faac891ce6c7f3a1e2afe5272",
                "sha256:07e21f14490324cc1160db101e9b6c1233c33985af4cb1d301dd02650fea1d7f",
                "sha256:0f6a5ed0cd7ab1da11f5c07a8ecada73fc55a70ef7bb6311a4109891341d7277",
                "sha256:0fd65cbbfdbf76bbf80c445d923b3accefea0fe2c2082049e0ce947c81fe1d3f",
                "sha256:20cac3123d791e4bf8482a580d98d6b5969ba348b9d5364df791ba3a666b660d",
                "sha256:528ce59ded2008f9e8543e0146acb3a98a9890da00adf8904b1e18c82099418b",
                "sha256:56e392b7c738bd70e6f46cf48c8194d3d1dd4c5a59fae4b30c58bb6ef86e5233",
                "sha256:675e0f23967ce71067d12b6944add505d5f0a251f819cfb44bdf8ee7072c090d",
                "sha256:6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888",
                "sha256:719d914f564f35cce4dc103808f8297c807c9f0297ac183ed81ae8b5650e698e",
                "sha256:768e777cc1ffdbf97c507f65975c8686ebafe0f3dc8925d02ac117acc4669ce9",
                "sha256:7f76d406c6b998d6410198dcb82688dcdaec7d846aa87e263ccf52efdcfeba30",
                "sha256:8c18ee4dddd5c6a811930c0a7c7947bf16387da3b394725f6063f1366311187d",
                "sha256:99051e03b445117b26028623f1a487112ddf61a09a27e2d25e6bc07d37d94f25",
                "sha256:a1413d06abfa942ca0553bf3bccaff5fdb36d55b84f2248e36228db871147dab",
                "sha256:a7157c9ac6bddd2908c35ef099e4b643bc0e0ebb4d653deb54891d29258dd329",
                "sha256:a958bf9d4834c72dee4f91a0476e7837b8a2966dc6fcfc42c421405f98d0da51",
                "sha256:bb370120de6d26004358611441e07acda26840e41dfedc259d7f8cc613f96495",
                "sha256:d0928076d9bd8a98de44e79b1abe50c1456e7abbb40af7ef58092086f1a6c729",
                "sha256:d858423f5ed444d494b15c4cc90a206e1b8c31354c781ac7584da0d21c09c1c3",
                "sha256:e6120d63b50e2248219f53302af7ec6fa2a42ed1f37e9cda2c76dbaca65036a7",
                "sha256:f2b1378b63bdb581d5d7af2ec0373c8d40d651941d283a2afd7fc71184b3f570",
                "sha256:facc6f925c3099ac01a1f03758100772560a0b020fb9d70f210404be08006bcb"
            ],
            "version": "==1.14.2"
        },
        "pyarrow": {
            "hashes": [
                "sha256:5d06a1df6c1c43a122cd05077a3d47ae8a53dabc0f3620aba78c7d434075b2c8",
                "sha256:92c7441d3a57870e34a47bd30d55dfd4c5d49d29893e3221955608e4a10fa57d",
                "sha256:df1b0b67ab4029628264219c5f236f3c828c030e14b455cfd2b2a9d41cf3609b"
            ],
            "index": "pypi",
            "version": "==0.9.0.post1"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==0.9.1"
        }
    },
    "develop": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "pipenv": {
            "hashes": [
                "sha256:325b26672172f11cf46ee5fa31cc4c751df24f920617ca21eeb68fce70a394e2",
                "sha256:dab6b2e5347757cda7973e6c6065c30c020c5ff8b0687306b83e962aea4adc6c"
            ],
            "index": "pypi",
            "version": "==11.10.0"
        },
        "virtualenv": {
            "hashes": [
                "sha256:1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54",
                "sha256:e8e05d4714a1c51a2f5921e62f547fcb0f713ebbe959e0a7f585cc8bef71d11f"
            ],
            "version": "==15.2.0"
        },
        "virtualenv-clone": {
            "hashes": [
                "sha256:4507071d81013fd03ea9930ec26bc8648b997927a11fa80e8ee81198b57e0ac7",
                "sha256:b5cfe535d14dc68dfc1d1bb4ac1209ea28235b91156e2bba8e250d291c3fb4f8"
            ],
            "version": "==0.3.0"
        }
    }
}

$ python -m pipenv.help (Linux):

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/local/bin/python3.6m
3.6: /usr/local/bin/python3.6
3.6.5: /usr/local/opt/python/libexec/bin/python
2.7.10: /usr/bin/python
3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST '
                     '2018; root:xnu-4570.51.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

LANG
XPC_FLAGS
HISTIGNORE
DISPLAY
TERM_SESSION_ID
EDITOR
JAVA_HOME
SSH_AUTH_SOCK
USER
LSCOLORS
PWD
HOME
LC_CTYPE
TERM_PROGRAM
TERM_PROGRAM_VERSION
Apple_PubSub_Socket_Render
TMPDIR
XPC_SERVICE_NAME
SHELL
TERM
SHLVL
CPPFLAGS
LOGNAME
LDFLAGS
PATH
PS1
PYTHON
HOMEBREW_GITHUB_API_TOKEN
_
OLDPWD
__CF_USER_TEXT_ENCODING
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/bin:/usr/local/sbin:/usr/local/opt/python/libexec/bin:/usr/local/opt/go/libexec/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHELL: /usr/local/bin/bash
EDITOR: code
LANG: en_US.UTF-8
PWD: /Users/hvolkoff/Workspace/Python/antenna_protocol

Contents of Pipfile ('/Users/hvolkoff/Workspace/Python/antenna_protocol/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyarrow = ">=0.9.0,<0.9.1"
logzero = "*"
async-timeout = "*"
aioreactive = "*"
uvloop = {version = "*", sys_platform = "!= 'win32'"}

[dev-packages]
pipenv = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/Users/hvolkoff/Workspace/Python/antenna_protocol/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "840ae8c8bc18ece383055eb1462561e4490412e84042a0bc25d5feb13b09f693"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aioreactive": {
            "hashes": [
                "sha256:201de56d7bc12854dda45a7e289dce49c99f7d3a50296feb48a401992f37b678",
                "sha256:6c31397f2be441fdcf951f74298ddb7bf64670b8e3b34cd0f537b02016eb40a9"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "async-timeout": {
            "hashes": [
                "sha256:00cff4d2dce744607335cba84e9929c3165632da2d27970dbc55802a0c7873d0",
                "sha256:9093db5b8ddbe4b8f6885d1a6e0ad84ae3155464cbf6877c387605244c285f3c"
            ],
            "index": "pypi",
            "version": "==2.0.1"
        },
        "logzero": {
            "hashes": [
                "sha256:34fa1e2e436dfa9f37e5ff8750e932bafe0c5abbb42e1f669e4cf5ce1f179142",
                "sha256:818072e4fcb53a3f6fb4114a92f920e1135fe6f47bffd9dc2b6c4d10eedacf27"
            ],
            "index": "pypi",
            "version": "==1.5.0"
        },
        "numpy": {
            "hashes": [
                "sha256:0739146eaf4985962f07c62f7133aca89f3a600faac891ce6c7f3a1e2afe5272",
                "sha256:07e21f14490324cc1160db101e9b6c1233c33985af4cb1d301dd02650fea1d7f",
                "sha256:0f6a5ed0cd7ab1da11f5c07a8ecada73fc55a70ef7bb6311a4109891341d7277",
                "sha256:0fd65cbbfdbf76bbf80c445d923b3accefea0fe2c2082049e0ce947c81fe1d3f",
                "sha256:20cac3123d791e4bf8482a580d98d6b5969ba348b9d5364df791ba3a666b660d",
                "sha256:528ce59ded2008f9e8543e0146acb3a98a9890da00adf8904b1e18c82099418b",
                "sha256:56e392b7c738bd70e6f46cf48c8194d3d1dd4c5a59fae4b30c58bb6ef86e5233",
                "sha256:675e0f23967ce71067d12b6944add505d5f0a251f819cfb44bdf8ee7072c090d",
                "sha256:6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888",
                "sha256:719d914f564f35cce4dc103808f8297c807c9f0297ac183ed81ae8b5650e698e",
                "sha256:768e777cc1ffdbf97c507f65975c8686ebafe0f3dc8925d02ac117acc4669ce9",
                "sha256:7f76d406c6b998d6410198dcb82688dcdaec7d846aa87e263ccf52efdcfeba30",
                "sha256:8c18ee4dddd5c6a811930c0a7c7947bf16387da3b394725f6063f1366311187d",
                "sha256:99051e03b445117b26028623f1a487112ddf61a09a27e2d25e6bc07d37d94f25",
                "sha256:a1413d06abfa942ca0553bf3bccaff5fdb36d55b84f2248e36228db871147dab",
                "sha256:a7157c9ac6bddd2908c35ef099e4b643bc0e0ebb4d653deb54891d29258dd329",
                "sha256:a958bf9d4834c72dee4f91a0476e7837b8a2966dc6fcfc42c421405f98d0da51",
                "sha256:bb370120de6d26004358611441e07acda26840e41dfedc259d7f8cc613f96495",
                "sha256:d0928076d9bd8a98de44e79b1abe50c1456e7abbb40af7ef58092086f1a6c729",
                "sha256:d858423f5ed444d494b15c4cc90a206e1b8c31354c781ac7584da0d21c09c1c3",
                "sha256:e6120d63b50e2248219f53302af7ec6fa2a42ed1f37e9cda2c76dbaca65036a7",
                "sha256:f2b1378b63bdb581d5d7af2ec0373c8d40d651941d283a2afd7fc71184b3f570",
                "sha256:facc6f925c3099ac01a1f03758100772560a0b020fb9d70f210404be08006bcb"
            ],
            "version": "==1.14.2"
        },
        "pyarrow": {
            "hashes": [
                "sha256:5d06a1df6c1c43a122cd05077a3d47ae8a53dabc0f3620aba78c7d434075b2c8",
                "sha256:92c7441d3a57870e34a47bd30d55dfd4c5d49d29893e3221955608e4a10fa57d",
                "sha256:df1b0b67ab4029628264219c5f236f3c828c030e14b455cfd2b2a9d41cf3609b"
            ],
            "index": "pypi",
            "version": "==0.9.0.post1"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==0.9.1"
        }
    },
    "develop": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "pipenv": {
            "hashes": [
                "sha256:325b26672172f11cf46ee5fa31cc4c751df24f920617ca21eeb68fce70a394e2",
                "sha256:dab6b2e5347757cda7973e6c6065c30c020c5ff8b0687306b83e962aea4adc6c"
            ],
            "index": "pypi",
            "version": "==11.10.0"
        },
        "virtualenv": {
            "hashes": [
                "sha256:1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54",
                "sha256:e8e05d4714a1c51a2f5921e62f547fcb0f713ebbe959e0a7f585cc8bef71d11f"
            ],
            "version": "==15.2.0"
        },
        "virtualenv-clone": {
            "hashes": [
                "sha256:4507071d81013fd03ea9930ec26bc8648b997927a11fa80e8ee81198b57e0ac7",
                "sha256:b5cfe535d14dc68dfc1d1bb4ac1209ea28235b91156e2bba8e250d291c3fb4f8"
            ],
            "version": "==0.3.0"
        }
    }
}

$ python -m pipenv.help (macOS):

Expected result

Pipenv should correctly install each OS specific version.

Actual result

Pipenv fails to install latest version of pyarrow under Linux

Steps to replicate

Install pyarrow under Linux with pipenv

Source

HeavenVolkoff

👍1

Most helpful comment

pyarrow 0.9.0 contains a packaging problem (iirc), and the wheels they built for macOS couldn’t be installed. To remedy this (you cannot re-upload a file with the same name on PyPI, in case you’re wondering), they released 0.9.0.post1, which only contains macOS wheels. This means that when you pip install pyarrow, you get 0.9.0.post1 on macOS, and 0.9.0 everywhere else.

When Pipenv does locking, it needs to (well, tries to) make the lock file cross-platform. With a wildcard installation (pyarrow = "*"), it needs to find a latest version for all platforms—but there isn’t one! pyarrow 0.9.0.post1 only works on macOS, and 0.9.0 does not work on macOS (they deleted the broken macOS wheels). That’s why locking fails.

pip works because it only needs to care about the platform you’re running on. There is a valid latest version no matter where it is running, and that’s good enough for pip.

Knowing the root cause, you can actually work around this problem by pipenv install "pyarrow!=0.9.0,!=0.9.0.post1". As long as Pipenv does not need to consider those versions, it can resolve pyarrow correctly (to 0.8.0, until a newer version is released). But please, please acknowledge this issue to the pyarrow team. They need to know this is a problem so not to do this again in the future.

uranusjr on 16 May 2018

👍2

All 18 comments

I get your frustration but you must understand that pip and pipenv are fundamentally different. Pipenv can also install the correct version, the issue is dependency resolution which is non-deterministic. This means that what ends up in your lockfile is to some extent dependent on what the author of the package does in their setup file. If they simply don’t cut a release for one specific platform I’m afraid there isn’t much we can do.

techalchemy on 13 Apr 2018

Hey, thanks for the quick answer.
About the issue. I do understand that pipenv and pip are very different things. However, it was my understanding that pipenv would install the latest available release of a package, if that was inside the required version range. In this case, on linux that would be the 0.9.0 and on macOS the 0.9.0.post1, considering that I installed without any version constraint.

HeavenVolkoff on 13 Apr 2018

@HeavenVolkoff It’s a pretty difficult situation. Generally Pipenv tries to use the same version if possible, so your build is stable across platforms, which is the point of a lock file. You wouldn’t be able to guarantee deterministic builds if it always installs the latest version, and this is generally what you want if you use Pipenv.

The best solution to this would be for pyarrow to release 0.9.0post1 for all platforms. This is the best practice for platforms with well-established version locking practices, since it is not as reasonable to require users to specify different versions on cross-platform builds. Python, unfortunately, does not have a well-established culture in this aspect, but it is still a nice thing to do anyway.

With that said, maybe Pipfile should be able to specify a package with different versions based on environment markers, as @frostming mentioned—see https://github.com/pypa/pipenv/issues/1967#issuecomment-381063389. This would provide a workaround to this problem.

uranusjr on 13 Apr 2018

I understand the problem, and agree that the best would be for pyarrow to standardise it's release version across platforms. But the possibility for multiple entries with different markers in Pipfile would be a good work-around IMHO. Anyway, thanks for the help, I will put a comment in the other issue in favor of this feature.

HeavenVolkoff on 13 Apr 2018

It’s not about having multiple entries or not. If you run resolution on platforms A or B, you get one result, but if you run resolution on platform C, you get a different result. Since these platforms are different and resolve differently, and pipenv uses a resolver to _resolve_ dependencies (that is the nature of the tool after all), platforms A and B don’t even know that anything is different about platform C. You see, this knowledge would be impossible to have unless you _were_ platform C. That’s the nature of how pythons packaging ecosystem operates unfortunately.

This would be no problem if in fact the difference were indicated in _markers_ but markers are not the same as version numbers or post releases. Pip doesn’t even touch the wheels for incompatible platforms, so we will literally never know about a version discrepancy like this. So I know you might think you’re asking for something simple (judging by your request that we simply add another entry), but you aren’t. Please appreciate that we don’t enjoy having issues like this, if it were easy to fix we would have done it already.

techalchemy on 14 Apr 2018

Hello, I think my issue might have been misunderstood. If you initialize a clean pipenv environment, under Linux, and attempt to install pyarrow, it will fail. Should this be the correct behavior for pipenv? Even considering pyarrow versions discrepancies under different platforms.

HeavenVolkoff on 14 Apr 2018

@HeavenVolkoff it shouldn't fail in a clean environment, but what you posted was installing from an existing lockfile -- can you give us the logs from a clean install if that is failing?

techalchemy on 14 Apr 2018

Sure, here it is...
install.txt
pipenv.txt

HeavenVolkoff on 14 Apr 2018

Well based on those logs pyarrow requires futures which you can never install on python 3 so I don’t see how this will ever work

techalchemy on 14 Apr 2018

So, that's the weird thing.
if I install pyarrow using only pip, inside the pipenv venv, it install correctly.
Here's the log: install.pip.txt
It also installs correctly with pipenv install --skip-lock pyarrow.
Here's the log: install.skip.txt

HeavenVolkoff on 14 Apr 2018

Oh. Upgrade pipenv. You were hitting a resolver bug that should be fixed.

techalchemy on 15 Apr 2018

Hi, I am on the latest pipenv release, and it still fails:

HeavenVolkoff on 15 Apr 2018

👍2

Also seeing this issue, latest pipenv.

kadrach on 16 May 2018

Can’t we just tell pyarrow to please make the post1 version available for every platform? Did anyone actually try to raise the issue to them?

uranusjr on 16 May 2018

@uranusjr I don't understand the cause of the issue well enough. I can pip install pyarrow just fine, but pipenv cannot. What is platform in this context?

pipenv install --skip-lock pyarrow also works.

kadrach on 16 May 2018

pip works because it only needs to care about the platform you’re running on. There is a valid latest version no matter where it is running, and that’s good enough for pip.

uranusjr on 16 May 2018

👍2

Thank you for the detailed explanation @uranusjr. There is an existing, closed issue https://github.com/apache/arrow/issues/1950 in the arrow project.

kadrach on 17 May 2018

@kadrach Thanks for the effort!

uranusjr on 17 May 2018

👍1

Was this page helpful?

0 / 5 - 0 ratings