Kedro: [KED-1055] Dependency issue with pyarrow

[AlexandreOuellet]

Description

There is a regression on windows for kedro 0.15.1 and 0.15.0 on windows.
When running pipenv install kedro===0.14.3, everything is fine, but when doing pipenv install kedro===0.15.0 or pipenv install kedro===0.15.1, it fails at a depency on pyarrow 0.14.1

Context

I can't install the latest version of kedro on windows 10

Steps to Reproduce

1. create a fresh python environment and activate it
2. pipenv install kedro

Expected Result

The dependency should be resolved, and kedro 0.15 should be able to install on windows

Actual Result

Fails to install kedro

[pipenv.exceptions.InstallError]: ['Collecting pyarrow==0.14.1 (from -r C:\\Users\\OUELAL\\AppData\\Local\\Temp\\pipenv-1e42tyv4-requirements\\pipenv-ocm118ym-requirement.txt (line 1))']
[pipenv.exceptions.InstallError]: ['ERROR: Could not find a version that satisfies the requirement pyarrow==0.14.1 (from -r C:\\Users\\OUELAL\\AppData\\Local\\Temp\\pipenv-1e42tyv4-requirements\\pipenv-ocm118ym-requirement.txt (line 1)) (from versions: 0.4.1, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.11.1, 0.12.0, 0.12.1, 0.13.0, 0.14.0)', 'ERROR: No matching distribution found for pyarrow==0.14.1 (from -r C:\\Users\\OUELAL\\AppData\\Local\\Temp\\pipenv-1e42tyv4-requirements\\pipenv-ocm118ym-requirement.txt (line 1))']

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V):
• Python version used (python -V): Python 3.6.8
• Operating system and version: Windows 10 Pro

[AlexandreOuellet]

as a workaround, you can pin the pyarrow version to 0.14.0, and it seems to do the trick.

Here is my Pipfile :

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
pyarrow = "==0.14.0"
kedro = "*"

[requires]
python_version = "3.6"

What I find strange is that you need to pin the pyarrow version within the packages for it to work.

[yetudada]

Hi @AlexandreOuellet! Thanks for raising this error. I'll put it on our backlog to investigate what's going on and reproduce the error. pyarrow is currently unpinned in our requirements.txt so we find it very strange that you had to do this as a workaround.

[AlexandreOuellet]

I've had a look, and pyarrow was pinned to 0.12.0 in the kedro 0.14.3 release.

Also, this doesn't happen with a pip install, only pipenv. I've had a similar issue with airflow, where essentially an unpinned dependency got updated, and it's requirements of a common library was pinned to a higher version. So airflow depended on X and Y. X also depended on an unpinned version of Y and suddenly the dependency got all screwed up when X got updated. Pip was not making an error (although it should have), but pipenv was correctly throwing an error.

edit: To clarify, it was essentially this issue : https://github.com/pypa/pipenv/issues/2596

[yetudada]

Hi @AlexandreOuellet, thanks for tracking down the root cause of this problem. I'm going to track this issue because another user has raised #95. Perhaps the solution to that issue will address this one too.

[lorenabalan]

Hi @AlexandreOuellet I wasn't able to reproduce this since the newer versions of pyarrow. Let us know if you've come up against this recently, otherwise I will go ahead and close this issue.

[lorenabalan]

Closing this as resolved, looks like it was fixed on either pipenv or pyarrow side.