Kedro: PyTorch Lightining V1.0 requires fsspec>0.8 which clashes with Kedro 0.16.5

[ssakhavi]

Description

The upcoming version of PyTorch Lightning requires fsspec V0.8>0, but Kedro has a dependency V0.7<

Context

My data science pipeline is dependent on PyTorch-Lightning and of I were to update it, it will clash with Kedro

• Kedro version used (pip show kedro or kedro -V): 0.16.5
• Python version used (python -V): 3.7
• Operating system and version: Ubuntu

[Minyus]

I'm suffering from the same issue.
BTW, the correct version of pytorch-lightning is 0.10.0, not V1.0.

[ssakhavi]

@Minyus
They've announced that 0.10 and 1.0 are the same.
https://github.com/PyTorchLightning/pytorch-lightning/releases/tag/0.10.0

[Minyus]

@ssakhavi
I see. Sorry, I just checked PyPI and not the release note.

[Minyus]

If QB team doesn't want users to use fsspec >= 0.7.0, how about showing just a warning instead of setting upper limit in requirements.txt?

[ssakhavi]

To be honest I don't know how much of a breaking change the newer fsspec introduces to Kedro.

Maybe someone from the team can elaborate.

[crypdick]

@ssakhavi you may want to see #489 , according to https://github.com/quantumblacklabs/kedro/issues/489#issuecomment-694135052 the fix is already in develop branch and will be rolled out in v0.17.0

[ssakhavi]

@crypdick excellent. Thanks.

[Minyus]

@crypdick
@ssakhavi

As of 4th Nov 2020, https://github.com/quantumblacklabs/kedro/blob/develop/requirements.txt#L5 sets fsspec>=0.5.1, <0.8, which means that current develop branch does not support fsspec >= 0.8 and cannot solve this PyTorch Lightning issue.

[lorenabalan]

Hi guys, we've updated the fsspec requirements to include 0.8.x and it will be made available in kedro 0.17.0. Thank you very much for your patience with this! Relevant commit: https://github.com/quantumblacklabs/kedro/commit/151cbcd1acaae25da4633ab87fabbf354d60059b