Kedro: Is anyone using Kedro with Pipenv?

[jayBana]

I would like to get some suggestions on some potential workflows for using Kedro with Pipenv.

[yetudada]

Hi @jayBana, what potential workflow would you be looking for with Pipenv? We just suggest conda because it fits into the Anaconda ecosystem.

However, all it really facilitates is being able to better manage your Python package dependencies. You can use anything.

[DmitriiDeriabinQB]

@jayBana I ran pipenv install kedro and the installation succeeded. You can then run pipenv shell and then execute any relevant kedro commands inside the shell. Please let us know if you have any specific questions about pipenv.

[jayBana]

At the moment, I have to do the following if I want to use Pipenv (For this example, I want this project to reside in the kedro-pipenv directory):

1. mkdir kedro-pipenv && cd kedro-pipenv
   
   
   
   • This is needed so that the virtualenv created is "tied" to the project directory which only really means that it has a name that is based on the directory name from which the pipenv install kedro or pipenv shell commands are executed.
   
   
   • Unfortunately, at the moment Pipenv doesn't have the functionality to support custom virtualenv names.
   
   
2. pipenv install kedro
   
   
   
   • In my example, this generates a virtualenv with name kedro-pipenv-AB9IGRnB which resides in the following location ~/.local/share/virtualenvs/kedro-pipenv-AB9IGRnB/
   
   
3. pipenv shell
4. kedro info
   
   
   
   • all is looking good, kedro was successfully installed in the virualenv handled by Pipenv
   
   
5. cd .. & kedro new
   
   
   
   • I navigate one directory up and provide kedro-pipenv as the directory name for the project. Given that the directory was already created before in step 1, this fails which is expected and I get the following message:
   
   

cookiecutter.exceptions.OutputDirExistsException: Error: "/Users/xyz/projects/kedro-pipenv" directory already exists
Run with --verbose to see the full exception
Error: Failed to generate project.

In order to "work around" this, I do the following whilst still in the same virtualenv as before:

• mv kedro-pipenv kedro-pipenv_tmp
  
  
  
  • rename the existing directory
  
  
• kedro new
  
  
  
  • I provide kedro-pipenv as the directory name for the project.
  
  
• mv kedro-pipenv_tmp/Pipfile* kedro-pipenv && rm -rf kedro-pipenv_tmp
  
  
  
  • This step is needed so that the Python version and the single kedro dependency is maintained.
  
  
• cd kedro-pipenv
• kedro install
• Using kedro build-reqs for managing project requirements.

I am familiar with conda as well and it seems that it is a much cleaner way of handling environments for Kedro at the moment.

However, for most of my other projects I have been using pyenv in conjunction with Pipenv for environment and dependency management. This allows me to have environment information tied to specifi project spaces by having a Pipfile in each of my projects' root directory.

[DmitriiDeriabinQB]

@jayBana If you need your Pipenv environment root to point to Kedro project root then your solution is probably the optimal one as things stand. If you can live with Pipenv environment root directory and Kedro project directory having different names, then you can do something like this:

mkdir kedro-pipenv && cd kedro-pipenv
pipenv install kedro
pipenv run kedro new  # create Kedro project inside 'kedro-pipenv'
cd <project-dir>
pipenv run kedro run  # still works even in nested directory 

As you already pointed out, it's easier with conda, since its environment (unlike pipenv) is not tied to any particular directory.

Please let me know if it resolves your issue.

P.S. We are trying to grow our StackOverflow community, which is probably a better place for questions like that. It would be great if next time you can raise a question there, thank you!

[jayBana]

@DmitriiDeriabinQB Thank you for the quick response. Your latest response answers my initial question.
I have posted this to StackOverflow as well with the kedro tag.