Kedro: [KED-1454] reload_kedro magic script not working with jupyter

Created on 22 Feb 2020 · 11Comments · Source: quantumblacklabs/kedro

Description

I created a new kedro environment and ran kedro jupyter notebook to create a notebook.
Then I execute the %reload_kedro command which gave the following error:
2020-02-22 09:45:31,139 - root - ERROR - Kedro appears not to be installed in your current environment or your current IPython session was not started in a valid Kedro project.

I also executed the kedro ipython command to get the following error:
Traceback (most recent call last):
File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/application/current.py", line 6, in
from contextvars import ContextVar
ModuleNotFoundError: No module named 'contextvars'

Context

How has this bug affected you? What were you trying to accomplish?
i am unable to use jupyter notebook with kedro context

Steps to Reproduce

create a new project
create a notebook with kedro jupyter notebook
make some catalog entries
try to run the %reload_kedro magic command

Expected Result

it should load the catalog entries

Actual Result

error

-- If you received an error, place it here.

2020-02-22 09:45:31,139 - root - ERROR - Kedro appears not to be installed in your current environment or your current IPython session was not started in a valid Kedro project.

ImportError Traceback (most recent call last)
in
----> 1 get_ipython().run_line_magic('reload_kedro', '')

/anaconda3/envs/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
2312 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
2313 with self.builtin_trap:
-> 2314 result = fn(args, *kwargs)
2315 return result
2316

~/Desktop/personal/code/DataScience/kaggle_future_sales/future_sales/.ipython/profile_default/startup/00-kedro-init.py in reload_kedro(path, line)
20 try:
21 import kedro.config.default_logger
---> 22 from kedro.context import KEDRO_ENV_VAR, load_context
23 from kedro.cli.jupyter import collect_line_magic
24 except ImportError:

ImportError: cannot import name 'KEDRO_ENV_VAR' from 'kedro.context' (/anaconda3/envs/python3/lib/python3.7/site-packages/kedro/context/__init__.py)

-------------------------------------------------------------------------------
Starting a Kedro session with the following variables in scope
startup_error, context
Use the line magic %reload_kedro to refresh them
or to see the error message if they are undefined
-------------------------------------------------------------------------------
ipython
Traceback (most recent call last):
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/application/current.py", line 6, in <module>
    from contextvars import ContextVar
ModuleNotFoundError: No module named 'contextvars'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/envs/future_sales/bin/ipython", line 5, in <module>
    from IPython import start_ipython
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/IPython/__init__.py", line 56, in <module>
    from .terminal.embed import embed
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/IPython/terminal/embed.py", line 16, in <module>
    from IPython.terminal.interactiveshell import TerminalInteractiveShell
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 19, in <module>
    from prompt_toolkit.enums import DEFAULT_BUFFER, EditingMode
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/__init__.py", line 16, in <module>
    from .application import Application
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/application/__init__.py", line 1, in <module>
    from .application import Application
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/application/application.py", line 38, in <module>
    from prompt_toolkit.buffer import Buffer
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/buffer.py", line 28, in <module>
    from .application.current import get_app
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/application/current.py", line 8, in <module>
    from prompt_toolkit.eventloop.dummy_contextvars import ContextVar  # type: ignore
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/eventloop/__init__.py", line 1, in <module>
    from .async_generator import generator_to_async_generator
  File "/anaconda3/envs/future_sales/lib/python3.6/site-packages/prompt_toolkit/eventloop/async_generator.py", line 5, in <module>
    from typing import AsyncGenerator, Callable, Iterable, TypeVar, Union
ImportError: cannot import name 'AsyncGenerator'

Your Environment

Include as many relevant details about the environment in which you experienced the bug:
kedro, version 0.15.5

Kedro version used (pip show kedro or kedro -V):
Python version used (python -V): Python 3.6.0 :: Continuum Analytics, Inc.
Operating system and version: macOS 10.14.6

Bug Report

Source

madhao

👍1

All 11 comments

@madhao Are you still dealing with this issue? We're going to put it on our backlog to fix.

yetudada on 9 Mar 2020

$ kedro ipython
In [1]: context.io.list()                                                                                           
Out[1]: 
['example_iris_data',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

In [2]: catalog.list()                                                                                              
Out[2]: 
['example_iris_data',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

Added a new dataset called new_dataset.

In [4]: %reload_kedro                                                                                               
2020-05-27 12:07:49,756 - root - INFO - ** Kedro project myproject
2020-05-27 12:07:49,756 - root - INFO - Defined global variable `context` and `catalog`

In [5]: context.io.list()                                                                                           
Out[5]: 
['example_iris_data',
 'new_dataset',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

In [6]: catalog.list()                                                                                              
Out[6]: 
['example_iris_data',
 'new_dataset',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

Using a new environment myenv.

$ kedro ipython --env myenv

In [1]: context.io.list()
...                                                                                       
Out[1]: 
['iris_data',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

In [2]: catalog.list()                                                                                              
Out[2]: 
['iris_data',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

Added my_dataset dataset:

In [8]:  %reload_kedro
...                                                                                              
2020-05-27 12:26:17,647 - root - INFO - ** Kedro project myproject
2020-05-27 12:26:17,647 - root - INFO - Defined global variable `context` and `catalog`

In [9]: context.io.list()                                                                                           
...
Out[9]: 
['iris_data',
 'my_dataset',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

In [10]: catalog.list()                                                                                             
Out[10]: 
['iris_data',
 'my_dataset',
 'parameters',
 'params:example_test_data_ratio',
 'params:example_num_train_iter',
 'params:example_learning_rate']

@madhao
Do you still experience this issue?

andrii-ivaniuk on 27 May 2020

Let me test

madhao on 27 May 2020

@madhao I also check it in jupyter notebook (running kedro jupyter notebook --env myenv) and it also works fine.

andrii-ivaniuk on 27 May 2020

Hi I am also encountering this issue after updating Kedro from 0.15.9 to 0.16.1
I got the red line under KEDRO_ENV_VAR while opening in pycharm. Everything works fine before upgrading to 0.16.1.

I think my problem is simply I do not know how to find back the KEDRO_ENV_VAR after upgrading, may you please help to have a look? I have already google-ing for the solution for around 2 hours.

Inside kedro_cli.py, line 54

from kedro.context import KEDRO_ENV_VAR, load_context

Cannot find reference 'KEDRO_ENV_VAR' in '__init__.py'

btw sorry I'm quite new on GitHub, still improving my commenting skill.
@andrii-ivaniuk
@madhao

Full terminal command results:

(kedro_env) myname@myname-MacBook-Pro introduction-kedro % kedro -version
/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/site-packages/kedro/context/__init__.py:46: DeprecationWarning: All the modules in kedro.context have been moved to kedro.framework.context, and kedro.context will be removed in Kedro 0.17.0. Please update import paths from kedro.context to kedro.framework.context in your Kedro project.
DeprecationWarning,
/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/site-packages/kedro/cli/__init__.py:44: DeprecationWarning: All the modules in kedro.cli have been moved to kedro.framework.cli, and kedro.cli will be removed in Kedro 0.17.0. Please update import paths from kedro.cli to kedro.framework.cli in your Kedro project.
DeprecationWarning,
Traceback (most recent call last):
File "/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 619, in main
kedro_cli = importlib.import_module("kedro_cli")
File "/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/Users/myname/Desktop/my projects ;D/introduction-kedro/kedro_cli.py", line 54, in
from kedro.context import KEDRO_ENV_VAR, load_context
ImportError: cannot import name 'KEDRO_ENV_VAR'
Traceback (most recent call last):
File "/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 619, in main
kedro_cli = importlib.import_module("kedro_cli")
File "/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/Users/myname/Desktop/my projects ;D/introduction-kedro/kedro_cli.py", line 54, in
from kedro.context import KEDRO_ENV_VAR, load_context
ImportError: cannot import name 'KEDRO_ENV_VAR'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/myname/anaconda3/envs/kedro_env/bin/kedro", line 8, in
sys.exit(main())
File "/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 623, in main
_handle_exception(f"Cannot load commands from {kedro_cli_path}")
File "/Users/myname/anaconda3/envs/kedro_env/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 643, in _handle_exception
raise KedroCliError(msg)
kedro.framework.cli.utils.KedroCliError: Cannot load commands from /Users/myname/Desktop/my projects ;D/introduction-kedro/kedro_cli.py

packages in environment at /Users/myname/anaconda3/envs/kedro_env

oscartse68 on 28 May 2020

Hi I am also encountering this issue after updating Kedro from 0.15.9 to 0.16.1
I got the red line under KEDRO_ENV_VAR while opening in pycharm. Everything works fine before upgrading to 0.16.1.
snip

Hi @oscartse68 👋

This is expected from the upgrade. If you look at the RELEASE.md release notes, there's a section that lays out how to migrate your project from 0.15.9 to 0.16.1. The relevant bit here is under the section Migration for CLI and KEDRO_ENV environment variable.

Let me know if that helps!

mzjp2 on 29 May 2020

🎉1

@mzjp2 Hi thanks! It works like charm, will look at the release note before raising the question next time 🤣 May I ask does quantumblack provide any YouTube/learning channel for Kedro?

oscartse68 on 31 May 2020

@mzjp2 Hi thanks! It works like charm, will look at the release note before raising the question next time 🤣 May I ask does quantumblack provide any YouTube/learning channel for Kedro?

@oscartse68 We're so glad that you're all sorted out on this one. I have one question and recommendation for you as a follow-up to this question you've asked.

My question is, "Why would you like to see a YouTube or video-based learning channel for Kedro?"

And then on that I would definitely recommend checking out the DataEngineerOne series on YouTube. It's a series of Kedro tutorials on different functionality done by a super user of Kedro.

yetudada on 31 May 2020

👍1

@yetudada Thankyou for your reply, I am quite new in engineering and would like to kickstart my project with kedro, DataEnginerOne is definitely a great channel for beginners!

oscartse68 on 31 May 2020

@oscartse68 This is good to know! If you think of anything that can be done to make explaining Kedro a bit easier then let us know but otherwise we hope that you enjoy DataEngineerOne. We're really big fans!

yetudada on 31 May 2020

@madhao Thanks one more time for reporting this issue. We are still waiting for your feedback.
Meanwhile, I'm going to close the issue since there is no evidence that it is still relevant.

You are free to reopen the issue if you still experience the problem.

andrii-ivaniuk on 2 Jun 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

[KED-1667] Chronocoding: Solving the Problem of State Tracking with Temporally Sensitive DAGs

tamsanh · 4Comments

Remove quotation marks (Part 2)

yetudada · 3Comments

Jupyter Notebook and iPython launch issues

josephhaaga · 3Comments

Parameter variations during single kedro execution instance (multiple Pipeline executions with different parameters)

Mar1cX · 4Comments

Reusing pipeline elements in a served model scenario

kaemo · 3Comments