Kedro: [KED-2373] ModuleNotFound when loading kedro context with 0.17

[jeffkayne]

Description

Hello,
I have faced an error while upgrading to Kedro 0.17 with the kedro context. I am able to use kedro ipython to start a Kedro session, with the context variable loaded. However, if I try and load the context directly from ipython using

from kedro.framework.context import load_context
context = load_context(proj_path)

I get an error. I have also tried using the new KedroSession, and I get the same error. I have also tried running the .ipython/profile_default/startup/00-kedro-init.py script and get the same error (see below).

Expected Result

I should be able to load the context, in order to use commands like context.io.load("")

Actual Result

I am getting a ModuleNotFoundError with the name of my package.
The error I get when I use from kedro.framework.context import load_context:

/Users/jeffreykayne/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/context/context.py:666: DeprecationWarning: `kedro.framework.context.load_context` is now deprecated in favour of `KedroSession.load_context` and will be removed in Kedro 0.18.0.
  DeprecationWarning,
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-82df189e622a> in <module>
      1 proj_path = Path.cwd()
----> 2 context = load_context(proj_path)

~/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/context/context.py in load_context(project_path, **kwargs)
    670 
    671     context_class = _get_project_settings(
--> 672         metadata.package_name, "CONTEXT_CLASS", KedroContext
    673     )
    674 

~/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/project/settings.py in _get_project_settings(package_name, property_name, default)
     34     package_name: str, property_name: str, default: Any = None
     35 ) -> Any:
---> 36     settings = import_module(f"{package_name}.settings")
     37     return getattr(settings, property_name, default)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py in import_module(name, package)
    125                 break
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 
    129 

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'ds'

The error I get when I use from kedro.framework.session import KedroSession:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-12-dac7a85b2eb6> in <module>
      4 
      5 metadata = _get_project_metadata("/Users/jeffreykayne/Documents/Clients/Econocom/econocom-ccai/ds")
----> 6 session = KedroSession.create(metadata.package_name, "/Users/jeffreykayne/Documents/Clients/Econocom/econocom-ccai/ds")

~/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/session/session.py in create(cls, package_name, project_path, save_on_close, env, extra_params)
    199             project_path=project_path,
    200             session_id=generate_timestamp(),
--> 201             save_on_close=save_on_close,
    202         )
    203 

~/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/session/session.py in __init__(self, session_id, package_name, project_path, save_on_close)
    167         self.save_on_close = save_on_close
    168         self._package_name = package_name
--> 169         self._store = self._init_store()
    170 
    171     @classmethod

~/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/session/session.py in _init_store(self)
    247     def _init_store(self) -> BaseSessionStore:
    248         store_class = _get_project_settings(
--> 249             self._package_name, "SESSION_STORE_CLASS", BaseSessionStore
    250         )
    251         classpath = f"{store_class.__module__}.{store_class.__qualname__}"

~/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages/kedro/framework/project/settings.py in _get_project_settings(package_name, property_name, default)
     34     package_name: str, property_name: str, default: Any = None
     35 ) -> Any:
---> 36     settings = import_module(f"{package_name}.settings")
     37     return getattr(settings, property_name, default)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py in import_module(name, package)
    125                 break
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 
    129 

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'ds'

Your Environment

• Kedro version used (pip show kedro or kedro -V):

Name: kedro
Version: 0.17.0
Summary: Kedro helps you build production-ready data and analytics pipelines
Home-page: https://github.com/quantumblacklabs/kedro
Author: QuantumBlack Labs
Author-email: None
License: Apache Software License (Apache 2.0)
Location: /Users/jeffreykayne/Documents/Clients/Econocom/econocom-ccai/ds/venv/lib/python3.7/site-packages
Requires: setuptools, cachetools, fsspec, PyYAML, python-json-logger, gitpython, click, pluggy, toml, jupyter-client, pip-tools, toposort, cookiecutter, jmespath, anyconfig
Required-by: 

• Python version used (python -V):

Python 3.7.9

[sansagara]

I have this issue too. It is present even when using one of the available kedro starters.

If i remove the .ipython directory, i don't get the error on kedro ipython, but the context variable is not even set! (Or the reload line magic).

What's worse: I think this is related to a bug I-m seeing where template variables are not expanded correctly when using them on catalog entries. So, I have catalog entries like:

raw_kol_profiles@pandas:
  <<: *csv_pandas
  filepath: ${pandas_prefix}/raw/orbit/${raw_version}/raw_kol_profiles.csv

This does not work anymore :(

[nagendratekuri]

is kedro 17 not supporting templated variables?

int_calls_planned:
<<: *parquet
filepath: ${data_prefix}/int/orbit/${version}/int_calls_planned

This is not working for me as well

[lorenabalan]

@jeffkayne

I'm looking into it, it might a use case we overlooked, but let me confirm. As a workaround in the meantime you can add the following snippet in your notebook before running load_context:

from kedro.framework.cli.utils import _add_src_to_path

_add_src_to_path(Path.cwd()/"src", Path.cwd())

Assuming your source directory is called "src" and Path.cwd() is path to the root of the project (feel free to adapt accordingly).

@sansagara @nagendratekuri

I think I know what your issue is, but it's not a bug - it's a feature we've been slowly introducing since 0.16.5. We're moving things away from the context. If you're using the TemplatedConfigLoader, the way to use it is via register_config_loader hook in hooks.py, as opposed to overriding the function ProjectContext._create_config_loader. In fact ProjectContext should eventually go away entirely, unless needed for very complex customisations.

See our migration guide.

• Copy changes made to ConfigLoader. If you have defined a custom class, such as TemplatedConfigLoader, by overriding ProjectContext._create_config_loader, you should move the contents of the function in src/<package_name>/hooks.py, under register_config_loader.

[sansagara]

Indeed @lorenabalan That was the issue!!