Pyo3: ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

[tcztzy]

🐛 Bug Reports

I am working for writing Python extension, but facing import submodule error.

ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

What do I expected

Import submodule like usual Python package.

🌍 Environment

• Your operating system and version:
  ArchWSL version 19.11.16.0 on Windows Subsystem Linux 2 version Windows Insiders Slow ring 10.0.19041

$ uname -a
Linux xxxxxx 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 GNU/Linux

• Your python version:

$ python -V
Python 3.8.1

• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?:

$ pyenv virtualenv system xxxxxx

• Your rust version (rustc --version):

rustc 1.43.0-nightly (c9290dcee 2020-02-04)

• Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?
  Yes.

💥 Reproducing

Please provide a minimal working example. This means both the rust code and the python.

Please also write what exact flags are required to reproduce your results.
src/lib.rs

use pyo3::prelude::*;
use pyo3::{wrap_pyfunction, wrap_pymodule};
use pyo3::types::IntoPyDict;

#[pyfunction]
fn subfunction() -> String {
    "Subfunction".to_string()
}

#[pymodule]
fn submodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(subfunction))?;
    Ok(())
}

#[pymodule]
fn supermodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

Cargo.toml

[package]
name = "supermodule"
version = "0.1.0"
authors = ["Tang Ziya <[email protected]>"]
edition = "2018"

[package.metadata.maturin]
classifier = [
    "Development Status :: 3 - Alpha",
    "Programming Language :: Python",
    "Programming Language :: Rust",
]

[lib]
name = "supermodule"
crate-type = ["cdylib", "rlib"]

[features]
default = []

[dependencies]
pyo3 = { git = "https://github.com/PyO3/pyo3", features = ["extension-module"] }

[dev-dependencies]

$ maturin develop
$ python -c "from supermodule.submodule import subfunction"
ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

[davidhewitt]

This is unfortunately a flaw in the way Python imports modules from native extensions. I think if you run import supermodule first, then from supermodule.submodule import subfunction will succeed.

[waymost]

I've run into this same issue with version 0.8.5 on Mac and no combination of imports that I've tried has worked, including the above suggestion.

[davidhewitt]

Thanks for the note. I must have mis-remembered that.

Based on stack overflow the solution looks like you might be able to create a symlink to the main library which has the name of the submodule: https://stackoverflow.com/questions/48706842/python-3-x-c-extension-module-and-submodule

[sunjay]

You can fix this in a somewhat hacky way by adding the module to sys.modules manually. From your Rust code, this can look like:

#[pymodule]
fn supermodule(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.run("\
import sys
sys.modules['supermodule.submodule'] = submodule
    ", None, Some(module.dict()))?;

    Ok(())
}

The documentation claims that sys.modules is writable, so this approach should not break anytime soon. The only caveat is that while this works when executed by a Python interpreter, it does not work with static analysis tools like pylint.

After a lot of trial and error, I am fairly convinced that the only "fully robust" solution that will also make static analysis tools happy is to have your entire module hierarchy represented in the file system. Whether you do this with Python files or with separate extension modules is up to you. I agree with the comments in PyO3/maturin#266 that we should look into how numpy, pandas, etc. handle this and see if there is a better way.

It would be great if in the meantime PyO3 could emit code to modify sys.modules automatically so from supermodule.submodule import ... would work out of the box.