ITK Python wrapping does not build _ITKPyBasePython.pyd on Windows

[tbirdso]

Description

After following instructions to build the ITK Python wrapping on Windows, an error is thrown when attempting to use ITK modules in a Python virtual environment. The _ITKPyBasePython.pyd file and other binaries are not located at the wrapping location pointed to by WrapITK.pth.

Steps to Reproduce

1. git clone https://github.com/InsightSoftwareConsortium/ITK.git
2. Configure ITK CMake build, setting PYTHON3_ROOT_DIR, PY_SITE_PACKAGES_PATH, and PYTHON3_EXECUTABLE (or PYTHON_EXECUTABLE) to point to the virtual environment.
3. Generate Visual Studio project and build the project in Visual Studio 2019.
4. Add ITKWrap.pth to the site-packages folder in the Python virtual environment.
5. Activate and enter the Python virtual environment.
6. from itk import imread
7. imread(r'image.nrrd')

Expected behavior

Image is read without error

Actual behavior

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\itkExtras.py", line 671, in imread template_reader_type = itk.ImageFileReader File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\itkLazy.py", line 78, in __getattribute__ itkBase.itk_load_swig_module(module, namespace) File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\itkBase.py", line 102, in itk_load_swig_module itk_load_swig_module(dep, namespace) File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\itkBase.py", line 102, in itk_load_swig_module itk_load_swig_module(dep, namespace) File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\itkBase.py", line 110, in itk_load_swig_module l_module = loader.load(swig_module_name) File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\itkBase.py", line 262, in load l_spec.loader.exec_module(l_module) # pytype: disable=attribute-error File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "C:\repos\ITK-builds\Wrapping\Generators\Python\itk\support\..\ITKPyBasePython.py", line 13, in <module> from . import _ITKPyBasePython ImportError: cannot import name '_ITKPyBasePython' from 'itk' (C:\repos\ITK-builds\Wrapping\Generators\Python\itk\__init__.py)

Versions

ITK 5.2.0

Environment

Windows
CMake 3.19.2
Python 3.8.5

Additional Information

I found the _ITKPyBasePython.pyd file in the itk-build/lib directory but not the itk-build/Wrapping/Generators/Python/itk directory as expected. However changing the filepath in ITKWrap.pth causes Python to not recognize the itk package at all.

The issue does not occur in Python virtual environments which install the ITK distribution with pip install itk.

[tbirdso]

I've found that copying all files (.pyd, .exp, and .lib) from itk-build/lib/Release to itk-build/Wrapping/Generators/Python/itk appears to temporarily solve the issue. I no longer receive an error when I attempt to use ITK functions in my Python environment. Are these simply being built to the wrong folder? In which case, is there a default CMake configuration that I should set to change this?

[dzenanz]

If you are building with shared libraries, did you update your path to include itk-build/bin/Release and maybe itk-build/lib/Release?

[tbirdso]

Yes, I've had my PATH and LIB environment variables set to include both paths.

[dzenanz]

@hjmjohnson I haven't tried building against a virtual environment since the Python build refactoring. Is it supposed to work?

[brad-t-moore]

I checked out master and ITKSetPython3Vars doesn't look modified. You should NOT have to explicitly set your virtual environment if you activate it before calling CMake. Otherwise, you should set the CMake hints Python3_ROOT_DIR and Python3_FIND_REGISTRY. The CMake documentation isn't clear to me on whether you need to set Python3_FIND_REGISTRY to LAST in order for Python3_ROOT_DIR to work.

That being said, none of the Python3 CMake variables affect where the .pyd files go. A bad setting of the Python3 hints (i.e., CMake using the wrong Python3 in your system) could result in errors where the .pyd files exist in the correct location but are not found (because they cause a linker error during execution). Last I checked, and this was before the Python build refactoring, the .pyd files had to be in the same directory as their .py namesakes. So if they aren't, that's a bug in the build.

If you are using the Python wrapping you do NOT need to modify your PATH or LIB environment variables to include ITK as I believe those don't affect Python's module search (but it's been a while, they might affect but that definitely would not be the preferred way to set that search path).

So, if the .pyd files aren't going with the .py files, that's a bug in ITK and there's no CMake variable to fix it.

[brad-t-moore]

@thewtex @dzenanz This is me being rusty on CMake, but we couldn't we explicitly/preemptively declare Python3_ROOT_DIR and Python3_FIND_REGISTRY (optionally if windows is detected) in the parent CMakeLists.txt so that they show up as configuration variables and provide our own docstring? I feel like people's first guess on how to specify Python is wrong (looking at the Discourse for this issue).

[dzenanz]

explicitly/preemptively declare Python3_ROOT_DIR and Python3_FIND_REGISTRY

We totally can, and should! Right now it is a secret setting. Patch incoming.

[thewtex]

@tbirdso well done on the report.

A patch to address the runtime output location is
https://github.com/InsightSoftwareConsortium/ITK/pull/2254