Yarp: Installation path for Python bindings is not searched by the interpreter if CMAKE_INSTALL_PREFIX is /usr/local

[PeterBowman]

Describe the bug
YARP selects an installation path for Python bindings which the Python interpreter is not able to find out-of-the-box.

To Reproduce
Configure YARP via cmake .. -DYARP_COMPILE_BINDINGS=ON -DCREATE_PYTHON=ON. CMake populates the following variables with no further action on my part (inspected via ccmake):

Python3_LIBRARY_RELEASE=/usr/lib/x86_64-linux-gnu/libpython3.8.so
_Python3_EXECUTABLE=/usr/bin/python3.8
_Python3_INCLUDE_DIR=/usr/include/python3.8
_Python3_LIBRARY_RELEASE=/usr/lib/x86_64-linux-gnu/libpython3.8.so

However, binaries (yarp.py and _yarp.so) are installed in the following path: CMAKE_INSTALL_PYTHON3DIR=lib/python3/dist-packages. This will not work for me (i.e. python3 -c 'import yarp' fails) because Python expects to find packages at /usr/local/lib/python3.8/dist-packages/ (among other paths), but note the currently configured value is lib/python3/ instead.

Expected behavior
Perhaps I could just tweak PYTHONPATH in my .bashrc, but it would be nice to have YARP select the best path for Python packages automatically. I believe this workaround wasn't necessary in the past.

Configuration

• OS: Ubuntu 20.04.2 LTS (focal), 5.8.0-44-generic
• YARP: 3.4.2+30-20210218.5+git5726d2dd4
• Compiler: g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
• Python: Python 3.8.5 (default, Jul 28 2020, 12:59:40) installed via libpython3-dev from Canonical's PPA

Additional context
I believe the lib/python3/ path in CMAKE_INSTALL_PYTHON3DIR originates from:

https://github.com/robotology/yarp/blob/d02f3fd603eeff33609fc88d2a94992622a26b5c/bindings/python/CMakeLists.txt#L32-L35

I can tweak CMAKE_INSTALL_PYTHON3DIR via ccmake, but the final installation path remains unchanged. The workaround for me consists of creating symlinks from lib/python3.8/dist-packages/ to lib/python3/dist-packages/, as suggested here.

Possible solution (also workaround, not ideal)
Use CMAKE_INSTALL_PYTHON3DIR instead of Python3_INSTDIR here:

https://github.com/robotology/yarp/blob/d02f3fd603eeff33609fc88d2a94992622a26b5c/bindings/python/CMakeLists.txt#L46-L47

https://github.com/robotology/yarp/blob/d02f3fd603eeff33609fc88d2a94992622a26b5c/bindings/python/CMakeLists.txt#L62-L63

See also
Perhaps related to some extent: https://github.com/robotology/yarp/issues/2509 (Support a user defined version of Python for YARP bindings).

[traversaro]

Related to https://github.com/robotology/robotology-superbuild/issues/428 . Apparently for python2 the existing logic (i.e. from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='') ) installed the bindings in lib/pythonMAJ.MIN/dist-packages/, but that changed when we started using Python 3. I am definitely in favor to switch to any logic such that if you install in /usr/local or /usr on Ubuntu/Debian the bindings can be searched without tweaking the PYTHONPATH.

cc @diegoferigo @GiulioRomualdi

[traversaro]

I am definitely in favor to switch to any logic such that if you install in /usr/local or /usr on Ubuntu/Debian the bindings can be searched without tweaking the PYTHONPATH.

The problem is that /usr/lib/python3 seems to be searched, and that is the reason why sysconfig.get_python_lib() returns it. Any idea why instead /usr/local/lib/python3 is not searched?

[traversaro]

I am definitely in favor to switch to any logic such that if you install in /usr/local or /usr on Ubuntu/Debian the bindings can be searched without tweaking the PYTHONPATH.

The problem is that /usr/lib/python3 seems to be searched, and that is the reason why sysconfig.get_python_lib() returns it. Any idea why instead /usr/local/lib/python3 is not searched?

Given this comment I updated the title to reflect more specifically the problem, @PeterBowman feel free to change it again if you think it make sense.

[traversaro]

I divided my comment in different sections to simplify its readability.

Why we need to use distutils.sysconfig.get_python_lib : for being platform indipendent

As for robotology-superbuild we are interesting in support both apt and conda-forge installation of Python bindings out of the box, I looked into this. On Ubuntu 20.04 apt we have:
~~~

sys.path
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
distutils.sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='')
'lib/python3/dist-packages'
~
while on Windows/conda-forge we have:
~
sys.path
['', 'C:\Users\STraversaro\Miniforge3\envs\python-tests\python39.zip', 'C:\Users\STraversaro\Miniforge3\envs\python-tests\DLLs', 'C:\Users\STraversaro\Miniforge3\envs\python-tests\lib', 'C:\Users\STraversaro\Miniforge3\envs\python-tests', 'C:\Users\STraversaro\Miniforge3\envs\python-tests\lib\site-packages']
distutils.sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='')
'Lib\site-packages'
~~~

From this two cases, we can see that hardcoding the Python install path to lib/python<MAJOR>.<MINOR>/dist-packages/ is not a viable solution.

The actual source of the problem

I investigate a bit, and this seems to be more a Debian/Ubuntu problem rather then a downstream YARP problem, see https://bugs.launchpad.net/ubuntu/+source/python3-stdlib-extensions/+bug/1832215 . If the issue gets fixed as suggested in the issue (but the issue seems to have been like that for more than an year), then probably we would need to modify our logic from:
~
execute_process(COMMAND ${Python3_EXECUTABLE} -c "from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix=''))"
OUTPUT_VARIABLE Python3_INSTDIR
OUTPUT_STRIP_TRAILING_WHITESPACE )
~
to
~
execute_process(COMMAND ${Python3_EXECUTABLE} -c "from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='${CMAKE_INSTALL_PREFIX}'))"
OUTPUT_VARIABLE Python3_INSTDIR
OUTPUT_STRIP_TRAILING_WHITESPACE )
~
as the actual install prefix would depend to the value passed to the prefix argument.

However, until the issue https://bugs.launchpad.net/ubuntu/+source/python3-stdlib-extensions/+bug/1832215 is solved, I am afraid that implementing Ubuntu-specific workarounds in the CMake may bring more problems then benefits, and so just specifying =DCMAKE_INSTALL_PYTHON3DIR=lib/python3.8/dist-packages/ seems the right strategy.

The bug we can actually quickly solve

However:

I can tweak CMAKE_INSTALL_PYTHON3DIR via ccmake, but the final installation path remains unchanged.

This is an unrelated bug, for fixing that we just need to modify the line:
https://github.com/robotology/yarp/blob/d02f3fd603eeff33609fc88d2a94992622a26b5c/bindings/python/CMakeLists.txt#L47
to
~
DESTINATION ${CMAKE_INSTALL_PYTHON3DIR})
~

[PeterBowman]

+1 from me for fixing the destination path (${Python3_INSTDIR} -> ${CMAKE_INSTALL_PYTHON3DIR}) so that we can at least have a working -DCMAKE_INSTALL_PYTHON3DIR=whatever option.

By the way, I came across this SO answer and the command python -c import site; site.getsitepackages()[0] looks promising. On both Ubuntu 18.04 and 20.04 (Python installed via Canonical's PPA) it returns /usr/local/lib/pythonx.y/dist-packages. Note this is an absolute path.

[traversaro]

By the way, I came across this SO answer and the command python -c import site; site.getsitepackages()[0] looks promising. On both Ubuntu 18.04 and 20.04 (Python installed via Canonical's PPA) it returns /usr/local/lib/pythonx.y/dist-packages. Note this is an absolute path.

How can this information be combined with CMAKE_INSTALL_PREFIX ? Just fyi, on conda/Windows the site.getsitepackages() returns:
~~~

import site
site.getsitepackages()
['', '\lib\site-packages']
~~~
and the path returned by if print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='')) is Lib\\site-packages'.

[PeterBowman]

You are right, I had not tested this with Conda, so I'm going to back off my words :). It seems difficult to find a unique solution that works well for all scenarios.

[traversaro]

Yes, it may be worth mentioning that CMake itself uses distutils.sysconfig.get_python_lib to compute some of its result variables https://cmake.org/cmake/help/latest/module/FindPython3.html#result-variables .