Pcl: [Bug] Wrong python system path import when add PCL library in CMakelists with Pybind11

[ltphy]

Describe the bug
There is a wrong python path imported when including PCL library together with Pybind11 in CMakelists.txt file, but it imports the python environment correctly when there is only pybind11.

Context
I want to create a CMakeLists.txt file that can build pybind11 and PCL together while the python path should be the current python environment.

Expected behavior
Should import the python from the current environment path.
When CMakelists only includes pybind11, the python path environment is used correctly.

• Python code:

import sys
print(sys.path)

['/home/phy/.conda/envs/env/lib/python37.zip', '/home/phy/.conda/envs/env/lib/python3.7', '/home/phy/.conda/envs/env/lib/python3.7/lib-dynload', '/home/phy/.local/lib/python3.7/site-packages', '/home/phy/.conda/envs/env/lib/python3.7/site-packages', '/home/phy/.conda/envs/env/lib/python3.7/site-packages/PyYAML-5.3.1-py3.7-linux-x86_64.egg', '/home/phy/.conda/envs/env/lib/python3.7/site-packages/numba-0.50.1-py3.7-linux-x86_64.egg', '/home/phy/.conda/envs/env/lib/python3.7/site-packages/llvmlite-0.33.0-py3.7-linux-x86_64.egg', '/home/phy/env']



md5-fc072728d7ffc8c23de5dbdc53b75359



import sys
print(sys.path)



md5-dcbb10422e0a98226ce9bdf5d93fb63d



['/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/phy/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', u'.',]



md5-63adbdec7cccee8bd045f25c8bb6f480



cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(cli_cplusplus)
find_package(Python3 COMPONENTS Interpreter)
find_package( pybind11 REQUIRED)
find_package( PCL 1.8 REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories( ${Boost_INCLUDE_DIR} ${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions("-Wall -DCPU_ONLY -std=c++11 -O3 -g")
target_link_libraries (main PRIVATE pybind11::embed ${PCL_LIBRARIES} -lstdc++fs)

Your Environment:

• OS: Ubuntu 18.04
• Compiler: [GCC 7.5.0]
• PCL Version [1.8.1]

[kunaltyagi]

@divmadan PTAL

I don't know how PCL would be producing the behavior since it doesn't know/care about Python as of pcl-1.11.1-rc1

This behavior might be coming from Boost.Python instead. Please see if replacing PCL with Boost Python reproduces the error

[ltphy]

I install Boost.Python using:

conda install -c anaconda py-boost

on my current environment, but It produces conflicts. Therefore, I create another environment and install Boost.Python

CMakelists.txt

cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(cli_cplusplus)

find_package(Python COMPONENTS Interpreter)
find_package( pybind11 REQUIRED)
find_package(Boost COMPONENTS python REQUIRED)

include_directories(${PYTHON_INCLUDE_DIRS})

include_directories( ${Boost_INCLUDE_DIR})

add_definitions("-Wall -DCPU_ONLY -std=c++11 -O3 -g")
target_link_libraries (main PRIVATE pybind11::embed ${Boost_LIBRARIES} ${PYTHON_LIBRARIES} -lstdc++fs)
target_include_directories(main PRIVATE ${PYTHON_INCLUDE_DIRS})

And in Python code it produces a correct python environment path:

['/home/phy/.conda/envs/env/lib/python37.zip', '/home/phy/.conda/envs/env/lib/python3.7', '/home/phy/.conda/envs/env/lib/python3.7/lib-dynload', '/home/phy/.local/lib/python3.7/site-packages', '/home/phy/.conda/envs/env/lib/python3.7/site-packages']

And when I try to build PCL with Pybind11 in the new environment. The pybind11 produces error when build cmakelists.

[divmadan]

Hi @ltphy!
I am developing python bindings for PCL as a part of GSoC 2020. I don't use python in cmake, rather I use the setuptools-cmake combination for compiling (see: pybind's example repo).
If you want, have a look at my CMakeLists.txt.

I agree with @kunaltyagi , I don't think PCL influences python environment. You can tinker with result variables of FindPython3, to get some inferences.

A few observations:

• ['/home/phy/.conda/envs/env/lib/python37.zip', '/home/phy/.conda/envs/env/lib/python3.7', 
  '/home/phy/.conda/envs/env/lib/python3.7/lib-dynload', '/home/phy/.local/lib/python3.7/site-packages', 
  '/home/phy/.conda/envs/env/lib/python3.7/site-packages', '/home/phy/.conda/envs/env/lib/python3.7/site-packages/PyYAML- 
  5.3.1-py3.7-linux-x86_64.egg', '/home/phy/.conda/envs/env/lib/python3.7/site-packages/numba-0.50.1-py3.7-linux- 
  x86_64.egg', '/home/phy/.conda/envs/env/lib/python3.7/site-packages/llvmlite-0.33.0-py3.7-linux-x86_64.egg', '/home/phy/env']
  

  • conda env
  • PYTHONPATH:
    
    
    
    • python3.7
    
    
    • site-packages, not dist-packages
    
    

• ['/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', 
  '/usr/lib/python2.7/lib-dynload', '/home/phy/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', 
  '/usr/lib/python2.7/dist-packages', u'.',]
  

  • Global, not conda env
  • PYTHONPATH:
  • python2.7
  • dist-packages, not site-packages

It might be something to do with some apt-package which installs python files as apt installs in dist-packages. To find those files, python has to look into dist-packages of python2.7. PYTHONPATH addition/removal of those specific dist-packages can help.
For example, I add the path to python3-clang (which I installed via apt) explicitly.

Some spitballing:

• I'm not well versed with conda, but if possible, disabling global packages search might help.
• Trying a stricter solution like a Docker environment, so that you can control explicitly what is used.

Hope this helps.

Also, on a side note, I would love to know about your project and its design, if possible :slightly_smiling_face:

[ltphy]

Thank you very much, @divmadan. I did try to use Docker.
However, the problem lies in the CMakelists configuration and does not belong to PCL.
Mr. Henry from pybind11 helps me to resolve the problem Issues 2385 :smile: