Pybind11: Windows 10: Enable Python Embedding Without Setting PYTHONPATH and PYTHONHOME

[wolfleyb]

I would like to reopen this issue that was started here. I have to set my PYTHONPATH and PYTHONHOME environment variables in order to embed the python interpreter in C++. Everything I read on StackOverflow says that one should avoid setting those env variables.

For example, when trying to run this simple code:

#include <pybind11/embed.h>
void main() {
   pybind11::scoped_interpreter guard{};
   pybind11::exec("print('hello world')");
}

I always get the following error if those variables are not set:

Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'python'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = 'C:\\Users\\ben.wolfley\\Desktop\\Test3\\vsstudio\\Debug\\pybind11app.exe'
sys.base_prefix = 'C:\\Users\\ben.wolfley\\Anaconda3'
sys.base_exec_prefix = 'C:\\Users\\ben.wolfley\\Anaconda3'
sys.executable = 'C:\\Users\\ben.wolfley\\Desktop\\Test3\\vsstudio\\Debug\\pybind11app.exe'
sys.prefix = 'C:\\Users\\ben.wolfley\\Anaconda3'
sys.exec_prefix = 'C:\\Users\\ben.wolfley\\Anaconda3'
sys.path = [
'C:\\Users\\ben.wolfley\\Anaconda3\\python38.zip',
'.\\DLLs',
'.\\lib',
'C:\\Users\\ben.wolfley\\Desktop\\Test3\\vsstudio\\Debug',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'`

Once PYTHONHOME and PYTHONPATH are set, it runs fine. It would be nice to run the interpreter without relying on PYTHONPATH and PYTHONHOME.

[bstaletic]

Is python in your %PATH%?

[YannickJadoul]

Does this also happen with a Python.org Python installation? If not, we might need to blame Anaconda, rather than pybind11?

[wolfleyb]

@YannickJadoul I ended up uninstalling Anaconda and installing the Python.org version and it worked. However, I still had to set PYTHONPATH and PYTHONHOME.

@bstaletic yes, I set my path to include my python home directory.

[YannickJadoul]

Wait, so does it work with Python.org without setting PYTHONPATH and PYTHONHOME? Or does it behave the same way as Anaconda?

[YannickJadoul]

Another thing to check: what happens if you try Python's raw C API embedding example, cutting pybind11 out of the loop? Does it fail the same way, or does that work?

[LeslieGerman]

@wolfleyb , regarding your question in the closed ticked:

@LeslieGerman Do you have the exact location where CPython was relying on PYTHONPATH?

Sorry I don't remember it was ~ 1-2 years ago.
But meanwhile CPython devs put lots of effort into re-designing and "streamline" how Python and (other apps) can locates Python's own DLLs, executable, and python modules.
Because it's been a pain for long...

E.g:
Similar "initfsencoding" related bug: https://bugs.python.org/issue37378
Other related bugs: https://bugs.python.org/issue14956#msg343633 , https://bugs.python.org/issue18309 , https://bugs.python.org/issue22213

And these were/are the CPython proposals:
https://www.python.org/dev/peps/pep-0432/
https://www.python.org/dev/peps/pep-0587/

As you can see PEP-432 started in 2012, and finally in June 2020 was withdrawn...

I know understand that you are not the only one who struggles with these issues, might be not too much comfort, so in a separate comment I post my limited solution which works for me. 😄

[LeslieGerman]

@wolfleyb , this is the sketch of the solution which works for me on Win 10, 64bit, with python 3.6.7, pybind11 2.4.3, in a conda environment 😄

I use CMake:

set(PYBIND11_CPP_STANDARD /std:c++17)
find_package(pybind11 CONFIG REQUIRED)
message(STATUS "Found pybind11 v${pybind11_VERSION}: ${pybind11_INCLUDE_DIRS} ${pybind11_DEFINITIONS} ${pybind11_LIBRARIES}")
message(STATUS "Found Python ${PYTHON_LIBRARY_SUFFIX} in: ${PYTHON_PREFIX}: ${PYTHON_INCLUDE_DIRS} ${PYTHON_LIBRARIES} ${PYTHON_SITE_PACKAGES}")

# Write PYTHONHOME env. var into source code by using PYTHON_LIBRARY, which was detected by pybind11.
configure_file( "${PROJECT_SOURCE_DIR}/include/detected_python.hpp.in" "${PROJECT_BINARY_DIR}/detected_python.hpp")

# ...
target_link_libraries(my_app PUBLIC pybind11::pybind11 pybind11::embed)

detected_python.hpp.in:

#pragma once

// Value of PYTHON_LIBRARY variable. The actual value is replaced during CMake configuration.
constexpr char gLinkedPythonLibraryPath[] = "@PYTHON_LIBRARY@";

My Python initializer code:

#include <detected_python.hpp>
#include <pybind11/embed.h>
#include <boost/dll.hpp>
// ...
namespace dll = boost::dll;
static void configurePython()
{
  const std::string pythonLibName = filesystem::path(gLinkedPythonLibraryPath).stem().string();

  // Now obtain the folder of gLinkedPythonLibraryPath
  boost::system::error_code ec;
  const dll::shared_library loadedPythonLibrary(
    pythonLibName, ec,
    dll::load_mode::rtld_lazy |               // POSIX only: don't do symbol resolution
      dll::load_mode::search_system_folders | // Search system folders for DLL/SO
      dll::load_mode::append_decorations);    // Use platform specific extension and prefix
  if (ec)
  {
    // error handling ... something fatally went wrong... return or throw
  }

  const filesystem::path loadedLibraryDir = loadedPythonLibrary.location().parent_path();
  const std::wstring calculatedHomeDir = loadedLibraryDir.wstring();

  if (std::getenv("PYTHONHOME")!=nullptr)
  {
    // error handling: ... PYTHONHOME variable was set, which is a no-no for us! ... return or throw
  } 

  // WARNING! According to Py_SetPythonHome(..) this must be static, or otherwise ensured that out-lives initialized CPython instance!
  static std::wstring pythonRunTimeHomeDir = std::move(calculatedHomeDir);
  Py_SetPythonHome(const_cast<wchar_t*>(pythonRunTimeHomeDir.data()));
}

// PYTHONHOME MUST be set BEFORE Python initialization (Py_Initialize()), i.e. before the Pybind11 interpreter created!
// So call configurePython() BEFORE any interpreter is created!

Summary:
I compile the content of PYTHON_LIBRARY CMake variable, which pybind11 provides during CMake config, into my code.
From that I obtain the folder of the library (I use boost::dll for this purpose, but you may simply do string operations, too), and I set that folder as PYTHONHOME by calling CPython API function: Py_SetPythonHome(..).
Important: do this initialization BEFORE any pybind interpreter is created!

This works for me
No guarantees provided, of course... 😄

[YannickJadoul]

No further reaction. I'll close this. Thanks, @LeslieGerman!