Cibuildwheel: Is there an example of building something on Linux/MacOS/Windows using make/cmake?

[bt2901]

I'm a bit confused about this, so I apologize if my question doesn't make any sense.

I'm trying to compile a C++ library which uses cmake to install it's Python interface. I managed to make it work inside a manylinux1 docker (as a result, the library is pip-installable on Linux), but I also need prebuilt macOS binaries. I figured that cibuildwheel was the way to go.

As far as I understand, I need to invoke cibuildwheel with the most build-related commands moved to the CIBW_BEFORE_BUILD variable. Also, I need to get rid of for PYBIN in /opt/python/*/bin; do since cibuildwheel takes responsibility for setting up the environment.

However, I'm fuzzy on details: should I call cibuildwheel after (or instead of) cmake? Should I somehow change the build process to explicitly invoke cibuildwheel in the end?

PS: If you need more details, my configuration is here.

[YannickJadoul]

@bt2901 I'm not entirely getting your build setup or issue. Is the C++ library built in a separate step before making wheels, rather than as a part of pip wheel/setup.py?

As far as I understand, I need to invoke cibuildwheel with the most build-related commands moved to the CIBW_BEFORE_BUILD variable.

So, as the name indicates CIBW_BEFORE_BUILD contains commands that are run before every build of a wheel. You can do anything you want there. But cfr. the above: why is building the library not a part of your setup.py?

Also, I need to get rid of for PYBIN in /opt/python/*/bin; do since cibuildwheel takes responsibility for setting up the environment.

I don't really understand this. cibuildwheel indeed takes care of calling pip wheel (which in turn invokes setup.py's build command, by default) for all Python versions specified by CIBW_BUILD and CIBW_SKIP. There's no need to loop, normally?

Feel free to have a look at my library's setup (based on pybind11's setup), where setup.py calls CMake to build the C++ code.

[bt2901]

Is the C++ library built in a separate step before making wheels, rather than as a part of pip wheel/setup.py?

Yes. The steps I need to take to build my library inside manylinux container:

• install boost
• install a newer version of cmake (required for a particular dependency)
• run cmake / make (they call setup.py bdist_wheel as a part of their execution)

As a result of this process, libartm.so is created. The setup.py itself is responsible just for Python API/compatibility.

The last bullet point is executed inside a for PYBIN in /opt/python/*/bin; do loop.

But cfr. the above: why is building the library not a part of your setup.py?

I think the main reason here is historical, but the current way has an advantage for developers: changes in Python API do not require re-building the library (and changes in python interface are much more common than changes in C++ core).

You suggest transferring the control of CMake to the setup.py, correct? I'll look into Parselmouth, thanks!

[YannickJadoul]

@bt2901 Right, I see. Is it then possible for example to separate the setup.py bdist_wheel step, and have the first two-and-a-half bullets called in CIBW_BEFORE_BUILD and the half bullet (setup.py bdist_wheel) called by cibuildwheel's pip wheel invocation?

At any rate, CIBW_BEFORE_BUILD and pip wheel are called within a loop over all Python versions specified by CIBW_BUILD and CIBW_SKIP, and the whole point of cibuildwheel is to do that loop for you, so you should not need it anymore.

I understand your further concerns. I have a similar thing in Parselmouth: during development, everything is driven by CMake. The difference is that CMake does not call setup.py, but I have setup.py set up to call CMake (and enable the packing of the produced .so as a wheel, and thus also enable pip wheel).

Another option would of course be that we would make the actual build command configurable with a CIBW_... option, but I'm not too eager for that. After all, setup.py/pip is the standardized way of creating a Python package and wheel, so allowing that to be changed could result in cibuildwheel not building correct wheels anymore (since wheel does not need to be involved anymore), in case of wrong use...

[bt2901]

So I finally got everything to work through setup.py and it does look better that way. Also, now it bundles some prebuilt boost libraries, which I guess is a correct action here.

There's an issue with MacOSX distribution, however. It fails at the final stage with the following:

  running build_py
  creating build
  creating build/lib
  creating build/lib/artm
  copying ./python/artm/__init__.py -> build/lib/artm
  copying ./python/artm/artm_model.py -> build/lib/artm
  copying ./python/artm/batches_utils.py -> build/lib/artm
  copying ./python/artm/dictionary.py -> build/lib/artm
  copying ./python/artm/hierarchy_utils.py -> build/lib/artm
  copying ./python/artm/lda_model.py -> build/lib/artm
  copying ./python/artm/master_component.py -> build/lib/artm
  copying ./python/artm/regularizers.py -> build/lib/artm
  copying ./python/artm/score_tracker.py -> build/lib/artm
  copying ./python/artm/scores.py -> build/lib/artm
  creating build/lib/artm/wrapper
  copying ./python/artm/wrapper/__init__.py -> build/lib/artm/wrapper
  copying ./python/artm/wrapper/api.py -> build/lib/artm/wrapper
  copying ./python/artm/wrapper/constants.py -> build/lib/artm/wrapper
  copying ./python/artm/wrapper/exceptions.py -> build/lib/artm/wrapper
  copying ./python/artm/wrapper/spec.py -> build/lib/artm/wrapper
  copying ./python/artm/wrapper/utils.py -> build/lib/artm/wrapper
  running egg_info
  creating python/bigartm.egg-info
  writing ./python/bigartm.egg-info/PKG-INFO
  writing dependency_links to ./python/bigartm.egg-info/dependency_links.txt
  writing requirements to ./python/bigartm.egg-info/requires.txt
  writing top-level names to ./python/bigartm.egg-info/top_level.txt
  writing manifest file './python/bigartm.egg-info/SOURCES.txt'
  reading manifest file './python/bigartm.egg-info/SOURCES.txt'
  writing manifest file './python/bigartm.egg-info/SOURCES.txt'
  creating ../../../../../../Users
  error: could not create '../../../../../../Users': Permission denied
  ----------------------------------------
  ERROR: Failed building wheel for bigartm
  Running setup.py clean for bigartm

(I build on Travis)

This is really weird because the build is happening in the "/Users/travis/build/bt2901/bigartm/" directory, hence '../../../../../../Users' and "/Users/" mean the same thing. I have no idea why setup.py tries to access this directory in such a roundabout way. Invoking python3 -m cibuildwheel --output-dir wheelhouse with sudo didn't help (it causes cibuildwheel to fail auto-detecting platforms)

The complete logs (for both Linux and OSX builds) are here: https://travis-ci.org/github/bt2901/bigartm/builds/674780234

Do you, by any chance, have any ideas what could be causing this?

[Czaki]

Why You try to do something in "User" directory? Maybe try use abspath?

With modern pip part of building wheel is done in tmp dir. Maybe also this part so it broke path?

[bt2901]

Why You try to do something in "User" directory?

That's the directory where my repository is cloned by either Travis or cibuildwheel. Am I supposed to do my wheel building in some other directory?

Maybe try use abspath?

The package_dir={'': './python/'} argument of setuptools.setup() is required to be relative path. In all other places, I use absolute paths.

I get what do you mean: I already had my share of errors caused by relative paths, os.chdir() and python being in an unexpected place. I think I fixed them all, but maybe I missed something. My setup.py is here, does anything seem wrong to you?

With modern pip part of building wheel is done in tmp dir. Maybe also this part so it broke path?

Do you mean that pip tries to copy every relevant file to <tmp dir>/build/lib/ and then move the result back into the required directory (but fails to correctly locate it because the paths are weird)?

[YannickJadoul]

@bt2901 I haven't looked deeply into your build setup, but could it be that you're somehow expecting CMake to be executed in a certain directory during that failing install step?

The catch might be that pip wheel copies your directory to a temp directory, first, seen for example here.

[YannickJadoul]

There are indeed a lot of paths in there setup.py. Any reason why you're not getting e.g. src_abspath from the setuptools build_ object, etc?

E.g., in the pybind11 example, the path is determined from build_ext.get_ext_fullpath(ext.name) (see here). I'm not familiar with making custom classes, but I would expect you can get all of this kind of data from there, rather than hardcoding CI paths and environment variables in there?

[Czaki]

I think that bad is hard coding path to src. Please try:

src_abspath = os.path.dirname(__file__)

instead of

working_dir = os.path.abspath(os.getcwd())

# maybe we are inside Travis container? Fallback
src_abspath = os.environ.get('CI_BUILD_DIR')
if src_abspath is None:
    if os.environ.get("AUDITWHEEL_PLAT"):
        # we are inside PyPa manylinux docker
        src_abspath = "/project/"
    elif shutil.which("python") == "/tmp/cibw_bin/python":
        # we are inside macosx virtual machine
        username = 'bt2901'  # TODO: do not hardcode this
        src_abspath = f"/Users/travis/build/{username}/bigartm/"
    else:
        src_abspath = working_dir**

For example your code will fail with pip install path/to/dir

[joerick]

Another suggestion (apologies!) - the os.chdir could a little suspect to me. I suspect you don't need to do this at all, as the convention for calling setup.py is from that working dir. But, if you do, be aware that most python code assumes the working dir doesn't change after it's imported. If you're sure you need to chdir, import only os and do the chdir operation before importing anything else.

[bt2901]

Huge thanks to everyone! I missed the last os.chdir() and indeed it did something bad.

I have another problem now: OSX build fails to include anything relevant in the wheel (the wheel takes 5KB of space and contains nothing but metadata) and Linux build fails with

+ mv '/tmp/repaired_wheels/*.whl' /output
mv: cannot stat `/tmp/repaired_wheels/*.whl': No such file or directory
Checking for common errors...
cibuildwheel-a3284d5b-e2f5-411a-8b88-d9719797ecd4

I strongly suspect that it is related to paths as well. I have some ideas on how to tackle this, but any advice helps.

[Czaki]

@bt2901 I think that problem with you code is using path in many places, so build files are placed wrong.

I go through your code. First thing. Do not install wheel on linux. auditwheel is pinned to specific version of wheel. And you can simply damage build system. cibuildwheel manage installing
proper version of base build package (pip, wheel, setuptools, virtualenv) so not update this package in CIBW_BEFORE_BUILD.

second thing. May setup.py executable without administrator privileges. You can place libraries files in package and it python should find it.

Not execute make install in setup.py. Package should be installed from source and do not need sudo privileges. Or set proper prefix to instal in the source directory. (If you remove sudo need I can try to found source of your problem)

BEFORE_BUILD is executed before each python version installation. You do not need to reinstall cmake every each run. Same with boost. Cmake is also distributed through pypi so you can try to install it with pip,

You can read current part from PWD environment variable. so you do not need to guessing proper path you can use (which is obsolete, because executing script does not change your path if you not source them)

CI_BUILD_DIR=$PWD

...

cd $CI_BUILD_DIR

[bt2901]

Thanks for the very detailed answer! I implemented some of your suggestions and everything goes smoother.

Now all builds are passing, but the content of wheel files is wrong:

• libartm.so is inside *.whl/lib/ directory which is probably correct
• wheel also stores a lot of build artifacts (such as CMakeCache.txt and a lot of extra folders)
• wheel does not contain artm/wrapper directory where I expect to find Python API. As far as I can tell, Python API is not included at all.

second thing. May setup.py executable without administrator privileges. You can place libraries files in package and it python should find it.

I'm not sure I understand this. Does setup.py require administrator privileges at the moment? I have sudo: required inside the travis.yml, but I'm pretty sure that it does not actually do anything (travis warns me that this option is deprecated).

BEFORE_BUILD is executed before each python version installation. You do not need to reinstall cmake every each run. Same with boost.

Yes, I'm aware of this. I was basing this part of code on github.com/joerick/cibuildwheel/issues/54: I create a temporary directory built-lib and install boost only if it does not exist already.

Cmake is also distributed through pypi so you can try to install it with pip,

Thanks for the tip!

[Czaki]

Now all builds are passing, but the content of wheel files is wrong:

• libartm.so is inside *.whl/lib/ directory which is probably correct
• wheel also stores a lot of build artifacts (such as CMakeCache.txt and a lot of extra folders)

I suggest to read about this
https://cmake.org/cmake/help/latest/variable/CMAKE_INSTALL_PREFIX.html
and process build in tmp dir then install it to build_dir

• wheel does not contain artm/wrapper directory where I expect to find Python API. As far as I can tell, Python API is not included at all.

When I run it locally it is included

I'm not sure I understand this. Does setup.py require administrator privileges at the moment? I have sudo: required inside the travis.yml, but I'm pretty sure that it does not actually do anything (travis warns me that this option is deprecated).

yes

[YannickJadoul]

@bt2901 Really, as said before, just have a look here: https://github.com/pybind/cmake_example

I'm not saying it's perfect, but it does a few things, where it makes sure that the directories from setup.py get passed on to CMake correctly, that CMake produces files in the right output directory, that CMake finds the correct version of Python, etc, etc.