Cibuildwheel: Using cibuildwheel for a python library with a cython extension

[assaftibm]

Hello,

I'd appreciate some help in using cibuildwheel in a project that I work on. Not sure if this is the right place to ask for help, so if there's a better place please let me know.

This project is an implementation of an algorithm for text clustering, named sequential Information Bottleneck (sIB). It is a product of IBM Research, and recently IBM contributed it to the open source community. The GitHub is here: https://github.com/IBM/sib.

The source code and supplementary files (setup.py and requirements.txt) are already in the repository, and I'm looking for some assistance in creating the build system.

In IBM, we used a custom-made build system for this. For creating Linux wheels, we used the manylinux dockers and a script for iterating over several python versions in each container. The wheel files were collected in a mounted shared directory.

One of the things that complicate the story is that the python code uses a cython extension and this extension needs to be built as part of the build process. In fact, there are two setup.py files - one for the wheel and one for the extension. Therefore, for every python version, a short script is called from the project root directory (For MacOS and Windows a similar script is used):

#!/bin/bash

# build the extension
cd src/sib/c_package
python setup.py build_ext --inplace

# build the wheel
cd ../../..
python setup.py sdist bdist_wheel 

I assume that this is not how the project is supposed to be built when using cibuildwheel. I looked at two examples: 1 and 2 but I'm not sure I fully understand them.

Looking at 2, if I got this right, build_wheels builds and packages the wheel file on every os mentioned in the matrix, and build_sdist packages the source code into a tar.gz file. Also, upload_pypi depends on the completion of the other two jobs since it uploads their result to the index.

What I'm not sure about is where the building of the extension should be done. Should it be a separate job before build_wheels, or part of the build_wheels job. And if it is part of build_wheels, should it be a separate run command, or invoked as part of the command python -m cibuildwheel --output-dir wheelhouse. In the latter case, I might need to change the main setup.py in some way to make he extension build part of the wheel build but I'm unfamiliar with a way to do that.

Any help with this will be appreciated.

Also, I wanted to ask why the package vcpython27 is being installed in the examples above.

Thanks.

[Czaki]

I strongly suggest to move build cython extension to top level setup.py. Because current solution does not allow to install package with simple pip install path.
Something like

ext = [Extension('sib.c_package.c_sib_optimizer_sprase',
                 ['sib/c_package/c_sib_optimizer_sprase.pyx'],
                 extra_compile_args=compile_extra_args,
                 extra_link_args=link_extra_args,
                 language="c++")]

There is no top level setup.py in repository.

If You do not want to change this You can use `CIBW_BEFORE_BUILD="cd src/sib/c_package; python setup.py build_ext --inplace"

More documentation in: https://cibuildwheel.readthedocs.io/en/stable/options/#before-build

vcpython27 is installed to allow build wheel for python 2.7 on Windows. New MSVC versions does not support python 2.7 by default and it is simpler to install vcpython27 than setup this,

[YannickJadoul]

To complement @Czaki's excellent answer: put succinctly, all cibuildwheel does for you is loop over all Python versions and call pip wheel . there (which in turn will call python setup.py bdist_wheel). Everything that's in setup.py is up to you (though it should produce a wheel, ofc).

As you say, sdist is a source distribution (so just a zip/tar.gz the source), should only be built once for all Python versions, and is not a wheel. So currently not handled by cibuildwheel (see #173, though). There are some good Python docs on this.

[assaftibm]

@Czaki and @YannickJadoul - many thanks for your help.

Indeed, I forgot to push the commit with the top setup.py when I opened this issue.

Based on the Extension definition that @Czaki wrote, I was able to update my setup.py and now it builds the extension as part of the wheel creation. Subsequently, I deleted the internal setup.py.

The repository is updated now and I tried to run a first build with cibuildwheel. It seems like I need to fix some issue with the python version (my code requires python 3.6 or newer, and the build fails when it attempts to build for python 2.7). Once I get the build to pass I'll close this issue.

[YannickJadoul]

(my code requires python 3.6 or newer, and the build fails when it attempts to build for python 2.7).

Have a look at the docs.

[assaftibm]

@YannickJadoul thanks, after some attempts, I managed to limit the build to versions >= 3.6.

However, the build is failing and I'm not sure why. Perhaps it is because it doesn't cythonize the pyx file into cpp and therefore gcc doesn't find the cpp file. Could you please take a look at the build log here? Errors are reported on lines 86 and 118 (unless I missed something else).

When I run pip wheel . -w built_wheel --no-deps locally it works.

[YannickJadoul]

I've never worked with Cython before. But you need to first transform the Cython source to C (or C++?), no?

Again, not really linked to cibuildwheel. Try checking out a clean version of your code (without all generated files), then run pip wheel . -w built_wheel --no-deps locally?

https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#basic-setup-py

[assaftibm]

Thanks @YannickJadoul
My problem was that Cython wasn't installed. I added it as a build requirement in pyproject.toml

[Czaki]

You forget to cythonize extension. You can look to this project for working example.

https://github.com/4DNucleome/PartSeg/blob/master/setup.py

[assaftibm]

@Czaki do you refer to the line:

ext_modules=cythonize(extensions),

that in my case looks:

ext_modules = extensions,

I see what you mean. I'll try that.

However, I'm not sure how my last builds passed and the artifacts contain the .so/.pyd files that are generated by compiling the cpp generated by cython. I mean - if it did not cythonize, how could it get a cpp file to compile. I didn't test the artifacts yet, so perhaps they won't work. This is the latest build output:
https://github.com/IBM/sib/actions/runs/174204817

[assaftibm]

I added the cythonize command to my setup.py, as I understand that this is how it should work, although even when I build without it, I can definitely see in pip logs that it runs 'cytohnize' and saves the cpp that it generated into a temporary location.. Then it uses the C++ compiler to build the binary file. From what I can tell, the only difference between with and without explicitly calling 'cytohnize' in the setup.py is that the wheel that is created with the cythonize command includes the .cpp file that cython generates, but both include the .so/.pyd file and it looks OK.

Anyway, I also encountered a new problem when I added 'cythonize' to setup.py. I get an error when building sdist:

Build sdist
1s
    CIBW_BEFORE_BUILD: pip install Cython
Run python setup.py sdist
  python setup.py sdist
  shell: /bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.7.8/x64
    CIBW_BEFORE_BUILD: pip install Cython
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
##[error]Process completed with exit code 1.

As you can see, I tried to add CIBW_BEFORE_BUILD: pip install Cython but it seems ineffective when running python setup.py sdist. So I'm not sure how to solve this. Perhaps I should add another run statement? In general, I would have preferred the build system to see that Cython is mentioned in pyproject.toml and to install it automatically, but it seems like pyproject.toml is effective only when running pip wheel?

[joerick]

python setup.py sdist doesn't involve cibuildwheel, so our options will do nothing there. All the CIBW_ options only apply when running cibuildwheel. If you need to preinstall something before calling python setup.py, you can use pip directly.

[assaftibm]

Thanks, I handled this issue.

I now encounter a different problem - I added tests to the project, and to run them I need to first install packages like numpy and scipy. On the platform used for macosx_10_9_x86_64 with pp36-pypy36_pp73, a pre-built binary wheel of numpy and scipy does not exist, so pip tries to build it from the source code. However, it fails and therefore the build fails.

I see multiple errors in the build of scipy, such as:

error: library mach has Fortran sources but no Fortran compiler found
    ----------------------------------------
    ERROR: Failed building wheel for scipy
  Successfully built numpy
  Failed to build scipy
  ERROR: Could not build wheels for scipy which use PEP 517 and cannot be installed directly

Do you know of any solution for this?

I know that this is not a cibuildwheel issue, but I thought you might have encountered it with the MacOS PyPy image.

[assaftibm]

This is the build that fails:
https://github.com/IBM/sib/runs/888902275?check_suite_focus=true

[Czaki]

brew install gcc to install gfortran,

[Congee]

@joerick

you can use pip directly.

where should I use pip other than in CIBW_BEFORE_BUILD?

FWIW, my setup.py depends on a library zstandard to decompress .xz tarballs. Where should I install the zstandard package?

[YannickJadoul]

@congee, what's the problem with CIBW_BEFORE_BUILD? That should work, no? There's also CIBW_BEFORE_ALL, nowadays.

[Congee]

@Congee, what's the problem with CIBW_BEFORE_BUILD? That should work, no? There's also CIBW_BEFORE_ALL, nowadays.

The problem is exactly what this thread talks about - third party packages imported by setup.py cannot be installed via pip. CIBW_BEFORE_ALL does not work, either.

Namely, I can do python setup.py build_ext but cannot do python -m cibuildwheel

Below are logs from failed CircleCI. As you can see, zstandard, cython, and requests are all "installed" but still the error ModuleNotFoundError: No module named 'zstandard' 😕

C:\cibw\python\pythonx86.3.8.4\tools\python.exe
C:\tools\miniconda3\python.exe
+ python --version
Python 3.8.4
+ python -c "import struct; print(struct.calcsize('P') * 8)"
+ python -m pip install --upgrade pip -c C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\resources\constraints.txt
Requirement already up-to-date: pip==20.1.1 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from -c C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\resources\constraints.txt (line 18)) (20.1.1)
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the 'C:\cibw\python\pythonx86.3.8.4\tools\python.exe -m pip install --upgrade pip' command.
+ pip --version
pip 20.1.1 from c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages\pip (python 3.8)
+ pip install --upgrade setuptools wheel -c C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\resources\constraints.txt
Requirement already up-to-date: wheel==0.34.2 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from -c C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\resources\constraints.txt (line 14)) (0.34.2)
Requirement already up-to-date: setuptools==46.4.0 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from -c C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\resources\constraints.txt (line 19)) (46.4.0)
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the 'c:\cibw\python\pythonx86.3.8.4\tools\python.exe -m pip install --upgrade pip' command.
+ pip install requests zstandard cython
Requirement already satisfied: requests in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (2.24.0)
Requirement already satisfied: zstandard in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (0.14.0)
Requirement already satisfied: cython in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (0.29.21)
Requirement already satisfied: certifi>=2017.4.17 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from requests) (2020.6.20)
Requirement already satisfied: idna<3,>=2.5 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from requests) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from requests) (1.25.10)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from requests) (3.0.4)
Requirement already satisfied: cffi>=1.11 in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from zstandard) (1.14.2)
Requirement already satisfied: pycparser in c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages (from cffi>=1.11->zstandard) (2.20)
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the 'c:\cibw\python\pythonx86.3.8.4\tools\python.exe -m pip install --upgrade pip' command.
+ pip wheel C:\Users\circleci\project -w C:\Users\circleci\AppData\Local\Temp\cibuildwheeldff_qkas\built_wheel --no-deps
Processing c:\users\circleci\project
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: 'c:\cibw\python\pythonx86.3.8.4\tools\python.exe' 'c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages\pip\_vendor\pep517\_in_process.py' get_requires_for_build_wheel 'C:\Users\circleci\AppData\Local\Temp\tmpl8e52vjd'
       cwd: C:\Users\circleci\AppData\Local\Temp\pip-req-build-7j1ptq50
  Complete output (16 lines):
  Traceback (most recent call last):
    File "c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 280, in <module>
      main()
    File "c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-xp73eej1\overlay\Lib\site-packages\setuptools\build_meta.py", line 149, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-xp73eej1\overlay\Lib\site-packages\setuptools\build_meta.py", line 130, in _get_build_requires
      self.run_setup()
    File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-xp73eej1\overlay\Lib\site-packages\setuptools\build_meta.py", line 145, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 15, in <module>
      import zstandard as zstd
  ModuleNotFoundError: No module named 'zstandard'
  ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\cibw\python\pythonx86.3.8.4\tools\python.exe' 'c:\cibw\python\pythonx86.3.8.4\tools\lib\site-packages\pip\_vendor\pep517\_in_process.py' get_requires_for_build_wheel 'C:\Users\circleci\AppData\Local\Temp\tmpl8e52vjd' Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the 'c:\cibw\python\pythonx86.3.8.4\tools\python.exe -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "C:\tools\miniconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\tools\miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\__main__.py", line 319, in <module>
    main()
  File "C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\__main__.py", line 236, in main
    cibuildwheel.windows.build(build_options)
  File "C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\windows.py", line 231, in build
    ], env=env)
  File "C:\Users\circleci\AppData\Roaming\Python\Python37\site-packages\cibuildwheel\windows.py", line 26, in call
    subprocess.check_call([str(a) for a in args], env=env, cwd=cwd, shell=True)
  File "C:\tools\miniconda3\lib\subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'wheel', 'C:\\Users\\circleci\\project', '-w', 'C:\\Users\\circleci\\AppData\\Local\\Temp\\cibuildwheeldff_qkas\\built_wheel', '--no-deps']' returned non-zero exit status 1.
PS C:\Users\circleci\project>

[Congee]

The error in CircleCI also happens in Travis CI ^^^

[Czaki]

@Congee If it is still same project IBM sib you need to add zstandard to requires in pyproject.toml

[build-system]
requires = ["setuptools", "wheel", "Cython"]

When pyproject.toml is present then isolated build mode is used and requires need to have listed all packages required for build.

[YannickJadoul]

The problem is exactly what this thread talks about - third party packages imported by setup.py cannot be installed via pip. CIBW_BEFORE_ALL does not work, either.

Namely, I can do python setup.py build_ext but cannot do python -m cibuildwheel

I don't see how it's the same error. You should be perfectly able to install something with pip in CIBW_BEFORE_BUILD.

Below are logs from failed CircleCI. As you can see, zstandard, cython, and requests are all "installed" but still the error ModuleNotFoundError: No module named 'zstandard' confused

Yes, this is weird. I'd suggest to print sys.path before importing zstandard to get some debug output.
If that doesn't show anything, then please show us some more details. Where's your build configuration and setup.py? Where's the failing build log (i.e., do the others Python versions work?)?

Also, please open a new issue. I don't see how this is linked to the original issue, so this is only causing more confusion.

[YannickJadoul]

@assaftibm & @Czaki, I think the original issue was resolved, right? At least it's been quiet here for a while. So I'll close this issue. Feel free to reopen if it wasn't resolved.

[Congee]

Thank you @Czaki @YannickJadoul My problem was solved already by adding a pyproject.toml. Should've replied earlier

[assaftibm]

Thanks @Czaki and @YannickJadoul for all your help.