Cibuildwheel: Repairing Windows wheels

[myselfhimself]

Hello,
I am attempting to build a wheel for a C++ Python binding of G'MIC with .dll dependents of a wheel file on Windows MSYS2 with Github Actions.
This is for python 3.8 for now. Once I have understood how to "repair" the wheel by including the dependent .dll files into the wheel file, I will proceed to use cibuildwheel most probably. My goal is to iterate on Python versions and use my MSYS2 gcc/g++ and bash commands instead of using Visual Studio.
I will give a count of my progress and struggles using cibuildwheel here. I am already using the latter for MacOS Clang builds (though disabled again for now).

Related to #329 and https://github.com/myselfhimself/gmic-py/issues/5 as well as this gmic-py windows-support pull-request

[myselfhimself]

One thing which I have not yet understood, is if cibuildwheel already has a default windows repair command in place, or if it does nothing for now, but just has the slot environment variable for injecting such a repair command.

My idea for repairing under msys2 is to run ldd on the compiled C++/Python .dll and copy all matching /mingw64/bin dll files into a directory, which then would be included within the wheel and if needed loaded again at module import time.

[myselfhimself]

I have read a bit through related https://github.com/joerick/cibuildwheel/blob/master/cibuildwheel/windows.py and https://github.com/joerick/cibuildwheel/blob/master/examples/github-minimal.yml already.
What is unclear for someone not reading the source is that the tool used for grabbing various Python versions for cross-version building is not MSYS2's pacman, not chocolatey, but nuget. This could be advertised in the documentation for the Windows platform support section (unless I am mistaken).

[mayeut]

You might want to have a look at https://discuss.python.org/t/delocate-auditwheel-but-for-windows/2589

[myselfhimself]

Hello yes I had found that discussion exactly as well and the last
commenting person Vinayak Mehta created a script named wheel_repair.py and
leveraging machomangler for repairing windows wheels
https://github.com/vinayak-mehta/pdftopng/blob/6e4a9d589228f66dfbb6f0197740b0b7c1bef9b3/scripts/wheel_repair.py
where there the dependent dlls that get inserted into the wheel are even
renamed to avoid conflicts with same named preinstalled dll on any target
install.
One issue is that this script is not yet a separate python package for
example, it is in the middle of a pdf2png project.

so 7. 11. 2020 v 12:11 odesílatel Matthieu Darbois [e-mail]
napsal:

You might want to have a look at
https://discuss.python.org/t/delocate-auditwheel-but-for-windows/2589

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/joerick/cibuildwheel/issues/459#issuecomment-723432194,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJU5QVJM4KF5RJIRMJFHWLSOUTNHANCNFSM4TMU2I6A
.

[myselfhimself]

I will find one solid way to repair hopefully next week, maybe using
wheel_repair.py by @vinayak-mehta and use cibuildwheel to iterate on python
versions.. will keep you posted

so 7. 11. 2020 v 12:28 odesílatel Jonathan-David SCHRODER <
[email protected]> napsal:

Hello yes I had found that discussion exactly as well and the last
commenting person Vinayak Mehta created a script named wheel_repair.py
and leveraging machomangler for repairing windows wheels
https://github.com/vinayak-mehta/pdftopng/blob/6e4a9d589228f66dfbb6f0197740b0b7c1bef9b3/scripts/wheel_repair.py
where there the dependent dlls that get inserted into the wheel are even
renamed to avoid conflicts with same named preinstalled dll on any target
install.
One issue is that this script is not yet a separate python package for
example, it is in the middle of a pdf2png project.

so 7. 11. 2020 v 12:11 odesílatel Matthieu Darbois <
[email protected]> napsal:

You might want to have a look at
https://discuss.python.org/t/delocate-auditwheel-but-for-windows/2589

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/joerick/cibuildwheel/issues/459#issuecomment-723432194,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJU5QVJM4KF5RJIRMJFHWLSOUTNHANCNFSM4TMU2I6A
.

[YannickJadoul]

One thing which I have not yet understood, is if cibuildwheel already has a default windows repair command in place, or if it does nothing for now, but just has the slot environment variable for injecting such a repair command.

The latter; see https://cibuildwheel.readthedocs.io/en/stable/options/#repair-wheel-command

[vinayak-mehta]

One issue is that this script is not yet a separate python package for example, it is in the middle of a pdf2png project.

It's not a separate Python package yet because of limitations I mention in https://github.com/vinayak-mehta/pdftopng/issues/3#issuecomment-723554567 I'm open to any feedback you all have about how I could improve it and possibly create a package out of it!

I'll keep an eye out on this issue as I'm interested in learning about other ways to build Windows wheels! (since @myselfhimself mentioned about using MSYS2 instead VS)

The latter; see https://cibuildwheel.readthedocs.io/en/stable/options/#repair-wheel-command

Yes! I'm using CIBW_REPAIR_WHEEL_COMMAND_WINDOWS to call that script on Windows.

@mayeut @YannickJadoul @joerick and other contributors, thank you for cibuildwheel!

[myselfhimself]

One thing which I have not yet understood, is if cibuildwheel already has a default windows repair command in place, or if it does nothing for now, but just has the slot environment variable for injecting such a repair command.

The latter; see https://cibuildwheel.readthedocs.io/en/stable/options/#repair-wheel-command

Thank you @YannickJadoul for this documentation pointer. So obviously cibuildwheel has no default windows repair wheel command at this time, and its default for the CIBW_REPAIR_WHEEL_COMMAND_WINDOWS environment variable is ''.

[YannickJadoul]

its default for the CIBW_REPAIR_WHEEL_COMMAND_WINDOWS environment variable is ''

Yep. There's no official PyPA tool either, so I don't think we'll add that before there's an officially supported tool. But you have the possibility to use your own.

[myselfhimself]

@vinayak-mehta thank you again for posting clear links to related cibuildwheel files in your project and explaining your usage.

As stated on your png2pdf's issue tracker just now, both your wheel repair .bat or .py commands would be useful or maybe an extension less python command, as under MSYS2, .py or .sh/.bash files are more linux-like than .bat, although all of those are executable (MSYS2 deals with .dll and .exe natively also).

[myselfhimself]

its default for the CIBW_REPAIR_WHEEL_COMMAND_WINDOWS environment variable is ''

Yep. There's no official PyPA tool either, so I don't think we'll add that before there's an officially supported tool. But you have the possibility to use your own.

@vinayak-mehta mentioned here that he sees 2 drawbacks before making his script onto pypa. I think this is really minor. So Vinayak, feel now encouraged twice by @YannickJadoul and me, to release it as a stand alone project, really!! Although this would mean extra maintainer work in the future for you, though it is pure-python, Python 3.x compatible at wide possible and for one OS only.

[YannickJadoul]

Does this mean we can consider your issue solved? (at least the part where this is related to cibuildwheel)

[myselfhimself]

https://github.com/vinayak-mehta/pdftopng/blob/355282f57760f0d494b4a36fa5a83ea097cad75b/.github/workflows/build.yml#L16 is a proof that cibuildwheel x wheel repairing x msvc works. If you like to have a clean tracker, then OK, this could be closed. @vinayak-mehta or I could come and reopen the issue or create a new one, when I have msys2 news or he/someone has independent windows wheel repair executable news.

[myselfhimself]

I have fixed wheel_repair.py for the .dll case (was working for .pyd-based projects already), within the gmic-py project only for now.

@vinayak-mehta would you agree, when you have time for it, to create an independent repository for this project, where I co-contribute or make PRs? We could slowly go towards a PyPA module from there, and have it referenced and used by cibuildwheel.

[YannickJadoul]

Just curious: is this mingw/msys-specific? Or would it work for all Windows wheels?
I still thought mingw isn't supported by Python, though, since Python is built with MSVC?

[myselfhimself]

There are several shipped Python versions on Windows. The Python that is
shipped as an MSYS2 Windows package is compiled using GCC, this is what one
sees when running python (for opening a python shell) or python --version.
Possibly the python.org's python executables are compiled with MSVC.

The wheel_repair.py script uses mostly machomangler
https://github.com/njsmith/machomachomangler to repair .dll or .pyd files
and is most probably ABI agnostic in terms of MSVC/MINGW preference,
because it has just been working for my mingw compiled gmic-py module's
.dll, as well as @vinayak-mehta 's MSVC .pyd in the past (if not mistaken).

út 10. 11. 2020 v 20:57 odesílatel Yannick Jadoul [e-mail]
napsal:

Just curious: is this mingw/msys-specific? Or would it work for all
Windows wheels?
I still thought mingw isn't supported by Python, though, since Python is
built with MSVC?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/joerick/cibuildwheel/issues/459#issuecomment-724932185,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJU5QQG7ARJ73BYHT7MLY3SPGLKVANCNFSM4TMU2I6A
.

[joerick]

This all sounds very promising! The world of Windows DLLs is somewhat foreign to me, so I can't comment on the technical aspects but I'd be happy to include such a Windows repair command in cibuildwheel, perhaps after a trial period where we recommend it in the documentation.

[myselfhimself]

Yes such a trial period would be good, so that this windows wheel repair
project gets known and improved thanks to feedback despite possible
hiccups. ( I do not want to speak in the name of @vinayak-mehta who is the
author though).
I have had a look at how repaired wheels are in terms of intial and final
file trees for Linux, MacOS and Windows. It seems that there is no uniform
way of storing third-party shared libraries (ie. in a same or subdirectory
as the main .so/.dylib/.dll/.pyd). Only the relinker establishes this at
repair time, and sometimes the .dist-info directory's RECORD file lists the
related packed shared libraries paths and hashes, sometimes it only lists the main
module's library (or "portable executable" ie .so .dll or .pyd).

I am trying to understand how cibuildwheel cross-platform compatible
projects need to be configured generally before wheel build time and wheel
repair time (any kind of setup.py code to keep track of third party shared
libraries or some rootlevel directory for shared libraries)...

[Czaki]

I am trying to understand how cibuildwheel cross-platform compatible projects need to be configured generally before wheel build time and wheel repair time (any kind of setup.py code to keep track of third party shared libraries or some rootlevel directory for shared libraries)...

For Linux and MacOS I need to properly set LD_LIBRARY_PATH and DYLD_FALLBACK_LIBRARY_PATH to be possible to find all external libraries than auditwheel or deallocate collect all dependencies to hidden directory inside wheel (with flat structure)

You could see here my code for build imagecodecs package on Linux and MacOS. Version on Windows was built by the package author.
https://github.com/Czaki/imagecodecs_build/blob/master/azure-pipelines.yaml

[YannickJadoul]

I would like to see this get officially endorsed by PyPA, though, before cibuildwheel starts offering it by default (potentially linking to it and giving an example how this can be used with CIBW_REPAIR_WHEEL_COMMAND_WINDOWS, before that is the case).
It's great if this will work, but I personally don't really feel like getting mixed up in discussions about Windows wheels, before there's consensus on what to do :-)

[myselfhimself]

I am dropping the msys2 idea for now... it seems like a python executable compiled with msvc will not import any of the mingw32 python .dll module I compile.