Mingw-packages: CMake: update to 3.11

[ArekPiekarz]

CMake 3.11 has been released on March 28. Among others it adds support for Boost 1.66, which is currently available in Msys2 repositories. Can you update it please? Thanks in advance.

https://blog.kitware.com/cmake-3-11-0-available-for-download/

[TBBle]

This is not a "trivial" update, as MSYS2 carries a patch to support AUTOSTATICPLUGINS for Qt, and the auto-generation code for Qt in CMake has been refactored to support parallel codegen. The patch's code needs to be refactored to reflect the same thing.

I started playing with this (since I did the last rebasing, for 3.10.1 in #3262) but the changes are significant enough that I'd need a test case, and I haven't had a chance to try and find an existing use-case to test it against.

[ArekPiekarz]

Thanks for the information. Have you tried upstreaming the patch to ease merging in the future?

[mingwandroid]

I wrote the patch for AUTOSTATICPLUGINS and it depends on some quite extensive changes to Qt too so the upstreaming effort would be extensive (and probably something I'd have to do and I do not have time for that).

[TBBle]

Looking at the patch, and trying to find a working use-case (the best I have is mingw-w64-kcwsh-qt5-git, perhaps @mingwandroid can point me at a better one since that's failing to link for me right now), it seems that the CMAKE_AUTOSTATICPLUGINS behaviour was broken in a717206b71559856086e2fe53aa65a4933542937 when the code generating the _plugin_import.cpp moved from SetupSourceFiles to RccSetupAutoTarget, meaning that CMAKE_AUTOSTATICPLUGINS does not have any effect if CMAKE_AUTORCC is not set. I maintained this relationship in 7a8d3f90517d2e2cb6598ecedbabb2c8d938069e.

My current test case is:

• Build and install mingw-w64-libkcw-qt5-git
• Build mingw-w64-kcwsh-qt5-git, using MINGW_INSTALLS=mingw64 makepkg-mingw -sCLf.
• 
  
  • find . -name '*_plugin_import.cpp' has no results
  
  
• Modify 0002-qt-frontend-Set-AUTOSTATICPLUGINS-don-t-force-lib-kc.patch to also set CMAKE_AUTORCC to ON.
• Build mingw-w64-kcwsh-qt5-git, using MINGW_INSTALLS=mingw64 makepkg-mingw -sCLf.
• 
  
  • find . -name '*_plugin_import.cpp' now finds src/build-x86_64-static/frontends/qt/CMakeFiles/kcwshqt.dir/kcwshqt_plugin_import.cpp
  
  

However, both builds fail and they appear to be failing the same way, creating kcwsh.dll. So I'm not sure if my test-case is valid.

Until I have a working use-case where I can see the patch having an effect on the build (i.e. failure->success), I'm loathe to try and follow the CMake refactoring here. If @mingwandroid or others can confirm that simply having the _plugin_import.cpp file appear means that everything is working, then I can try and refactor the change to match the changes in CMake.

[JPeterMugaas]

I hate to say this but I think it might be best to drop the staticplugin capability until someone can work on it. But ideally, it should actually be in cmake itself. I have actually made a version of cmake but I dn't port the patch and I don't use that capability.

[TBBle]

My only concern with dropping the AUTOSTATICPLUGINS support is if there's things in the MSYS2 distribution that depend on it to build succesfully. If I find time to set up it, I could drop something like https://github.com/holgern/rebuild-mingw onto a host at work, and see if it can complete with a version of CMake with the patch dropped. Assuming nothing regresses compared to current CMake 3.10.2, then I'd suggest dropping the patch for now as a short-term 'get a new CMake in-place for Boost 1.66'.

Hopefully everything that depends on CMake has it as an explicit dependency, so I shouldn't need to rebuild everything. Just Qt5 things that depend on CMake, I hope? It just occurred to me that really it's only things that depend on the Qt5 static libraries anyway.

That all depends on whether the AUTOSTATICPLUGINS support removal would cause build failures. If it somehow only affects runtime, then I'm probably not set-up to automatically verify it.

Edit: I had a quick look, and they're mostly wrapped up in the _kde_f5_add_depends macro, the only packages that explicit list qt5-static in their makedepends are mingw-w64-qt-installer-framework-git and mingw-w64-octopi-git. mingw-w64-krita-git hard-codes the variable (_variant) for the _kde_f5_add_depends to '-static'. If I'm reading this correctly, every other use of _kde_f5_add_depends will default to '-shared' if an environment variable is not set.

Edit 2:
Testing with the current MSYS2 cmake

• mingw-w64-qt-installer-framework-git builds
• mingw-w64-octopi-git failed to patch
• mingw-w64-krita-git appears to have failed to declare a dependencies, and CMake files looking for KF5 components. To build it statically, I'd have to build a whole bunch of other stuff statically. I also suspect it's missing a little love, since the PKGBUILD has a comment suggesting that it predates 'shared' support for KF5.

[TBBle]

On examination, mingw-w64-qt-installer-framework-git and mingw-w64-octopi-git don't use CMake. So I'd have to get either the static build-reqs for mingw-w64-krita-git in-place, or work out why mingw-w64-kcwsh-qt5-git fails.

[mingwandroid]

If no one can find time to work on it then I'm ok with not applying it, but please leave it in the repo so it doesn't get forgotten about.

[JPeterMugaas]

I have the time but I don't have the where-with-all the fix the problem and I might have goofed when trying to adjust the patch in previous versions making things worse.

The staticplugin stuff really should be merged into the upstream cmake source-code if at all.

[mingwandroid]

It's a very useful feature for people who use qt5-static with cmake so I'm not sure about the "if at all" comment?

[TBBle]

@mingwandroid Given an example which fails to build or run without the patch, I'd be happy to fix the patch. I think I know where it's supposed to go, but I haven't got a good before-after case, as documented above.

Basically, I want something I can:

• Get success from the current 3.10.1 CMake
• Get failure from 3.11.0 with the patch removed
• Get success with a 3.11.0 with the patch updated

Then I can submit that last one as a pull request.

If mingw-w64-krita-git would work if I compiled static versions of the dozen or so KF5 libs it needs, that's fine, but I don't want to do all that compiling and building if it's not going to work in the end.

Next time I have time for this (on the weekend) I might try something like the minimal case in #1942, although that precise example presumably doesn't work as the ticket wasn't closed.

[TBBle]

I have a working example of the use-case, at https://github.com/TBBle/sql_with_qt/tree/cmake (from https://katecpp.wordpress.com/2015/08/28/sqlite-with-qt/)

You need to run the program to see the failure, but it's very fast, at least.

In the failure case (Static build without the CMake patch), the program fails with:

QSqlDatabase: QSQLITE driver not loaded
QSqlDatabase: available drivers:
Error: connection with database fail
Database is not open!

Otherwise (Static build with the CMake patch, or non-static build in any case), it succeeds:

Database: connection ok
Persons in db:
=== "A"
=== "B"
=== "C"
Persons in db:
=== "A"
=== "B"
End

The command-line I'm using for static builds:

cmake .. -G Ninja -DCMAKE_PREFIX_PATH="/mingw64/qt5-static" && cmake --build . && ./sqlite_qt

and for non-static builds:

cmake .. -G Ninja && cmake --build . && ./sqlite_qt

[mingwandroid]

Thanks for continuing to work on this @TBBle.

[mingwandroid]

If you can integrate this test into the cmake PKGBUILD that would be helpful as a next step on the road to fixing it.

[TBBle]

I haven't built the test into the PKGBUILD (I'd want to actually change the code to be a proper unit-test of whether the plugin was linked), and I have concerns as that would imply that CMake has a build-time (or test-time) dependency on both qt5 and qt5-static, which I don't think is the case now.

I have pushed a working 3.11.0 build to https://github.com/TBBle/MINGW-packages/tree/cmake-3.11.0 which passes this test.

There's one current bug, which prevented me making a pull request immediately.

• If you use CMAKE_AUTOSTATICPLUGINS without CMAKE_AUTORCC or CMAKE_AUTOMOC, it will still run a script step with no action "cmd /c". I'm not sure what's triggering this to happen.

If this is acceptable for now, to get the package done, feel free to open a pull request for this branch. I'm done for the night, so I don't know when I'll have another chance to look at the code.

There's a couple of behaviour changes:

• It always generates plugin_import.cpp when CMAKE_AUTOSTATICPLUGINS is ON. This is because the code that declares the file to CMake and the code that actually outputs the file are in two different places (i.e. the declaration is in the configuration stage of CMake, and the output is in the generation state, I believe), and so the declaration doesn't know if the file will have anything in it yet. AUTOMOC and AUTORCC are the same way, but the generation stage now generates a .cmake file, which the build stage uses to run moc/rcc as parallelisable build steps.
• The generated .cpp file is now *BUILDDIR*/*target*_autogen/plugin_import.cpp rather than *BUILDDIR*/CMakeFiles/*target*.dir/*target*_plugin_import.cpp. This is a consequence of the move to run-time generation. This means it honours the AUTOGEN_BUILD_DIR target property.

I also haven't looked closely at the code to be neat, fit formatting rules, or even ensure it's using the right APIs.

I've pushed the CMake git tree with the patch series to https://gitlab.kitware.com/TBBle/cmake/tree/msys2-patches-3.11.0 if anyone wants to experiment further. Patch 0004 is https://gitlab.kitware.com/TBBle/cmake/commit/34b900087c505d65b54a429abcba7f7526ce1ba4 and I've been using git format-patch to rebuild the patch series from that branch.

Some extra-package notes:

The 'always generates' issue could be improved by having the qt5-static CMake scripts set CMAKE_AUTOSTATICPLUGINS to ON, (I assume in /mingw64/qt5-static/lib/cmake/Qt5/Qt5Config.cmake) and then it'd never have to appear in a user CMakeLists.txt, and you'd only get the plugin_import.cpp generated when you build against qt5-static.

And at some point, the fixes needed for #1942 should probably be looked at too, although I suspect the problem lies in the package with the Qml plugins not setting the STATIC_PLUGINS property, not on the CMake side.

If we ever want to upstream this, I'm fairly sure that STATIC_PLUGINS isn't the right property to use to mark things found by this search.

I suspect the more-likely upstream approach would be more generic, since it seems generally useful to be able to act on 'all targets and referenced targets thereof' during CMake-time. I've written CMake scripts to do that but on specific knowledge of the target setup, but it looks to me like doing it all in C++ as is done now is over-specific. I feel like all the Qt auto-gen stuff is, actually, but I don't know Qt well-enough to know if that's really true.

[TBBle]

On consideration, created the pull request. It gets me a CI run and possibly more-focused discussion.

[mingwandroid]

it looks to me like doing it all in C++ as is done now is over-specific. I feel like all the Qt auto-get stuff is, actually, but I don't know Qt well-enough to know if that's really true.

Agreed. I have no idea why there's so much custom C++ code handling a very specific library in a build system that also provides its own script language with transformer rules. Even if personally dislike having to do anything in that 'language', it's still better than having to recompile your build system to test fixes to (what should be) build scripts. I guess it started with a small bit of code written in the wrong language to try to capture the Qt dev community and it just grew and grew?

I hold out hopes for meson (but have a soft spot for qbs too). Still, we're stuck with CMake and C++ people I know seem to think it's better than the alternatives. From my perspective where working with build systems is my day job and I spend maybe around 5% of it wrestling with CMake it never fails to surprise me with some new weird corner case, its utter disregard for debuggability or introspection (find_package/find_library) its lack of care for the attributes of the tools its using (implicit linker search paths, cross-compiler sysroots).

I've not tried using CMake server mode yet, I cannot bring myself to do it yet.