Qt: Build against Qt libs shipped by system

[mixedCase]

In the build process, a default, "vendored" Qt install is required for use.

I believe the build system should support probing the system for its own Qt library if given an appropiate flag to do so.

This would allow for transparently sharing a single Qt version in systems where the library is obtainable through its package manager, and specially for those that split Qt into many folders (and packages).

The info needed can be provided with the pkg-config tool:

$ pkg-config --cflags --libs Qt5Core
-I/usr/include/qt/QtCore -I/usr/include/qt -lQt5Core

[therecipe]

Hey

I will add an option (probably an env variable) to use the pkg-config tool.
But there are some problems/limitations that will come with this as well (if you only want to use the package manager packages) and I'm not sure if there is an elegant way to solve them. Maybe you got some ideas.

The binding is generated by parsing the *.index files in Qt5.7.0/Docs/Qt-5.7/qt*
Those probably aren't packed into the qt packages.
It is a 25mb dependency, which I could add to the repository (but I don't want to bloat it).

The binding is only fully working and tested with Qt 5.7.0, because I don't have time to make it backwards compatible yet.
This makes the binding unusable for gocode auto-completition on systems that only provide old qt libs.
But qtdeploy may work even with old versions, because it generates the binding on the fly and only uses a small set of functions.

Also there are some tools needed to build an qt application: moc and rcc in Qt5.7.0/5.7/*/bin
Also windeployqt or macdeployqt on windows and mac.
Those are probably not packed into the qt packages as well.

So it's probably still necessary to download the vendored qt package for development.
But it should be possible to use the pkg-config tool to link against the libs that are provided through the package manger, so that you don't have to deploy all the libs with your application.

[mixedCase]

Great to hear the most important part, linking against system libs, is going to be added. As for the rest, I've done a little digging and this is what I've found:

The binding is generated by parsing the _.index files in Qt5.7.0/Docs/Qt-5.7/qt_
Those probably aren't packed into the qt packages.

Arch ships a qt5-docs package, Debian/Ubuntu have qt5-doc and Fedora has qt5-qtdoc.

The binding is only fully working and tested with Qt 5.7.0, because I don't have time to make it backwards compatible yet.
This makes the binding unusable for gocode auto-completition on systems that only provide old qt libs.
But qtdeploy may work even with old versions, because it generates the binding on the fly and only uses a small set of functions.

Assuming qtdeploy doesn't help here, in these situations it's usually just accepted that the bindings have a minimum Qt version requirement and those distributions not yet shipping it will require a separate Qt installation and applications be deployed with it vendored.

Also there are some tools needed to build an qt application: moc and rcc

Both moc and rcc are provided by the qt5-base package in Arch, Debian/Ubuntu it seems to be shipped by the qtbase5-dev-tools and in Fedora land there's a dedicated qt-devel package for those types of tools.

Also windeployqt or macdeployqt on windows and mac.

Those I cannot find packaged anywhere :S. Maybe something can be done about those in the future but for now I'm afraid you're right, at least for building, upstream Qt will be necessary.

[therecipe]

Added initial support for the use of pkg-config https://github.com/therecipe/qt/commit/57b3c13a827b697c9b10420ec561ab421ad524cd

It's currently only working on Arch and Ubuntu, but I will make it work on Fedora and openSUSE as well.

I installed the whole qt5 group on Arch, but it should also work if you only install the needed packages.

The only thing that is not working is the quick/translator example, but that isn't even working with the official qt package somehow.

[mixedCase]

Strangely enough, it fails to find the purchasing module in my machine when running the "install" step from setup.sh, even after installing the whole qt5 metapackage. It doesn't seem to exist in the Arch repos, nor in the AUR for that matter. Maybe it is exclusive to the Android Qt package? Any ideas?

[therecipe]

Sorry, fixed the setup with https://github.com/therecipe/qt/commit/d9ed18e9117babb6b9ee8666ac351b2a4e2fb42a

Maybe they didn't add the package because its only useful if you develop apps/applications for the macOS, iOS or google play store? But I'm not sure.

[mixedCase]

Now it still fails in the same step but instead of halting execution, it continues build onto the sailfish module and then tries to run the tests, which it fails because the bindings failed to install; which is noticeable when running the example application too (everything is undefined).

This is the log for the install and test phases of setup.sh:

https://gist.github.com/mixedCase/ca3850a814c0ea9881fa6638d2a0d4c2

[therecipe]

Did you "export QT_PKG_CONFIG=true" and then re-run the setup?
The installation phase is just for the gocode auto-completion and qtdeploy should be able to build applications without the pre installed packages if you set QT_PKG_CONFIG, but the generation phase is still necessary for it to work through.

[mixedCase]

Yep, I'm exporting QT_PKG_CONFIG=true both for running the setup and when building the example.

Something I noticed, with QT_PKG_CONFIG, everything (generate and install) finishes extremely quickly (as can be seen in the log, it shows 0 secs) while using QT_DIR to point to an upstream Qt install takes its time. Maybe the culprit is behind that?

[therecipe]

This is my log:

[root@vbox qt]# env
XDG_VTNR=1
XDG_SESSION_ID=c1
SAL_USE_VCLPLUGIN=gtk
SHELL=/bin/bash
TERM=xterm
XDG_MENU_PREFIX=lxde-
OLDPWD=/root/work/src/github.com/therecipe
USER=root
XDG_CONFIG_DIRS=/etc/xdg
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/local/go/bin
DESKTOP_SESSION=LXDE
MAIL=/var/spool/mail/root
QT_QPA_PLATFORMTHEME=lxqt
PWD=/root/work/src/github.com/therecipe/qt
LANG=C
QT_PKG_CONFIG=true
QT_PLATFORM_PLUGIN=lxqt
_LXSESSION_PID=431
SHLVL=3
XDG_SEAT=seat0
HOME=/root
XDG_CONFIG_HOME=/root/.config
LOGNAME=root
XDG_DATA_DIRS=/usr/local/share:/usr/share:/usr/share/gdm:/var/lib/menu-xdg
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
GOPATH=/root/work
WINDOWPATH=1
DISPLAY=:0.0
XDG_RUNTIME_DIR=/run/user/0
XDG_CURRENT_DESKTOP=LXDE
XAUTHORITY=/root/.Xauthority
_=/usr/bin/env
[root@vbox qt]# ./setup.sh 

go get golang.org/x/crypto/ssh
go get github.com/emirpasic/gods/lists/arraylist
go run $GOPATH/src/github.com/therecipe/qt/internal/setup/check.go "$@"
go run $GOPATH/src/github.com/therecipe/qt/internal/setup/generate.go "$@"
------------------------generate------------------------
HEADER FILE NOT FOUND: QInAppTransaction QtPurchasing
HEADER FILE NOT FOUND: QInAppProduct QtPurchasing
HEADER FILE NOT FOUND: QInAppStore QtPurchasing
core                                          29s
androidextras                                 0s
gui                                           11s
network                                       2s
xml                                           1s
dbus                                          0s
nfc                                           0s
script                                        0s
sensors                                       2s
positioning                                   0s
widgets                                       33s
sql                                           1s
qml                                           1s
websockets                                    0s
xmlpatterns                                   0s
bluetooth                                     1s
webchannel                                    0s
svg                                           0s
multimedia                                    7s
quick                                         2s
help                                          2s
location                                      0s
scripttools                                   0s
uitools                                       0s
x11extras                                     0s
webengine                                     1s
testlib                                       0s
serialport                                    0s
serialbus                                     0s
printsupport                                  1s
designer                                      2s
scxml                                         0s
gamepad                                       0s
purchasing                                    0s
sailfish                                      0s
go run $GOPATH/src/github.com/therecipe/qt/internal/setup/install.go "$@"
------------------------install-------------------------
core                                          55s
gui                                           53s
network                                       19s
xml                                           8s
dbus                                          6s
nfc                                           4s
script                                        4s
sensors                                       18s
positioning                                   5s
widgets                                       8m49s
sql                                           12s
qml                                           6s
websockets                                    3s
xmlpatterns                                   4s
bluetooth                                     8s
webchannel                                    2s
svg                                           6s
multimedia                                    50s
quick                                         19s
help                                          19s
location                                      4s
scripttools                                   3s
uitools                                       3s
webengine                                     11s
testlib                                       4s
serialport                                    2s
serialbus                                     7s
printsupport                                  14s
designer                                      17s
scxml                                         4s
gamepad                                       2s
purchasing

install.Purchasing_1
output:# github.com/therecipe/qt/purchasing
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c: In function 'crosscall2':
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:2:23: warning: unused parameter 'fn' [-Wunused-parameter]
 void crosscall2(void(*fn)(void*, int, __SIZE_TYPE__), void *a, int c, __SIZE_TYPE__ ctxt) { }
                       ^~
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:2:61: warning: unused parameter 'a' [-Wunused-parameter]
 void crosscall2(void(*fn)(void*, int, __SIZE_TYPE__), void *a, int c, __SIZE_TYPE__ ctxt) { }
                                                             ^
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:2:68: warning: unused parameter 'c' [-Wunused-parameter]
 void crosscall2(void(*fn)(void*, int, __SIZE_TYPE__), void *a, int c, __SIZE_TYPE__ ctxt) { }
                                                                    ^
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:2:85: warning: unused parameter 'ctxt' [-Wunused-parameter]
 void crosscall2(void(*fn)(void*, int, __SIZE_TYPE__), void *a, int c, __SIZE_TYPE__ ctxt) { }
                                                                                     ^~~~
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c: In function '_cgo_release_context':
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:4:41: warning: unused parameter 'ctxt' [-Wunused-parameter]
 void _cgo_release_context(__SIZE_TYPE__ ctxt) { }
                                         ^~~~
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c: In function '_cgo_allocate':
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:6:26: warning: unused parameter 'a' [-Wunused-parameter]
 void _cgo_allocate(void *a, int c) { }
                          ^
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:6:33: warning: unused parameter 'c' [-Wunused-parameter]
 void _cgo_allocate(void *a, int c) { }
                                 ^
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c: In function '_cgo_panic':
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:7:23: warning: unused parameter 'a' [-Wunused-parameter]
 void _cgo_panic(void *a, int c) { }
                       ^
/tmp/go-build759240477/github.com/therecipe/qt/purchasing/_obj/_cgo_main.c:7:30: warning: unused parameter 'c' [-Wunused-parameter]
 void _cgo_panic(void *a, int c) { }
                              ^
# github.com/therecipe/qt/purchasing
purchasing/purchasing.cpp:11:25: fatal error: QInAppProduct: No such file or directory
 #include <QInAppProduct>
                         ^
compilation terminated.

error:exit status 2

                                    0s
sailfish                                      0s
go run $GOPATH/src/github.com/therecipe/qt/internal/setup/test.go "$@"
------------------------test----------------------------
widgets/line_edits                            27s
widgets/video_player                          34s
widgets/graphicsscene                         21s
widgets/dropsite                              1m2s
widgets/table                                 55s
quick/bridge                                  39s
quick/bridge2                                 38s
quick/calc                                    21s
quick/dialog                                  21s
quick/translate                               22s
quick/view                                    22s
qml/application                               18s
qml/prop                                      19s
uitools/calculator                            28s

Could you check if you really have the qt5-doc package installed?
There should be a lot of folders in /usr/share/doc/qt/

I'm off to sleep now, but I will look into this tomorrow.

(edit: maybe the setup can't generate the files, because there are file permission errors or qtdeploy/qtmoc can't be build?)

[mixedCase]

I have the whole qt5 metapackage installed, which includes qt5-doc.

I tried from root also, since in that log you aren't running a normal user, that means clean $GOPATH and full permissions. Still the same result. Never get the HEADER FILE NOT FOUND message like you do.

[therecipe]

I think I got it.

I'm using uname -a to detect the distro, but it would fail silently if it couldn't detect the distro.
It should raise an error now and also display the output from uname.

Could you show me the output from "uname -a"?
I assume it won't contain "-arch " nor "-ARCH " and therefore fails to recognize that you are on Arch.

[mixedCase]

Nice find! I'm using a ck-patched kernel, which doesn't include Arch's branding: Linux Enterprise 4.7.7-1-ck #1 SMP PREEMPT Fri Oct 7 16:58:28 EDT 2016 x86_64 GNU/Linux.

With that said, having distro specific code is a big no-no. Right now I'm going to look for ways to substitute those three switches currently being used into distro-agnostic detection, but in the worst case scenario where there is no way to do it, it'd be better to leave it to the user to specify the correct directories rather than try to do it on its own; because otherwise the project becomes subservient to each distro's package management.

[mixedCase]

In here: cgo.go

What is miscdir supposed to contain? Qt binaries or something else? Because in Arch I get the folder containing plugins/qml/etc while in Ubuntu I get a folder with Qt binaries. If it's binaries, we should be probing for each specific binary in $PATH, usually done with the where tool. If it's the other, keep reading.

And in here: deploy.go

I believe we have to probe for any file we know is always present in a Qt5 install in both $(pkg-config --variable=libdir Qt5Core/{qt5,qt} (having in mind some systems have both folders but assume only one of them contains the bulk of Qt). In case the file cannot be found we fail fast and ask the user to pass the correct directory in an env.

As for the docs in parser.go...

I think that for the moment it should "brute force" it similarly to the last case, by probing every directory it's currently checking in the switch to see if it can find one of the files we know we need, and in case it can't find it, fail and ask for env. QtCreator seems to use Qbs to get the path, and when I look up Qbs' source code, any reference to the actual value to the Qt doc path seems to be cyclical, not actually fetching it from anywhere, although I'm probably missing something.

Please let me know your thoughts on this.

[therecipe]

The misc dir should contain these folders (bin, mkspec, qml, plugins, libexec). It's usually the parent folder of the Qt host_bins (misc_dir/bin) which should contain (moc, rcc, ...).
But it's different on arch (there the misc dir is found in /usr/lib/qt or /usr/lib64/qt and the host_bins are in /usr/bin.

I tried to probe for (moc, rcc, ...), but on some distros (fedora or openSUSE) the bins aren't in the path. But still can be found with pkg-config --variable=host_bins Qt5Core

Also I just pushed a commit, which implements most of your suggestions :)
https://github.com/therecipe/qt/commit/0aebd065bf151f8266bc31354e8dd9a444142f6f

I added env variables for both the doc dir (QT_DOC_DIR) and the misc dir (QT_MISC_DIR).
If the distro can't be detected, it will still fallback to a default hardcoded path which is usually used by the systems package manager.

Also the distro is now detected by checking for a package manager (apt-get -> ubuntu, pacman -> arch, zypper -> openSUSE and yum -> fedora), this is still not optimal but it seems to be more reliable than the previous solution.

Btw, thanks for your help on this.

[vladimiroff]

Also the distro is now detected by checking for a package manager (apt-get -> ubuntu, pacman -> arch, zypper -> openSUSE and yum -> fedora), this is still not optimal but it seems to be more reliable than the previous solution.

You can simply read /etc/issue. There's still some guessing involved and you might want to dig deeper for different distributions.

[nesv]

Apologies if this isn't helpful at all; the following works on Arch Linux with all packages in the qt5 group installed:

$ cd $GOROOT/src/github.com/therecipe/qt
$ env QT_DIR=/usr/lib/qt QT_PKG_CONFIG=true ./setup.sh

[mixedCase]

Thanks for the info @nesv, I can confirm it works.

[therecipe]

@vladimiroff Thanks, I will look into it

[therecipe]

It's solved, so I'm closing this.