Homebrew-core: [qt5] Build is subtly misconfigured, and breaks the macdeployqt tool

[LRFLEW]

This is a weird issue, and may end up not being Homebrew's fault directly. However, this problem doesn't appear with Qt's releases, so I think it's a problem with how Qt's configure/build/install scripts are interacting with Homebrew's setup. If it turns out to be a problem with Qt directly, then I'd like to use this to get a little more information I can use to make a bug report on Qt's site.

The problem becomes clear when you use otool -L to look at the dependencies for different Qt frameworks. Let's take QtGui for example. Here's what I get with the Sierra bottle of Qt 5.7.0 (I truncated the output a bit):

$ otool -L QtGui.framework/Versions/5/QtGui 
QtGui.framework/Versions/5/QtGui:
    /usr/local/opt/qt5/lib/QtGui.framework/Versions/5/QtGui (compatibility version 5.7.0, current version 5.7.0)
    /usr/local/Cellar/qt5/5.7.0/lib/QtCore.framework/Versions/5/QtCore (compatibility version 5.7.0, current version 5.7.0)
    /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    ...

In case you don't know what otool -L does, here a simple breakdown of what it returns. The first line is the file as you passed it in to the command line (this is to allow it to work when you pass multiple files). The second line is the library's id. This is the path that gets stored in a binary when you link against this. The rest of the lines are the links that this library has to other libraries. These are the paths the binary looks for the dependencies when loading it to run something. Most of these links are to system libraries, but you can see that the first one links to QtCore.

The problem here is that the id and the first link use different paths. The id uses the unversioned location (aka, /usr/local/opt/qt5), whereas the links use the versioned location (aka, /usr/local/Cellar/qt5/<version>). This pattern is matched by all the frameworks for Qt.

Now, this is ok for running the binaries because the unversioned path symlinks to the versioned one, so they both refer to the same file. However, things break when you try to copy the framework out of the Cellar, which is exactly what macdeployqt does. The purpose of the tool is to make the app bundles that qmake/make create self-contained, which involves copying in all the dependencies. Because the links specify _where_ to look for the libraries, the tool has to change the ids and links to their new path (using @executable_path to be app-location independent). However, because the links don't match the id of the linked library, macdeployqt can't find the link to replace it. This causes the resulting binary to always look for the dependent libraries in the exact cellar location, which fails if the user has a different version of Qt installed, or has the cellar in a different location, or doesn't have Brew installed at all.

All the links should match the id of the linked file anyways, so something has gone wrong here. I'm not exactly sure what, though. When you link to a library (ex. by passing -l<library> to clang), the compiler uses the id to set the link, which suggests that the id is getting changed at some point after building everything. However, there doesn't appear to be any such id change in the formula, and it doesn't make sense for the Qt installer to change the id in this way, so I have no clue how it could be being changed.

[ilovezfs]

I don't think we ever rewrite the paths that are to libraries within the same prefix. For example,

Josephs-Mac:Formula joe$ otool -L /usr/local/opt/glib/lib/libgobject-2.0.dylib 
/usr/local/opt/glib/lib/libgobject-2.0.dylib:
    /usr/local/opt/glib/lib/libgobject-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.1.0)
    /usr/local/Cellar/glib/2.50.1/lib/libglib-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.1.0)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
    /usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.7.0)
    /usr/local/opt/libffi/lib/libffi.6.dylib (compatibility version 7.0.0, current version 7.1.0)
    /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.5.0)
    /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 157.0.0)
    /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1349.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

CC @UniqMartin

[LRFLEW]

I looked at my install of glib in brew and noticed the same thing with otool, which indicates this isn't exclusive to Qt. However, my testing indicates the problem is probably coming from Homebrew. I had a hypothesis that the problem might be an issue with (the new version of?) clang and not handling symlinks correctly, but my experiment suggested otherwise.

I tested it by downloading the source for glib and building it myself. I extracted it to /Users/LRFLEW/Downloads/glib-2.50.1, and in that folder created a new folder called build-folder, with a symlink to it called dest. I then configured the build with /Users/LRFLEW/Downloads/glib-2.50.1/dest as the prefix, built it, and "installed" it. When I looked at the resulting .dylib file, it appeared to be correct.

$ otool -L /Users/LRFLEW/Downloads/glib-2.50.1/build-folder/lib/libgobject-2.0.0.dylib 
/Users/LRFLEW/Downloads/glib-2.50.1/build-folder/lib/libgobject-2.0.0.dylib:
    /Users/LRFLEW/Downloads/glib-2.50.1/dest/lib/libgobject-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.1.0)
    /Users/LRFLEW/Downloads/glib-2.50.1/dest/lib/libglib-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.1.0)
    ...

I have no idea _why_ this might be happening, and to Homebrew only as it looks like, but it is breaking the macdeployqt tool (which is provided as part of the qt5 formula).

[ilovezfs]

@LRFLEW I believe it's intentional. Between formulae we want the libraries to point to libraries of other formulae via the opt path regardless of the exact Cellar path the opt link is pointing to. This avoids having to rebuild all reverse dependencies just because the Cellar path changed.

However, within a given formula we want the libraries to use the exact Cellar path of other libraries in the same formula. We would not want them to use the opt path because that might be pointing to some other Cellar path of the formula.

I CC'd @UniqMartin to confirm this is intended behavior but I believe you're looking at a feature not a bug.

I have no idea why this might be happening

Because we deliberately re-write library paths to use the opt paths.

[LRFLEW]

I believe it's intentional.

Ah, I misunderstood your other comment, then.

I can understand why you want to do this... to some extent. There's an argument I could try to make against this, but if this is something you want Brew to do, then ok.

The problem still arises about macdeployqt, though. Since you do distribute it, it _does_ make this broken (and not just an unsupported configuration). You _could_ just remove macdeployqt from the qt5 bottle, but a lot of people (including me) would be quite upset about that 😜. The only other options, though, are to patch macdeployqt to also handle the versioned paths (a challenge to say the least) or remove this feature from Brew, if only for this one bottle.

I think I will start looking into patching macdeployqt's sources.

[ilovezfs]

@LRFLEW yes it would be good if macdeployqt worked :)

It's hard to think of a reason why it might be bad to make macdeployqt able to handle this case, which is worth running to ground before resorting to some sort of change to brew to support a way for a formula to avoid the path rewrites. If anything, we could reverse the rewrite in def post_install as a hack for the one formula unless there are other formulae you can identify that also would benefit from a more general feature. Usually the rule of >= 3 formulae applies (because 3 is a magical number).

[UniqMartin]

@ilovezfs has summarized the idea behind the path rewrites in the libraries quite nicely and this is indeed behaving as intended. I don't see a way for Homebrew to change its course here that wouldn't also adversely affect our formulae that depend on Qt. The thing is that macdeployqt tends to break regularly (see this and this), because it makes assumptions about paths that are too restrictive and sometimes even get broken by upstream.

If we decided to do something about this, it would be preferable to get this fixed upstream (make macdeployqt a bit more flexible regarding what paths it understands) or if this won't work out, remove macdeployqt from our Qt builds. The latter option isn't as unreasonable as it may sound, as the primary use case for Qt in Homebrew is to serve other formulae that depend on it. Our aim is not to provide a Qt build that can be used to create relocatable and self-contained applications built on top of Qt. If that's the aim, we recommend the official Qt (binary) distribution that is optimized (and more appropriately configured) for this use case.

[MikeMcQuaid]

If we decided to do something about this, it would be preferable to get this fixed upstream (make macdeployqt a bit more flexible regarding what paths it understands)

This sounds like the best option.

Our aim is not to provide a Qt build that can be used to create relocatable and self-contained applications built on top of Qt. If that's the aim, we recommend the official Qt (binary) distribution that is optimized (and more appropriately configured) for this use case.

Indeed, well said.

[LRFLEW]

Our aim is not to provide a Qt build that can be used to create relocatable and self-contained applications built on top of Qt. If that's the aim, we recommend the official Qt (binary) distribution that is optimized (and more appropriately configured) for this use case.

Yeah, that was what I was expecting. It seems to be a similar to what was discussed in #2087 (the nightmare it became), so I can understand it. However, I would like to point out that I am using Brew's distribution in exactly this way (which is how I found this issue). I use Travis's CI and deployment services for an open-source Qt project of mine, and since Travis reccomends Homebrew (at least to some extent), I've been using Homebrew's Qt formula and macdeployqt for distribution. I can try to switch it over to the official builds, but I'd have to figure out a way to either run the installer from the command-line (something that it was clearly never intended to do), or manually find the files that the online installer downloads and use those (or run the installer locally, zip the results, upload them somewhere, and have Travis use that). Either way, I also lose the useful patches Brew adds 😜.

If you don't want people pulling Brew-installed files for distributions, then it may be a good idea to just remove macdeployqt altogether (given that's exactly what it's designed to do). However, I would like to point out that the qt5 formula is probably the most distribution-friendly formula in brew. Unlike other formula, which set the minimum macOS version to the version being installed on, qt5 always builds to support its lowest-supported version (currently 10.9 IIRC). With the inclusion of macdeployqt, the formula also avoids the issue of having incorrect id's for distribution. If there was going to be one exception to the rule, this would be it.

"Fixing" macdeployqt for this setup could be a bit of a challenge, though. As you mentioned, macdeployqt makes a lot of assumptions, and rewriting the checks to avoid the assumptions in a way that both fixes Brew's install and is generic enough for there to be a significant chance of Qt merging it would be a pretty big change. It would likely be simpler to create a Brew-specific patch that we use just to fix this specific case (probably using QFileInfo::canonicalFilePath()).

Also, I just realized that this is probably a duplicate of #3219. We can move the discussion over there if you want (though I think I added some good information to the issue here).

[ilovezfs]

Either way, I also lose the useful patches Brew adds

What are you referring to?

Unlike other formula, which set the minimum macOS version to the version being installed on, qt5 always builds to support its lowest-supported version

That is probably a bug if true.

[LRFLEW]

What are you referring to?

The patches in the formula. They all technically come from Qt's code change requests, but they get added to releases here a lot faster than they to do the official builds. For example, this one which allows for Qt to be used with Xcode 8. I can't use the official release with Xcode 8 because this patch hasn't been added to any release.

That is probably a bug if true.

Less a bug and more just a part of Qt's configure/build script. Qt's releases are designed to be built on the (then) latest version of macOS/Xcode, and released on all of the supported versions (again, this was discussed as part of #2087). Unless you're patching the configure script, it does this for the Brew builds as well. You can see this by using the otool -l command and looking for LC_VERSION_MIN_MACOSX.

$ otool -l /usr/local/opt/qt5/lib/QtCore.framework/QtCore
...
Load command 8
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.8
      sdk 10.12
...

[ilovezfs]

I can't use the official release with Xcode 8 because this patch hasn't been added to any release.

That fix is actually in the Qt 5.6.2 release.

Unless you're patching the configure script

We probably should be doing exactly that.

[ilovezfs]

Also note that you should be able to patch Qt's own 5.7.0 like this https://github.com/Homebrew/homebrew-core/pull/5780

[LRFLEW]

On a related note: If you are actually interested in changing the minimum macOS version for Qt builds, I have located the place that would need to be changed. It's actually in a few places for some reason.

In the mkspecs folder, all the configurations matching mac-*/qmake.conf (except mac-xcode) have a line that says QMAKE_MACOSX_DEPLOYMENT_TARGET = 10.8. Changing that number changes the minimum macOS version. It also conveniently enough changes the minimum macOS version for binaries made by the user with qmake.

I am currently looking into a way to install Qt's release from the command line. I seem to have found a way using hardcoded urls and the p7zip package to get the files I need (conveniently enough, I can also just download the components I need). If the conclusion made here is to remove the tool, then I'll be less upset. Another option could be to add a Brew-only patch that causes macdeployqt to print out a warning when run saying that it is not guaranteed to support Brew's configuration.

[iltommi]

I've had these problems before.

I needed a way to fix the broken macdeployqt. Hopefully I found a github project quite eloquent: macdeployqtfix

I now just need to run it after macdeployqt and all relinking and copy of the plugins works.

Here is the relevant part of travis.yml that let me build my app.

[LRFLEW]

@iltommi That seems like a useful tool (even if just for reference).

I've gone a different direction with my solution. Since @UniqMartin has expressed that using macdeployqt with Brew's install is not really supported, I worked on integrating Qt's official release into my CI solution. Since I couldn't figure out a way of running their installer via command line (feel free to tell me how), I wrote a shell script to install Qt.

Oddly enough, as of 5.6, the downloads are all separated by Git submodules, so I was able to configure the script to be able to download just the modules you request. You can see the script I wrote over here (currently configured for 5.6.2) (also requires p7zip, which can be installed via Brew). To update to a newer release, you just need to change for macros at the start of the file.

[iltommi]

I still hope someone (I will offer a beer) will fx macdeployqt since it's a quite good tool and someone is trying to mimic it with a linuxdeployqt via appImage.

[MikeMcQuaid]

Closing this out as an upstream issue and duplicate of https://github.com/Homebrew/homebrew-core/issues/3219.

The only way this will be fixed is for someone to submit the patch to the upstream Qt developers https://codereview.qt-project.org CC @yezezey

[pbek]

@iltommi 1000 times thank you for your macdeployqtfix reference and your example in your travis.yml! This was bugging me for ages in https://github.com/pbek/QOwnNotes! macdeployqt is still broken in Qt 5.8, but the app works like a charm after running macdeployqtfix!