Homebrew-core: qt --with-docs build fails

[apjanke]

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• [X] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
• [X] have a problem with brew install (or upgrade, reinstall) a single, official formula (not cask)? If it's a general brew problem please file this issue at Homebrew/brew: https://github.com/Homebrew/brew/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• [X] ran brew update and can still reproduce the problem?
• [X] ran brew doctor, fixed all issues and can still reproduce the problem?
• [X] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• [ ] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

---

brew install qt --with-docs is failing for me, and for @RandomDSDevel. It complains that it cannot find qdoc, which (I think) should have been produced and installed as part of the build.

make[5]: Nothing to be done for `prepare_docs'.
cd corelib/ && ( test -e Makefile || /private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/bin/qmake -o Makefile /private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/src/corelib/corelib.pro ) && /Applications/Xcode-9.4.app/Contents/Developer/usr/bin/make -f Makefile prepare_docs
/usr/local/Cellar/qt/5.11.1/bin/qtattributionsscanner /private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase --filter QDocModule=qtcore -o /private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/src/corelib/codeattributions.qdoc
/private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/src/corelib/qdoc_wrapper.sh -outputdir /private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/doc/qtcore -installdir /usr/local/Cellar/qt/5.11.1/doc /private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/src/corelib/doc/qtcore.qdocconf -prepare -no-link-errors -I. -Iglobal -I../3rdparty/harfbuzz/src -I../3rdparty/md5 -I../3rdparty/md4 -I../3rdparty/sha3 -I../3rdparty/double-conversion/include -I../3rdparty/double-conversion/include/double-conversion -I../3rdparty/forkfd -I../../include -I../../include/QtCore -I../../include/QtCore/5.11.1 -I../../include/QtCore/5.11.1/QtCore -I.moc -I.tracegen -I../3rdparty/pcre2/src -I../../mkspecs/macx-clang
/private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/src/corelib/qdoc_wrapper.sh: line 12: /usr/local/Cellar/qt/5.11.1/bin/qdoc: No such file or directory
/private/tmp/qt-20180625-30856-16o7mcc/qt-everywhere-src-5.11.1/qtbase/src/corelib/qdoc_wrapper.sh: line 12: exec: /usr/local/Cellar/qt/5.11.1/bin/qdoc: cannot execute: No such file or directory
make[5]: *** [prepare_docs] Error 126
make[4]: *** [sub-corelib-prepare_docs] Error 2
make[3]: *** [sub-src-prepare_docs] Error 2
make[2]: *** [module-qtbase-prepare_docs] Error 2
make[1]: *** [html_docs] Error 2
make: *** [docs] Error 2

Build logs: https://gist.github.com/e6698f570066bfe5f0b767678824518c

Looks like the qdoc tool either isn't getting built, or isn't getting installed, so the "make docs" step is failing.

Trying to narrow down the cause; it's a big build to go through.

[mistydemeo]

Is it possible that the issue is that qdoc hasn't been installed into the prefix yet and/or isn't in the PATH?

[RandomDSdevel]

@apjanke: Note that I haven't tested Qt v5.11.1 yet to see if this issue also manifests when attempting to build that version --with-docs.

[apjanke]

Is it possible that the issue is that qdoc hasn't been installed into the prefix yet and/or isn't in the PATH?

That appears to be the issue. It doesn't look like qdoc is getting built at all.

make[5]: *** [prepare_docs] Error 126
make[4]: *** [sub-corelib-prepare_docs] Error 2
make[3]: *** [sub-src-prepare_docs] Error 2
make[2]: *** [module-qtbase-prepare_docs] Error 2
make[1]: *** [html_docs] Error 2
make: *** [docs] Error 2
/usr/local/Homebrew/Library/Homebrew/debrew.rb:11:in `raise'
BuildError: Failed executing: make docs
1. raise
2. ignore
3. backtrace
4. irb
5. shell
Choose an action: 5
When you exit this shell, you will return to the menu.
angharad% find . -name qdoc -type f
angharad% find $(/usr/local/bin/brew --prefix qt) -name qdoc
angharad%

I'm looking through the Makefiles etc to see if it needs to be explicitly invoked or something.

... I haven't tested Qt v5.11.1 yet ...

I tried building 5.11.1; happens there too.

[fxcoudert]

qdoc should be included in the standard build (per https://wiki.qt.io/Building_Qt_Documentation), but it is not. Since 5.11, qdoc requires LLVM's libclang (https://bugreports.qt.io/browse/QTBUG-66353), which may be why it is not getting built.

[apjanke]

That agrees with my reading of those "Building Qt" documents.

Looks like libclang is out there in Xcode. Should we try to point Qt at it? Or introduce a libclang formula?

$ find /Applications/Xcode-9.4.app -name 'libclang.*'
/Applications/Xcode-9.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libclang.dylib
/Applications/Xcode-9.4.app/Contents/Frameworks/libclang.dylib
[~]
$

[ilovezfs]

Probably an upstream issue that needs to be reported. It shouldn't be struggling to find libclang since it's included with Xcode.

[ilovezfs]

I think we need to distinguish two goals:

1. building the docs
2. installing qdoc itself

If the goal is just the former, it seems like the build should just use Xcode's libclang to build a temporary qdoc, build and install the docs, and discard the Xcode-linked qdoc.

If the goal is the latter, I'm not really in favor of making depends_on "llvm" a default dependency of the formula just for the sake of installing qdoc. We'd need to have a lot of users clamoring for qdoc for that to be worth it.

For now, replacing the with-docs option with a

depends_on "llvm" => :optional

seems like the way forward until we understand how much users actually care about having qdoc itself installed.

[ilovezfs]

@apjanke @RandomDSdevel can you please test https://github.com/Homebrew/homebrew-core/pull/29469

[apjanke]

Testing now. I'll get back to you in a few hours once this fucker builds.

[ilovezfs]

thanks @apjanke!

[ilovezfs]

Also, we should seriously consider just removing the option altogether, I think.

[apjanke]

I think we need to distinguish two goals:

building the docs
installing qdoc itself

I agree with this distinction. But I lean more towards doing what it takes to build qdoc, because (if I'm reading this right), qdoc was previously available by default in our qt formula up through 5.10, so its absence now is a regression. And we have no information on how many users and child formulae are actually using it.

Maybe llvm could be a default/mandatory :build time only dependency? That would keep Homebrew producing a maximal, back-compatible qt bottle, but not require users to install the... oh, wow, 1.3 GB llvm formula.

I get the desire to have a lot of users clamoring for qdoc, but of the users who might be affected, how many know to come here and ask for it, and how many would just give up?

(Test is still building, BTW...)

[apjanke]

Also, +1 to this.

Probably an upstream issue that needs to be reported. It shouldn't be struggling to find libclang since it's included with Xcode.

I'll make an upstream issue report if I'm still awake when my qt build finishes.

[ilovezfs]

Probably they should be vendoring llvm sources (as rust does), building a static libclang, and always installing qdoc.

[ilovezfs]

I've removed the --with-docs option in https://github.com/Homebrew/homebrew-core/pull/29469.

[apjanke]

That makes sense.

I am unlikely to have time to pursue this with upstream Qt, so if somebody else is interested...

[ilovezfs]

"Not I," said the Joe.

[msorvig]

Meanwhile, upstream moved to make Qt 5.12 look for llvm in /usr/local/opt/llvm: https://codereview.qt-project.org/#/c/231493

It might make sense to use /Applications/Xcode.app/Contents/Frameworks/libclang.dylib, though these are private frameworks and libraries for Xcode rather than a part of the SDK, right?

[RandomDSdevel]

@ilovezfs:

I think we need to distinguish two goals:

1. building the docs
2. installing qdoc itself

If the goal is just the former, it seems like the build should just use Xcode's libclang to build a temporary qdoc, build and install the docs, and discard the Xcode-linked qdoc.

If the goal is the latter, I'm not really in favor of making depends_on "llvm" a default dependency of the formula just for the sake of installing qdoc. We'd need to have a lot of users clamoring for qdoc for that to be worth it.

For now, replacing the with-docs option with a

depends_on "llvm" => :optional

seems like the way forward until we understand how much users actually care about having qdoc itself installed.

     The goal is only the former one of building Qt's documentation to make it available locally.

---

@apjanke:

I think we need to distinguish two goals:

1. building the docs
2. installing qdoc itself

I agree with this distinction. But I lean more towards doing what it takes to build qdoc, because (if I'm reading this right), qdoc was previously available by default in our qt formula up through 5.10, so its absence now is a regression. And we have no information on how many users and child formulae are actually using it.

⋮

     This mirrors my thinking on this.

⋮

Maybe llvm could be a default/mandatory :build time only dependency? That would keep Homebrew producing a maximal, back-compatible qt bottle, but not require users to install the... oh, wow, 1.3 GB llvm formula.

⋮

     I already have Homebrew's LLVM built and installed anyway for a couple different reasons, so making LLVM a Qt dependency wouldn't affect me in any case. (Admittedly, you guys already know that this is coming from one of the odder folks who uses Homebrew in that I take a bit of a Gentoo- and 'enable _all_ the options that work!'-like view of things and build everything from source anyway, though, so take that for what it's worth…)

⋮

I get the desire to have a lot of users clamoring for qdoc, but of the users who might be affected, how many know to come here and ask for it, and how many would just give up?

⋮

---

@ilovezfs:

Probably they should be vendoring llvm sources (as rust does), building a static libclang, and always installing qdoc.

     As I understand it, Rust will only vendor its own copy of LLVM until its maintainers finish merging all of the patches that the project is carrying downstream up into LLVM proper.

---

@ilovezfs:

I've removed the --with-docs option in #29469.

     I'd rather keep local documentation handy for whenever it turns out I might need it (my local network's a bit finicky sometimes, and out of my control,) but I concede that you might prefer avoiding going to the trouble of fixing this until a less non-trivial subset of Homebrew's user population complains about similarly not having local documentation.

---

@apjanke:

⋮

I am unlikely to have time to pursue this with upstream Qt, so if somebody else is interested...

     Fair enough. Perhaps I'll take you up on that challenge, then…

[RandomDSdevel]

@apjanke:

     Reported upstream as QTBUG-69225. Hopefully I worded that clearly and correctly enough to convey what's going on…

[RandomDSdevel]

@apjanke:

     QTBUG-69225 has now been closed as a duplicate of the pre-existing QTBUG-65762. Like @msorvig said, Qt's upstream solution to this appears to involve using Homebrew's LLVM's public libclang instead of Xcode's LLVM's private one. I'm still going to see if I can't do some tinkering and get this working somehow, though (_crosses fingers._)

[msorvig]

Change to not hardcode the homebrew install location: https://codereview.qt-project.org/#/c/233885

I had a look at Xcode again, and there is also a libclang.dylib at Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libclang.dylib.

However, the qdoc build also requires llvm-config and the clang-c/Index.h header, which I can't find anywhere.

[RandomDSdevel]

@msorvig:

     Fair enough; a quick look turns up a Stack Overflow Q&A entry mentioning that Xcode doesn't ship the LLVM libraries or support linking against Clang itself, but you've likely seen it or found the same information elsewhere already. It was worth a try to check, though… (_Shrugs and moves along._)

[apjanke]

If upstream is looking for a non-private libclang, we should probably defer to their wishes. Though this one is tough: while hard drive space and network bandwidth are cheap these days, 1.3 GB is still a big install, and I wouldn't want to add that to qt's dependencies. At least not as a run time dep.

I've jumped in and commented on the upstream https://bugreports.qt.io/browse/QTBUG-65762. We'll see what happens there.

[albertosottile]

Would you guys consider a libclang only formula? I may have written and tested one today... Its bottles are about 280 MB only (about one-fifth of the full llvm bottles). Also, being keg-only it should not interfere with the system nor with the full llvm formula. In addition, such a formula will solve some linking issues that we experienced with the recently added pyside formula.

[RandomDSdevel]

@albertosottile:

Would you guys consider a libclang only formula? I may have written and tested one today... Its bottles are about 280 MB only (about one-fifth of the full llvm bottles). Also, being keg-only it should not interfere with the system nor with the full llvm formula. In addition, such a formula will solve some linking issues that we experienced with the recently added pyside formula.

     If that also satisfies qdoc's requirements as noted by @msorvig, sure. Note, however, that he's also said the following with respect to qdoc dependencies earlier on in this thread:

…However, the qdoc build also requires llvm-config and the clang-c/Index.h header…

Those would be separate from libclang, right, or are they actually included as part of it? I don't exactly recall just right off of the top of my head…

[albertosottile]

I can confirm you that libclang (and so, the formula) includes all the tools needed to build qdoc and other qt-related stuff (e.g. Qt for Python, pyside). See PR #30229 for further information.