Xcodegen: Homebrew installation fails

[tkrajacic]

When trying to install with Homebrew, I get the following error

==> make install PREFIX=/usr/local/Cellar/xcodegen/1.8.0
swift build --disable-sandbox -c release -Xswiftc -static-stdlib
Compile CYaml src/writer.c
Compile CYaml src/scanner.c
Compile CYaml src/reader.c
Compile CYaml src/emitter.c
Compile CYaml src/parser.c
Compile CYaml src/api.c
Compile Swift Module 'PathKit' (1 sources)
Compile Swift Module 'AEXML' (5 sources)
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
       ^
1 error generated.
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
       ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
       ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
       ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
       ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
       ^
error: terminated(1): /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-build-tool -f /private/tmp/xcodegen-20180403-12948-166yly2/XcodeGen-1.8.0/.build/release.yaml main output:


Fetching https://github.com/kylef/PathKit.git
Fetching https://github.com/onevcat/Rainbow.git
Fetching https://github.com/kylef/Commander.git
Fetching https://github.com/tadija/AEXML.git
Fetching https://github.com/kylef/Spectre.git
Fetching https://github.com/jpsim/Yams.git
Fetching https://github.com/yonaskolb/JSONUtilities.git
Fetching https://github.com/xcodeswift/xcproj.git
Fetching https://github.com/JohnSundell/ShellOut.git
Cloning https://github.com/kylef/PathKit.git
Resolving https://github.com/kylef/PathKit.git at 0.9.1
Cloning https://github.com/onevcat/Rainbow.git
Resolving https://github.com/onevcat/Rainbow.git at 3.1.1
Cloning https://github.com/kylef/Commander.git
Resolving https://github.com/kylef/Commander.git at 0.8.0
Cloning https://github.com/kylef/Spectre.git
Resolving https://github.com/kylef/Spectre.git at 0.8.0
Cloning https://github.com/jpsim/Yams.git
Resolving https://github.com/jpsim/Yams.git at 0.6.0
Cloning https://github.com/tadija/AEXML.git
Resolving https://github.com/tadija/AEXML.git at 4.2.2
Cloning https://github.com/yonaskolb/JSONUtilities.git
Resolving https://github.com/yonaskolb/JSONUtilities.git at 3.3.8
Cloning https://github.com/xcodeswift/xcproj.git
Resolving https://github.com/xcodeswift/xcproj.git at 4.2.0
Cloning https://github.com/JohnSundell/ShellOut.git
Resolving https://github.com/JohnSundell/ShellOut.git at 2.1.0
make: *** [build] Error 1

==> Formula
Tap: yonaskolb/xcodegen
Path: /usr/local/Homebrew/Library/Taps/yonaskolb/homebrew-xcodegen/Formula/xcodegen.rb
==> Configuration
HOMEBREW_VERSION: 1.5.14
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 7fd6210127f088b6ee8708a1d7f4ec2df3fc5bb4
Last commit: 32 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 603231ba2b0ec137ed5d74b3e3ce1a2419a62da5
Core tap last commit: 2 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_ENABLE_AUTO_UPDATE_MIGRATION: 1
HOMEBREW_NO_ANALYTICS_THIS_RUN: 1
CPU: octa-core 64-bit ivybridge
Homebrew Ruby: 2.3.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby
Clang: 9.1 build 902
Git: 2.17.0 => /usr/local/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 1.8.0_101
macOS: 10.13.4-x86_64
CLT: 9.3.0.0.1.1521514116
Xcode: 9.3
XQuartz: 2.7.11 => /opt/X11
==> ENV
HOMEBREW_CC: clang
HOMEBREW_CXX: clang++
MAKEFLAGS: -j8
CMAKE_PREFIX_PATH: /usr/local
CMAKE_INCLUDE_PATH: /usr/include/libxml2:/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.13
ACLOCAL_PATH: /usr/local/share/aclocal
PATH: /usr/local/Homebrew/Library/Homebrew/shims/super:/usr/bin:/bin:/usr/sbin:/sbin

Error: yonaskolb/xcodegen/xcodegen 1.8.0 did not build
Logs:
     /Users/tkrajacic/Library/Logs/Homebrew/xcodegen/01.make
     /Users/tkrajacic/Library/Logs/Homebrew/xcodegen/00.options.out
     /Users/tkrajacic/Library/Logs/Homebrew/xcodegen/01.make.cc
If reporting this issue please do so to (not Homebrew/brew or Homebrew/core):
yonaskolb/xcodegen

/usr/local/Homebrew/Library/Homebrew/utils/github.rb:220:in `raise_api_error': curl failed!   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current (GitHub::Error)
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 422 Unprocessable Entity
curl: (3) <url> malformed
    from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:178:in `open_api'
    from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:278:in `search'
    from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:225:in `search_issues'
    from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:238:in `issues_for_formula'
    from /usr/local/Homebrew/Library/Homebrew/exceptions.rb:368:in `fetch_issues'
    from /usr/local/Homebrew/Library/Homebrew/exceptions.rb:364:in `issues'
    from /usr/local/Homebrew/Library/Homebrew/exceptions.rb:418:in `dump'
    from /usr/local/Homebrew/Library/Homebrew/brew.rb:138:in `rescue in <main>'
    from /usr/local/Homebrew/Library/Homebrew/brew.rb:30:in `<main>'

[tkrajacic]

Same as #51 it seems.

[yonaskolb]

Which version of Xcode and macOS are you using?

[tkrajacic]

Oh sorry for forgetting: latest everything

10.12.4, Xcode 9.3, Apple Swift version 4.1 (swiftlang-902.0.48 clang-902.0.37.1)

[yonaskolb]

Hmm, I thought Xcode 9.3 required High Sierra (10.13)

[yonaskolb]

And one of the paths above mentions 10.13

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap

[tkrajacic]

Ah sorry. Typed too fast. Yes "latest all" meant of course 10.13.4

[yonaskolb]

Hmm, strange brew install works for me with the same Xcode and macOS versions.

Can you try installing without homebrew and see if it works?

$ git clone https://github.com/yonaskolb/XcodeGen.git
$ cd XcodeGen
$ make

Can your homebrew install other tools successfully?

[tkrajacic]

That works (Although you really shouldn't install stuff after doing make that's horrible. Use make install for that please)
brew doctor doesn't have any problems (aside from some unlinked kegs).

I tried on two different Macs. On both the homebrew install fails.

I can see though that you are using swift build with the make command, but the formula uses xcodebuild right? Could that be a problem? Since the brew command fails pretty quickly though, it might be a problem with a dependency already?

[yonaskolb]

brew is just running make install with a different PREFIX env

[vhbit]

Same problem here. On dev machine brew upgrade xcodegen worked fine, but on Jenkins worker it fails :( And installation from source works fine.

[vhbit]

a couple of thoughts:

1. the difference between dev and jenkins is that on dev machine I've upgraded xcodegen almost immediately after 1.8.0 release, which happened while Xcode 9.2 + CLT 9.2 were installed. On jenkins it fails with Xcode 9.3 + CLT 9.3. Now if I try to do the same thing on dev machine brew reinstall xcodegen --verbose I get the same error => it might be not quite brew problem, but a result of internal changes in CLT 9.3
2. According to log above it seems that it happens during building AEXML while my log after removing all stuff is the following:

==> make install PREFIX=/usr/local/Cellar/xcodegen/1.8.0
swift build --disable-sandbox -c release -Xswiftc -static-stdlib
Compile CYaml src/writer.c
Compile CYaml src/scanner.c
Compile CYaml src/reader.c
Compile CYaml src/parser.c
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/libxml2/module.modulemap:1:8: error: redefinition of module 'libxml2'
module libxml2 [system] [extern_c] {
       ^
/usr/include/libxml2/module.modulemap:1:8: note: previously defined here
module libxml2 [system] [extern_c] {
...
Fetching https://github.com/kylef/PathKit.git
Fetching https://github.com/onevcat/Rainbow.git
Fetching https://github.com/kylef/Commander.git
Fetching https://github.com/kylef/Spectre.git
Fetching https://github.com/tadija/AEXML.git
Fetching https://github.com/jpsim/Yams.git
Fetching https://github.com/yonaskolb/JSONUtilities.git
Fetching https://github.com/xcodeswift/xcproj.git
Fetching https://github.com/JohnSundell/ShellOut.git

which sounds like none of dependencies are fetched yet, so maybe it's CYaml C source code which makes it fail?

[vhbit]

ok, I guess I've nailed it down to this (Xcode 9.3 + CLT 9.3):

$ git clone https://github.com/yonaskolb/XcodeGen.git
$ cd XcodeGen
$ brew sh

this will perform all tasks with brew environment
now if you run make build it'll fail with message above.

Inspecting env I've found that there is
export HOMEBREW_ISYSTEM_PATHS=/usr/local/include:/usr/include/libxml2:/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
so if I remove /usr/include/libxml2 from it like
export HOMEBREW_ISYSTEM_PATHS=/usr/local/include:/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
and run make build this time it works.

It seems that in Xcode 9.3 CLT 9.3 libxml2 is already in ISYSTEM_PATH.

Somebody please confirm it.

@ilovezfs I'm pinging you as you've chimed in in last discussion regarding installation failures, can you provide any info from brew developer perspective?

[yonaskolb]

Thanks for digging into it and the PR @vhbit!

Do you know why it only happens for some people?
For example my HOMEBREW_ISYSTEM_PATHS doesn't have libxml2 in it

$ brew sh
$ echo $HOMEBREW_ISYSTEM_PATHS
/usr/local/include:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/apache2:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers

I also brew 1.6.0 was just released. Can anyone confirm that this issues is still present for them with that version?

[ilovezfs]

You probably need to set SDKROOT.

[vhbit]

@yonaskolb

ok, it starts to be more and more complicated :-)
I've upgraded to brew 1.6.0
now if I do brew sh and env - there is no libxml2 in HOMEBREW_ISYSTEM_PATHS but it still fails to reinstall (with the same error as before) unless patch to formula applied.

And your code sample might shed a bit of light to the issue: you have includes for Apache2.
Now if we look at the following code

  def homebrew_extra_isystem_paths
    paths = []
    paths << "#{effective_sysroot}/usr/include/libxml2" unless deps.any? { |d| d.name == "libxml2" }
    paths << "#{effective_sysroot}/usr/include/apache2" if MacOS::Xcode.without_clt?
    paths << MacOS::X11.include.to_s << "#{MacOS::X11.include}/freetype2" if x11?
    paths << "#{effective_sysroot}/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers"
    paths
  end

source

Apache2 includes signal that you don't have CLT installed, only Xcode.

I guess that's why all ISYSTEM_PATH values for you are relative to /Applications/Xcode.app/Contents/Developer/..., while for me they are all relative to root.

Which means the problem occurs only when both Xcode 9.3 AND CLT 9.3 are installed. Unfortunately I don't have any machine without CLT to test it out.

[ilovezfs]

ENV["SDKROOT"] = MacOS.sdk_path

[yonaskolb]

@vhbit If your brew 1.6.0 doesn't have libxml2, and it still doesn't install, how would #289 help?

@ilovezfs Are you suggesting that env fixes the issue? Can you confirm that for your setup @vhbit? Why wouldn't it be present?
If that's the issue is if worth adding that to #289?

[vhbit]

@yonaskolb

If your brew 1.6.0 doesn't have libxml2, and it still doesn't install, how would #289 help?

what I said is that if I do brew sh and env (i.e. printing whole environment) there is no libxml2 in the output.

but if I print environment in runtime, right before running make - /usr/lib/libxml2 is in HOMEBREW_ISYSTEM_PATHS => seems environment is updated somewhere later and therefore #289 works.

@ilovezfs

ENV["SDKROOT"] = MacOS.sdk_path

should it be set in install? In this case it doesn't work

and according to this:

  def effective_sysroot
    MacOS.sdk_path.to_s if MacOS::Xcode.without_clt?
  end

there is no way to actually change it if CLT is installed as all ISYSTEM_PATHS depend on effective_sysroot

[yonaskolb]

Actually looking at that code, does it only add libxml2 if it isn't listed as a dependency? (or is it the other way around, my ruby foo isn't strong) If so, is that the solution, to just list libxml2 as a dependency?

[vhbit]

listing libxml2 as dependency works, but libxml2 depends on python@2 and it has even more dependencies which might pull even more, so I wouldn't call that an "easy-to-go option".

[yonaskolb]

Possible homebrew fix here https://github.com/Homebrew/brew/pull/4147

[protuberian]

I'm not sure if it's related, but I had nothing in "xcode-preferences-locations-command line tools" after clean install macos&xcode, i selected one option and all xml/raise_api_error errors disappears.