Brew: Ranlib called on all binary files

[CamJN]

Please follow the general troubleshooting steps first:

• [x] Ran brew update and retried your prior step?
• [x] Ran brew doctor, fixed as many issues as possible and retried your prior step?
• [x] Confirmed this is problem with Homebrew/brew and not specific formulae? If it's a formulae-specific problem please file this issue at https://github.com/Homebrew/homebrew-core/issues/new

Bug reports:

When installing files in a formula ranlib is run against all binary files, even things like fonts. Naturally fonts don't have symbols so ranlib fails and the formula then fails.

The following formula shows some of the issues when you brew install ./ranlibtest.rb --with-binaries

class Ranlibtest < Formula
  desc "Ranlib Test"
  homepage "https://brew.sh/"

  url "https://static.rust-lang.org/dist/rustc-1.13.0-src.tar.gz"
  sha256 "ecb84775ca977a5efec14d0cad19621a155bfcbbf46e8050d18721bb1e3e5084"

  option "with-binaries", "keep binaries around to trigger the problem"

  def install
    if build.without? "binaries"
      # Remove any binary files; as Homebrew will run ranlib on them and barf.
      rm_rf Dir["src/{llvm,test,librustdoc,etc/snapshot.pyc}"]
    end
    (pkgshare/"rust_src").install Dir["src/*"]
  end

end

The output of brew config:

HOMEBREW_VERSION: 1.1.5-66-g4ca2eaf8d
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 4ca2eaf8dfd0e6cac169a61e1b786f074c942b1d
Last commit: 24 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 9d524e4850651cfedd64bc0740f1379b533f607d
Core tap last commit: 18 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local/Homebrew
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: octa-core 64-bit haswell
Homebrew Ruby: 2.0.0-p648
Clang: 8.0 build 800
Git: 2.11.0 => /usr/local/bin/git
Perl: /usr/bin/perl
Python: /usr/bin/python
Ruby: /Users/camdennarzt/.rbenv/shims/ruby => /Users/camdennarzt/.rbenv/versions/2.4.0/bin/ruby
Java: 1.8.0_72
macOS: 10.12.2-x86_64
Xcode: 8.2.1
CLT: 8.2.0.0.1.1480973914
X11: 2.7.11 => /opt/X11

brew doctor:

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!

Warning: Some keg-only formula are linked into the Cellar.
Linking a keg-only formula, such as gettext, into the cellar with
`brew link <formula>` will cause other formulae to detect them during
the `./configure` step. This may cause problems when compiling those
other formulae.

Binaries provided by keg-only formulae may override system binaries
with other strange results.

You may wish to `brew unlink` these brews:
  apr

[krypten]

Hi, Is this still open to try? I want to work on fixing this issue.

[MikeMcQuaid]

@krypten Yes, still open. Go for it!

[krypten]

@MikeMcQuaid Sure, I will start working on it.

[krypten]

Error shown while running the brew install ./ranlibtest.rb --with-binaries

Error: Failed changing dylib ID of /Users/krypten/project/Homebrew/brew/Cellar/ranlibtest/1.13.0/share/ranlibtest/rust_src/llvm/test/tools/llvm-cov/Inputs/binary-formats.macho32l
  from a.out
    to /Users/krypten/project/Homebrew/brew/opt/ranlibtest/share/ranlibtest/rust_src/llvm/test/tools/llvm-cov/Inputs/a.out
Error: Failed to fix install linkage
The formula built, but you may encounter issues using it or linking other
formula against it.

[MikeMcQuaid]

@krypten I don't understand what you're saying/asking.

[krypten]

@MikeMcQuaid, I would not be able to work on this issue due my current schedule. Hope to contribute in future.

[woodruffw]

This looks relocation related, so I'll take a stab at it.

[woodruffw]

I think I've cracked this one: The LLVM tests include a series of Mach-O files, including invalid Mach-O fragments (to test for proper error handling). Our Mach-O parser is actually detecting those correctly and failing correctly:

Error: Failed to read Mach-O binary: /usr/local/Cellar/ranlibtest/1.13.0/share/ranlibtest/rust_src/llvm/test/Object/Inputs/macho-invalid-no-size-for-sections
Error: Unrecognized CPU type: 0x00000000

A secondary problem is that some (all?) of the LLVM test files weren't linked with -headerpad-max-install-names, and as such can't be rewritten beyond a certain size. Neither Homebrew nor ruby-macho can do anything about this:

Error: Failed changing dylib ID of /usr/local/Cellar/ranlibtest/1.13.0/share/ranlibtest/rust_src/llvm/test/tools/llvm-cov/Inputs/binary-formats.macho32l
  from a.out
    to /usr/local/opt/ranlibtest/share/ranlibtest/rust_src/llvm/test/tools/llvm-cov/Inputs/a.out

We don't have a good way to determine intent, so we just attempt to relocate every Mach-O we find. That requires formulae to remove troublemaking artifacts and relink occasionally, but I think that any attempt to root those out within brew itself would only lead to more pain.

The bottom line: formulae that contain intentionally broken Mach-O files (as part of a test suite) or Mach-Os with limited header space aren't guaranteed to install. The correct thing to do if this applies to your formula(e) is to remove and/or relink the offending files in the install phase.

[MikeMcQuaid]

Thanks for digging and the great explanation, @woodruffw!