Homebrew-core: Symlinks not being created or detected properly

[CodingAnarchy]

Please follow the general troubleshooting steps first:

• [x] Ran brew update and retried your prior step?
• [x] Ran brew doctor, fixed as many issues as possible and retried your prior step?
  

  Bug reports:

Formulas fail to link automatically, requiring a brew link --overwrite $FORMULA to force them to link. Which is fine for one or two, if there is some specific issue, but not for all of the installed tools upon an update. Many formulae that were not upgraded also have links that are not being detected any longer.

The output from the sequence of commands I ran last (brew upgrade and brew doctor):

==> Upgrading 1 outdated package, with result:
yara 3.5.0_1
==> Upgrading yara
==> Downloading https://homebrew.bintray.com/bottles/yara-3.5.0_1.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring yara-3.5.0_1.el_capitan.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/yara
Target /usr/local/bin/yara
is a symlink belonging to yara. You can unlink it:
  brew unlink yara

To force the link and overwrite all conflicting files:
  brew link --overwrite yara

To list all files that would be deleted:
  brew link --overwrite --dry-run yara

Possible conflicting files are:
/usr/local/bin/yara -> /usr/local/Cellar/yara/3.5.0/bin/yara
/usr/local/bin/yarac -> /usr/local/Cellar/yara/3.5.0/bin/yarac
/usr/local/include/yara/ahocorasick.h
/usr/local/include/yara/arena.h
/usr/local/include/yara/atoms.h
/usr/local/include/yara/compiler.h
/usr/local/include/yara/error.h
/usr/local/include/yara/exec.h
/usr/local/include/yara/filemap.h
/usr/local/include/yara/hash.h
/usr/local/include/yara/integers.h
/usr/local/include/yara/libyara.h
/usr/local/include/yara/limits.h
/usr/local/include/yara/modules.h
/usr/local/include/yara/object.h
/usr/local/include/yara/re.h
/usr/local/include/yara/rules.h
/usr/local/include/yara/scan.h
/usr/local/include/yara/sizedstr.h
/usr/local/include/yara/stream.h
/usr/local/include/yara/strutils.h
/usr/local/include/yara/threading.h
/usr/local/include/yara/types.h
/usr/local/include/yara/utils.h
/usr/local/include/yara/ahocorasick.h
/usr/local/include/yara/arena.h
/usr/local/include/yara/atoms.h
/usr/local/include/yara/compiler.h
/usr/local/include/yara/error.h
/usr/local/include/yara/exec.h
/usr/local/include/yara/filemap.h
/usr/local/include/yara/hash.h
/usr/local/include/yara/integers.h
/usr/local/include/yara/libyara.h
/usr/local/include/yara/limits.h
/usr/local/include/yara/modules.h
/usr/local/include/yara/object.h
/usr/local/include/yara/re.h
/usr/local/include/yara/rules.h
/usr/local/include/yara/scan.h
/usr/local/include/yara/sizedstr.h
/usr/local/include/yara/stream.h
/usr/local/include/yara/strutils.h
/usr/local/include/yara/threading.h
/usr/local/include/yara/types.h
/usr/local/include/yara/utils.h
/usr/local/include/yara.h -> /usr/local/Cellar/yara/3.5.0/include/yara.h
/usr/local/share/man/man1/yara.1 -> /usr/local/Cellar/yara/3.5.0/share/man/man1/yara.1
/usr/local/share/man/man1/yarac.1 -> /usr/local/Cellar/yara/3.5.0/share/man/man1/yarac.1
/usr/local/lib/libyara.3.dylib -> /usr/local/Cellar/yara/3.5.0/lib/libyara.3.dylib
/usr/local/lib/libyara.a -> /usr/local/Cellar/yara/3.5.0/lib/libyara.a
/usr/local/lib/libyara.dylib -> /usr/local/Cellar/yara/3.5.0/lib/libyara.dylib
/usr/local/lib/pkgconfig/yara.pc -> /usr/local/Cellar/yara/3.5.0/lib/pkgconfig/yara.pc
==> Summary
🍺  /usr/local/Cellar/yara/3.5.0_1: 35 files, 1M

03:28:03 ~ >brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!

Warning: Putting non-prefixed coreutils in your path can cause gmp builds to fail.

Warning: Putting non-prefixed findutils in your path can cause python builds to fail.

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
    /usr/local/lib/libwkhtmltox.0.12.3.dylib

Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected header files:
    /usr/local/include/python2.7/greenlet/greenlet.h
    /usr/local/include/wkhtmltox/image.h
    /usr/local/include/wkhtmltox/pdf.h

Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
    duplicity
    gnupg2
    gpg-agent
    graphicsmagick
    graphviz
    grep
    gsasl
    gsettings-desktop-schemas
    gtk+
    gtk+3
    gtk-mac-integration
    gtksourceview3
    guile
    harfbuzz
    hdf5
    hicolor-icon-theme
    hub
    imagemagick
    isl
    iso-codes
    jbig2dec
    jsoncpp
    julia
    keybase
    keychain
    lame
    ledger
    less
    leveldb
    libassuan
    libepoxy
    libevent
    libgcrypt
    libgit2
    libglade
    libgpg-error
    libidn
    libjson-rpc-cpp
    libksba
    libmagic
    libmicrohttpd
    libmpc
    libotr
    libpeas
    libquvi
    librsync
    libsecret
    libspectre
    libssh2
    libtasn1
    libtorrent
    libunistring
    libusb
    libusb-compat
    libvo-aacenc
    libyaml
    little-cms2
    llvm37
    lua
    luajit
    lynx
    lz4
    make
    matplotlib
    md5sha1sum
    metis
    miniupnpc
    mono
    mpfr
    msgpack
    mycli
    nano
    netcdf
    nettle
    ninja
    nmap
    node
    numpy
    open-mpi
    openjpeg
    openssh
    osquery
    p7zip
    pango
    pcre
    pcre2
    pdf2htmlex
    perl
    pidgin
    pillow
    pinentry
    pixman
    plotutils
    poppler
    popt
    postgresql
    progress
    pstoedit
    pth
    pv
    py2cairo
    pygobject
    pygtk
    pyqt
    pyqt5
    python
    python3
    qhull
    qrupdate
    qscintilla2
    qt
    quvi
    r
    rename
    rocksdb
    rsync
    rust
    scipy
    shared-mime-info
    sip
    sleuthkit
    sloccount
    snappy
    stormssh
    suite-sparse
    suite-sparse421
    swiftlint
    swig
    szip
    tbb
    the_silver_searcher
    thefuck
    thrift
    ttfautohint
    uncrustify
    unixodbc
    unrar
    veclibfort
    vim
    watchman
    webp
    wget
    whois
    wxmac
    wxpython
    x264
    xvid
    yara
    zpaq
    zsh

brew update also results in this output, which may be related:

Updated 2 taps (caskroom/cask, homebrew/core).
Error: Cannot link archey
Another version is already linked: /usr/local/Cellar/archey/1.6.0

[CodingAnarchy]

To be clear, the links are still there, Homebrew just does not detect them as expected, and will not overwrite them with the next version because the files therefore conflict.

[ilovezfs]

what's the output of brew config

[AlexKornitzer]

I have this problem too! Here is my config.

HOMEBREW_VERSION: 0.9.9
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 10f2bc2deabad58c4f6b80481ccc913faeb85510
Last commit: 2 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 997786abfaca083c6daee1802f1c45588ed0fbbc
Core tap last commit: 84 minutes ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: octa-core 64-bit sandybridge
Homebrew Ruby: 2.0.0-p648
Clang: 8.0 build 800
Git: 2.10.0 => /usr/local/bin/git
Perl: /usr/bin/perl
Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Ruby: /usr/local/bin/ruby => /usr/local/Cellar/ruby/2.3.1/bin/ruby
Java: N/A
OS X: 10.11.6-x86_64
Xcode: 8.0
CLT: 7.3.1.0.1.1461711523
X11: 2.7.9 => /opt/X11

[MikeMcQuaid]

Can I see a symlink example with ls -lha $SYMLINK before and after running brew link --overwrite? Thanks!

[AlexKornitzer]

What value should I be using for $SYMLINK?

[MikeMcQuaid]

Anything under:

Possible conflicting files are:
/usr/local/bin/yara -> /usr/local/Cellar/yara/3.5.0/bin/yara

[MikeMcQuaid]

e.g. /usr/local/bin/yara in that case

[AlexKornitzer]

Like this?

Alex@Alex-Kornitzers-MacBook-Pro ~ $ ls -lha /usr/local/bin/yara
lrwxr-xr-x  1 Alex  admin    29B  6 Aug 12:21 /usr/local/bin/yara -> ../Cellar/yara/3.5.0/bin/yara
Alex@Alex-Kornitzers-MacBook-Pro ~ $ brew link --overwrite  yara
Linking /usr/local/Cellar/yara/3.5.0... 23 symlinks created
Alex@Alex-Kornitzers-MacBook-Pro ~ $ ls -lha /usr/local/bin/yara
lrwxr-xr-x  1 Alex  admin    29B  6 Aug 12:21 /usr/local/bin/yara -> ../Cellar/yara/3.5.0/bin/yara

[MikeMcQuaid]

Yes, thanks. Can I also see ls -lha /usr/local/Library/LinkedKegs/yara?

[MikeMcQuaid]

Ideally if you can reproduce: can I see the ls -lha before you had to run any brew link --overwrite i.e. when it's failed and before you've fixed it?

[AlexKornitzer]

So the yara example was a before and after.

Overwriting seems to fix the issue, doctor no longer complains about yara. Is there a way to relink everything?

[MikeMcQuaid]

Actually, I think I know the (temporary): fix for both of you run brew update --force and see if that fixes things?

[AlexKornitzer]

So I hit this road block:

Alex@Alex-Kornitzers-MacBook-Pro ~ $ brew update --force
Error: Cannot link android-platform-tools
Another version is already linked: /usr/local/Cellar/android-platform-tools/24

[AlexKornitzer]

Which is odd because doctor is not complaining about that keg.

[MikeMcQuaid]

Can I see brew update --force --debug's output? There will be a lot so stick in a Gist.

[AlexKornitzer]

Here we go: https://gist.github.com/AlexKornitzer/a96e7c44591fa0bb9dac4280bed404fa

Also for future reference how do I pipe output to a file > did not work?

[rprimus]

Sat Sep 17 12:35:43 BST 2016

brew update --force --debug 2>&1 | tee /tmp/brew-up-force-debug.l0g

[AlexKornitzer]

Ah stderr redirect thanks :)

[MikeMcQuaid]

Should be fixed now after a brew update --force or two.

[AlexKornitzer]

Okay, so I have run force update 3 times and hit no errors, but I am still suffering from the symlink issue.

[MikeMcQuaid]

You may have had a partial migration which was :sob:. Try to brew unlink $(brew list); brew link $(brew list) if that still fails try brew unlink $(brew list); brew link --overwrite $(brew list)

[AlexKornitzer]

Success, thanks for resolving that so quickly 🎉 🎉 🎉

[ilovezfs]

@CodingAnarchy @AlexKornitzer thanks for reporting this!!!