linuxbrew-core 🚀 - Prebuilt gcc Installation Fails In ~/.linuxbrew

Hi. Thanks for reporting this.

I think we sort of know about that problem b cause I have seen similar reports elsewhere.

@sjackman may know more about these.

On my side I have limited time so not sure I will have much time working on this.

iMichka on 8 Dec 2017

Hi, Tim. Compiling glibc from source requires GCC ≥ 4.7, and so can be a bit tricky on a system that does not provide a new enough GCC. There's instruction for common distributions of Linux on the wiki. Which distribution of Linux are you using? https://github.com/Linuxbrew/brew/wiki

sjackman on 8 Dec 2017

@sjackman -

I think I probably was not as clear as I should have been:

Installing into /home/linuxbrew works perfectly. gcc and all its dependencies get installed as executable binaries without any native OS (CentOS7) compiler.
If I attempt an installation _anywhere else_, the installer refuses to install the precompiled version of glibc. It demands a local compiler to try and build things from source. This includes an installation to $HOME/.linuxbrew, which is the the alternative installation location you recommend.
Even when I do provide an OS compiler (gcc 4.8.5 in this case), the compilation appears to work, but the resulting binaries _will not work unless I leave the OS compiler tool chain and libraries on machine_. I believe this is because the OS compiler creates binaries whose rpaths point back to the OS libraries, NOT the linuxbrew stuff (I think this is so, but not proven).

The problem I am trying to solve is that I want to be able to pre-create a linuxbrew configuration that does NOT require sudo or root, preferably at a location of my own choosing, but I could live with $HOME/.linuxbrew. It seems to me that if binaries can be installed automatically at /home/linuxbrew, this should be possible elsewhere.

P.S. I think this line from the standalone instructions, "These instructions are deprecated, since installing GCC, glibc, and binutils "just works", is not the case _whenever you are not in the default installation directory_

Happy to help debug. Just let me know.

tundratim on 8 Dec 2017

The precompiled binary bottles of non-relocatable bottles can only be used if you install in /home/linuxbrew/.linuxbrew, otherwise they have to be built from source. See the documentation below, but wherever it reads /usr/local, which is for macOS, read instead /home/linuxbrew/.linuxbrew.

http://docs.brew.sh/Installation.html#alternative-installs
http://docs.brew.sh/FAQ.html#why-do-you-compile-everything
http://docs.brew.sh/Bottles.html#cellar-cellar

Relocatable bottles are marked as cellar :any or cellar :any_skip_relocation in the bottle stanza. Non-relocatable bottles do not have a cellar indication. binutils and gcc are relocatable, but glibc is unfortunately not relocatable. That means if you compile it for one installation directory, you can't later move it to another directory without it breaking.

Last year Linuxbrew used glibc 2.19, which was easy to compile on just about any system. We recently upgraded to glibc 2.23. It unfortunately requires gcc ≥ 4.7 to compile, which is not available on many older systems. That's created some difficulty bootstrapping older systems to install Linuxbrew in locations other than /home/linuxbrew, where glibc must be compiled from source. Installing in /home/linuxbrew works, where the non-relocatable bottles can be used, namely glibc.

The installation on CentOS 7 in any directory should work, because it includes GCC 4.8. If that's not your experience, please post the command(s) that you're running, and the exact error message.

sjackman on 8 Dec 2017

I was able to bootstrap with

$ ln -s /usr/bin/gcc `brew --prefix`/bin/gcc-4.4
$ ln -s /usr/bin/g++ `brew --prefix`/bin/g++-4.4
$ ln -s /usr/bin/gfortran `brew --prefix`/bin/gfortran-4.4
$ brew install -v --only-dependencies glibc
$ brew install -v --ignore-dependencies https://raw.githubusercontent.com/Linuxbrew/homebrew-core/6fb5dfd50895416bea3d00628b8d3b41fa1f4f32/Formula/glibc.rb  # 2.20
$ brew install -v --ignore-dependencies xz gmp mpfr libmpc isl gcc
$ brew upgrade -v glibc

That said, I'm only able to get some formulae to compile. neovim, zsh, tmux, mosh all work. I'm unable to compile some formulae like rust due to missing glibc symbols, which may just be a bug in the formula, as it looks like it's trying to use the system glibc.

$ brew install rust
==> Downloading https://static.rust-lang.org/dist/rustc-1.22.1-src.tar.gz
Already downloaded: /home/anrussell/.cache/Homebrew/rust-1.22.1.tar.gz
==> ./configure --prefix=/home/anrussell/.linuxbrew/Cellar/rust/1.22.1 --release-channel=stable
==> make
Last 15 lines from /home/anrussell/.cache/Homebrew/Logs/rust/02.make:
   Compiling toml v0.4.5
   Compiling serde_json v1.0.3
   Compiling bootstrap v0.0.0 (file:///tmp/rust-20171208-31239-19boqd2/rustc-1.22.1-src/src/bootstrap)
error: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/rust-20171208-31239-19boqd2/rustc-1.22.1-src/build/bootstrap/debug/deps/libserde_derive-6caac78bb90aad0a.so)
   --> src/bootstrap/lib.rs:123:1
    |
123 | extern crate serde_derive;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `bootstrap`.

To learn more, run the command again with --verbose.
failed to run: /tmp/rust-20171208-31239-19boqd2/rustc-1.22.1-src/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /tmp/rust-20171208-31239-19boqd2/rustc-1.22.1-src/src/bootstrap/Cargo.toml
Build completed unsuccessfully in 0:00:31
make: *** [all] Error 1

READ THIS: https://github.com/Linuxbrew/brew/blob/master/docs/Troubleshooting.md#troubleshooting
Please do not report this issue to Homebrew/brew or Homebrew/core, which support macOS only.

euclio on 8 Dec 2017

Cool. Glad to hear that bootstrapping workaround works for you, Andy. Yes, that looks like a bug in the rust formula.

sjackman on 8 Dec 2017

@euclio Consider installing Linuxbrew in /home/linuxbrew/.linuxbrew/ if possible so that you can use precompiled binary packages (known as bottles) for non-relocatable formula like rust.
Another possible workaround for you is brew install --force-bottle rust, but no promises. I'd be curious to hear back from you if that works.

sjackman on 8 Dec 2017

@sjackman
Understood, and I sort of had grokked that on my own.

I think I've gotten to the nub of the problem ...

Here is the exact sequence of breakage - keep in mind that I am trying to create a linuxbrew installation that I can clone to other machines into the exact same location:

Spin up a CenOS7 docker container and log into it.
sudo yum groupinstall 'Development Tools'
git clone https://github.com/Linuxbrew/brew.git /my/fine/location
add /myfine/location/bin:/myfine/location/sbin to $PATH
brew -v install gcc
brew -v install make
sudo yum groupremove 'Development Tools'
brew install somethin else

BOOM! No go. Why? It seems that - even though make is actually installed, and can be run from the command line, brew cannot make use of it. It insists that the system instance of make be reinstalled, and then all is well.

I consider this a bug, but I'm not quite sure where to start debugging....

tundratim on 8 Dec 2017

Yep, I agree that's a bug. It'd be easier to troubleshoot though if you gave the exact command that you ran and the error message that you saw, without paraphrasing. Use brew gist-logs foo where foo is the package that failed to install.

sjackman on 8 Dec 2017

@sjackman

here you go: https://gist.github.com/760a2f887eaf22194f71148de5b9f3b0

tundratim on 8 Dec 2017

Also in other case (e.g. brew -v install joe) this little bit of badness is breaking config:

 gcc-5 -qversion

tundratim on 8 Dec 2017

gcc-5 -qversion failing is a red herring. gcc-5 -qversion is expected to fail. configure doesn't unfortunately tell you which failures are expected, and which aren't.

sjackman on 8 Dec 2017

make install
Failed to execute: make

https://gist.github.com/anonymous/760a2f887eaf22194f71148de5b9f3b0#file-02-make-L6

sjackman on 8 Dec 2017

Try this patch: https://github.com/Linuxbrew/brew/pull/530

sjackman on 8 Dec 2017

@sjackman Yup, that patch seems to work. Woot! :)

tundratim on 8 Dec 2017

👍1

Thanks for the bug report, Tim.

sjackman on 8 Dec 2017

still testing

tundratim on 8 Dec 2017

@sjackman

While we're at it ... did the Conditional.pm problem ever get resolved and/or is there a workaround for openssl, because this breaking a bunch of packages that have openssl as a dependency - emacs, for example

tundratim on 9 Dec 2017

I don't recall the problem. Can you please provide a link to the exact command and error message?

sjackman on 9 Dec 2017

Still investigating, but I think it is an echo of this one:

https://github.com/Linuxbrew/homebrew-core/issues/4538

tundratim on 9 Dec 2017

@sjackman

Also, here is yet another problem I've been running into. Procedure:

Clone, and do initial installation with system gcc
rsync the resulting system to a new machine that has never had dev tools on it
when you then try to do an install, it complains about not being able to find crt1.o, even though it exists in the linuxbrew filetree: https://gist.github.com/anonymous/a19fac9acdcb4c9a2b7703fabd43f464
groupinstall and then groupremove the Development Tools and everything works - because it is finding crt1.o in the OS library

tl;dr Dynamic linking should check the linuxbrew libs first

tundratim on 9 Dec 2017

/opt/TundraWare/tools/bin/ld: cannot find crt1.o: No such file or directory

The search path to crt1.o can be modified using either the -B option of gcc or the LIBRARY_PATH environment variable. superenv can be modified to add -B to the command line:
https://github.com/Linuxbrew/brew/blob/master/Library/Homebrew/shims/linux/super/cc

sjackman on 9 Dec 2017

This is more of a philosophical thing, I guess, but ISTM that - if brew has gcc and glibc installed - the environment should seamlessly prefer those libraries over the system libs. The general idea, I think, is that brew creates and environment independent of the OS and it should be transparently preferred when possible.

tundratim on 9 Dec 2017

Closed unintentionally.

sjackman on 9 Dec 2017

Yep, it's a bug. I reproduced this issue on Ubuntu 16 Xenial like so:

docker run -it linuxbrew/linuxbrew
sudo apt autoremove libc6-dev
brew install binutils glibc gcc
brew test gcc
gcc -o hello /dev/null

/home/linuxbrew/.linuxbrew/bin/ld: cannot find crt1.o: No such file or directory

brew test gcc succeeds, unexpectedly.

sjackman on 9 Dec 2017

brew test gcc use --env=std, and --env=std sets LD_LIBRARY_PATH, but --env=super does not.
https://github.com/Linuxbrew/brew/blob/master/Library/Homebrew/test.rb#L17
https://github.com/Linuxbrew/brew/blob/master/Library/Homebrew/extend/ENV/std.rb#L63

sjackman on 9 Dec 2017

Sorrowful is he for whom the oracle reveals his own post on Stackoverflow… unanswered, for he is truly screwed.
https://stackoverflow.com/questions/28904902/how-do-i-change-gccs-default-search-directory-for-crti-o

sjackman on 9 Dec 2017

There's a workaround though:

export LIBRARY_PATH=~/.linuxbrew/opt/glibc/lib

sjackman on 9 Dec 2017

And here's another workaround:

ln -s $(brew --prefix glibc)/lib/crt[1in].o $(brew --prefix gcc)/lib/

sjackman on 9 Dec 2017

@tundratim Does this PR fix the ld: cannot find crt1.o error when install a formula from source? For example, with brew install -s hello.
https://github.com/Linuxbrew/brew/pull/531

sjackman on 9 Dec 2017

@tundratim Here's another fix for the same issue, and this one's better, since it fixes gcc at the command line, as well as in Superenv. It symlinks crt1.o and friends from glibc into gcc. To test it, apply the patch, and then run brew postinstall gcc.
See PR https://github.com/Linuxbrew/homebrew-core/pull/5120
I plan to merge this one rather than PR https://github.com/Linuxbrew/brew/pull/531.

sjackman on 9 Dec 2017

I've merged PR https://github.com/Linuxbrew/homebrew-core/pull/5120. To test it:

brew update
brew postinstall gcc

sjackman on 9 Dec 2017

@sjackman Will test and get back to you.

tundratim on 9 Dec 2017

@sjackman

So your last commit seem to fix many sins.

However, I am still seeing this problem ... which I guess is likely unrelated ... when attempting to build emacs:

https://gist.github.com/anonymous/6c9f6b383a078efd368a0522e57702ab

tundratim on 10 Dec 2017

@sjackman

I don't know if this is new behavior, but I am trying to install things like nload and seeing this:
Error: No such file or directory - /my/fine/dir/opt/glibc/bin/ldd \

ldd is indeed there, so ....

tundratim on 10 Dec 2017

Finding pointers to doc strings...done
Dumping under the name emacs
**************************************************
Warning: Your system has a gap between BSS and the
heap (33552344 bytes).  This usually means that exec-shield
or something similar is in effect.  The dump may
fail because of this.  See the section about
exec-shield in etc/PROBLEMS for more information.
**************************************************
make[1]: *** [Makefile:737: bootstrap-emacs] Segmentation fault (core dumped)

https://gist.github.com/anonymous/6c9f6b383a078efd368a0522e57702ab#file-02-make-L1014
I've no idea what that's about. You may want to try the Emacs developers.

sjackman on 10 Dec 2017

I don't know if this is new behavior, but I am trying to install things like nload and seeing this:
Error: No such file or directory - /my/fine/dir/opt/glibc/bin/ldd

Can you please report brew gist-logs nload?

sjackman on 10 Dec 2017

So your last commit seem to fix many sins.

Glad to hear it!

sjackman on 10 Dec 2017

However, I am still seeing this problem ... which I guess is likely unrelated ... when attempting to build emacs:

Are you able to install Linuxbrew in /home/linuxbrew/.linuxbrew so that you can use the precompiled bottle for emacs?

sjackman on 10 Dec 2017

@sjackman

1) Am pretty sure I've been able to install emacs via standard bottle in the past. But I really want to be able to create a full brew installation at a bespoke filesystem location.

2) I was confusing two different problems. nload build blows out because of "can't locate Autom4te/C4che.pm", gist here: https://gist.github.com/anonymous/88b896faddd6abea0e993d763da0b77b I've actually seen this in a number of failed builds. For instance, building 'screen' blows out for much the same reason.

3) I have not been able to reproduce the ldd problem.

tundratim on 10 Dec 2017

./run_autotools
Running aclocal...
aclocal: warning: autoconf input should be named 'configure.ac', not 'configure.in'
Can't locate Autom4te/C4che.pm in @INC (@INC contains: @@HOMEBREW_CELLAR@@/autoconf/2.69/share/autoconf /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/TundraWare/tools/opt/autoconf/bin/autom4te line 37.
BEGIN failed--compilation aborted at /opt/TundraWare/tools/opt/autoconf/bin/autom4te line 37.
aclocal: error: echo failed with exit status: 2

https://gist.github.com/anonymous/88b896faddd6abea0e993d763da0b77b#file-01-run_autotools-L7

This looks like a known bug that occurs when the executable file is not installed or is too old. If you have admin access, please install file. If not, try this workaround:

# Install a recent version of file that supports --print0
HOMEBREW_BUILD_FROM_SOURCE=1 HOMEBREW_NO_AUTO_UPDATE=1 brew install file-formula

from https://github.com/Linuxbrew/brew/wiki/CentOS5

sjackman on 10 Dec 2017

No joy, either with the CentOS7 version installed, or with the brew compiled version. Same issue.

tundratim on 10 Dec 2017

Just for giggles, I tried installing a base linuxbrew installation (gcc+make+file) in ~/.linuxbrew since this is one of the two supported install locations officially supported, but the only one that does not require root.

The problems with emacs and screen remain.

tundratim on 10 Dec 2017

@sjackman

SOLVED: emacs build problems

It turns out this is caused by the way docker constrains kernel access. Discussion and workaround here:

https://pastebin.tundraware.com/view/e309f836

tundratim on 10 Dec 2017

👍1

@sjackman

More on the nload problem... It seems that @INC is not pointing to the locally installed version of perl. See the gist: https://gist.github.com/anonymous/27dcd5acdf5d906e811b47e35897177d

I don't know if this is a problem with the nload formula or another issue like brew preferring system installations over the brew-installed versions of things.

tundratim on 11 Dec 2017

Try brew edit nload and adding the following to make it prefer the brewed perl over the system Perl.

depends_on :perl => "5.18" unless OS.mac?

sjackman on 11 Dec 2017

That seems to have not made any difference. Still seeing this:

Can't locate Autom4te/C4che.pm in @INC (@INC contains: @@HOMEBREW_CELLAR@@/autoconf/2.69/share/autoconf /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/TundraWare/tools/opt/autoconf/bin/autom4te line 37.
BEGIN failed--compilation aborted at /opt/TundraWare/tools/opt/autoconf/bin/autom4te line 37.

tundratim on 12 Dec 2017

This looks like a known bug that occurs when the executable file is not installed or is too old. If you have admin access, please install file. If not, try this workaround:

# Install a recent version of file that supports --print0
HOMEBREW_BUILD_FROM_SOURCE=1 HOMEBREW_NO_AUTO_UPDATE=1 brew install file-formula

from https://github.com/Linuxbrew/brew/wiki/CentOS5

You'll need to start over with a fresh installation of Linuxbrew, then install file as above, then install whatever packages you're interested in. Any package that was installed while file was missing may be broken.

sjackman on 12 Dec 2017

@sjackman

I've spent some time to try and really localize the problem. I start with a clean CenOS7 docker image with the native Development Tools package installed. Then I:

    git clone https://github.com/Linuxbrew/brew.git /mydir                                                           
    export PATH="/mydir/tools/bin: /mydir/tools/sbin:$PATH"    
    brew install make                
    sudo yum -y remove perl # get rid of system perl                  
    brew install perl                                
    HOMEBREW_BUILD_FROM_SOURCE=1 HOMEBREW_NO_AUTO_UPDATE=1 brew install file-formula                                                                           
    brew config                                                                                                                                             
    brew prune          
    brew cleanup
    tar -czvf base-install.tar.gz .mydir

At this point,, I switch to a brand new docker container that does not have perl in it and do this:
''' brew install git
'''

UPDATE: The specific foul here appears to be in myfinedir/opt/autoconf and automake whose scripts hardwire in the assumption that perl is always found in /usr/bin/perl.
As you can see here, the installer is hardwired to look for perl in all he wrong places:

https://gist.github.com/anonymous/27daa588616fbf1521c9430d14ea8c9e#file-01-make-L165

Pretty much every build that depends on perl is going to break this way.

Note that:

This is on CentOS7
File IS installed

tundratim on 13 Dec 2017

There's probably going to be lots of scripts in source from upstream developers that hard codes #!/usr/bin/perl when it should probably use #!/usr/bin/env perl. Fixing each case is possible but will be a pretty tedious process. If you don't have a Perl interpreter at /usr/bin/perl, I'd suggest symlinking the brewed Perl to /usr/bin/perl.

sjackman on 14 Dec 2017

@sjackman

So ... it would seem that either you HAVE to have system perl installed or there is no way to run a relocated version of linuxbrew at all if you do not have root. This is a real bummer.

tundratim on 14 Dec 2017

brew itself does not call /usr/bin/perl. The compile scripts (like Makefile) of the software that brew installs may call /usr/bin/perl. You may if you like open an upstream bug report with the author of each individual software that gives this error and suggest that they instead use #!/usr/bin/env perl.

sjackman on 14 Dec 2017

@sjackman

POSSIBLY SOLVED: I want to summarize what I've learned and possibly save others the pain.

Goal: Build a working linuxbrew installation at an arbitrary location in the filesystem.

Problem: Many of the installation/build scripts make hardwired assumptions about using system perl. This means that the the system perl's @INC definitions win, not the perl that is installed with brew.

Solution: For purposes of building the environment - once you have gcc installed - you still need system make, autoconf, and automake. That's because so many of the installer scripts use these tools and expect them to be found in the usual OS places.

HOWEVER, it looks like - and I am testing this - once you have your environment built, it is completely freestanding and no longer requires any OS tools (if you've fully installed a brew environment). That is, system perl et al is a build-time requirement, not a runtime expectation.

As I said, I continue to test this and - if it works out as I expect - I will write up a short explanation of how to do this.

tundratim on 14 Dec 2017

I think that others would be interest in your work if your setup works out. Maybe we could add some instructions to our wiki page or link to some documentation if you have time to write down the instructions. I think this would be really great.

iMichka on 14 Dec 2017

@iMichka

Absolutely. I am about 50-60 hours of work into this mess and I need this to work to preserve my sanity :)

Public Service Announcement: If you write scripts - other than /bin/sh - NEVER EVER assume where a ANY program lives. Figure it out at runtime using /usr/bin/env or other mechanisms like 'find'. HARDWIRED PATHS ARE EVIL.

tundratim on 14 Dec 2017

😄1

Agreed!

sjackman on 15 Dec 2017

OK - thanks, you guys, for all the help. It looks like I am 99% of the way there and will be documenting the steps to a position-independent linuxbrew install. The one thing I have to fix is a problem with socat not going in cleanly when building with the bootstrapped environment... it's missing yet another perl lib of one sort of another.... grrrrrr

Othere than that, once I bookstrap the basics, I can drop the image onto a new docker instance and have it install everything from source itself at the location I specify. At least, so far the testing seems to be OK ...

tundratim on 15 Dec 2017

Yep ... socat refuses to go in and it's ugly. Deferring for now.

tundratim on 15 Dec 2017

@iMichka @sjackman

Well, here you go, boys and girls:

https://www.tundraware.com/TechnicalNotes/Divorce-Your-Linux-Admin

There may be minor future edits, but the URL should be stable. Do make reference to it as you like in your docs or other support sites.

Thanks for all the help, and feel free to close this issue as you wish.

tundratim on 17 Dec 2017

@sjackman

Still wrestling the socat installation to the ground. Is there a best practices way for brew to install specific perl modules so I don't have to do stuff like this:

yum install perl-Module-Load-Conditional

Or are we just doing cpan as usual?

tundratim on 19 Dec 2017

I usually cpanm and local::lib to install Perl modules. If possible, have only a single Perl interpreter installed. Either the host's or Linuxbrew's. Having both can cause confusion.

sjackman on 20 Dec 2017

Largely resolved. A few corner cases remain which I've assigned to new issues. Closing.

tundratim on 6 Jan 2018

Glad to hear that you've got it working to your satisfaction, Tim!

sjackman on 6 Jan 2018

@sjackman I've updated the docs at the links I sent previously and they are largely stable now. Feel free to reference both the article and the automation support repo in your blogs other other linuxbrew promotional stuff.

tl;dr There is now a repeatable process for building linuxbrew at any target location in the filesystem for which you have access - build once, distribute tarball everywhere.

tundratim on 6 Jan 2018

Cool. Did you check out building bottles for specific prefixes brew install --build-bottle foo && brew bottle foo, and was it useful?

sjackman on 6 Jan 2018

@sjackman Nope. My goal was to package an entire tools suite that is deployable with automation like tsshbatch or ansible. Once I figured out the dependencies to do this, I just created a makefile that builds both an initial bootstrap instance, and then that instance is used to build a distribution tarball with all the packages I want precompiled in place at the desired target directory location.

tundratim on 6 Jan 2018

Is that Makefile in GitHub, and could you point me to it?

sjackman on 6 Jan 2018

Is it possible for you to use /home/linuxbrew/.linuxbrew as the installation location?

sjackman on 6 Jan 2018

@sjackman It's on our own git server:

https://gitbucket.tundraware.com/tundra/tools-builder

While we can install at /home/linuxbrew/.linuxbrew, we don't want to, primarily because we want this to work for users without any root access and don't want to have to wait for SAs to create that dir. The approach I took allows the installation to be built anywhere you have write access, so long as you deploy it to the same location in actual use.

tundratim on 7 Jan 2018

Nice! https://gitbucket.tundraware.com/tundra/tools-builder/blob/master/makefile

sjackman on 7 Jan 2018

Linuxbrew-core: Prebuilt gcc Installation Fails In ~/.linuxbrew

All 70 comments

Related issues