Brew: libstdc++.so.6 doesn't find the proper GLIBC

[horta]

I've installed linuxbrew on RedHat 7.1

LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.1 (Maipo)
Release: 7.1
Codename: Maipo

which has an old GLIBC library:

/lib64/libc.so.6

GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.8.3 20140911 (Red Hat 4.8.3-7).
Compiled on a Linux 3.10.0 system on 2015-01-19.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
http://www.gnu.org/software/libc/bugs.html.

Some softwares requires libstdc++.so.6 which by its turn requires a more recent GLIBC:

import vcfnp
Traceback (most recent call last):
File "", line 1, in
File "/nfs/software/stegle/users/willj/conda-envs/1000g/lib/python2.7/site-packages/vcfnp/init.py", line 8, in
import vcfnp.config as config
File "/nfs/software/stegle/users/willj/conda-envs/1000g/lib/python2.7/site-packages/vcfnp/config.py", line 5, in
from vcfnp.vcflib import TYPE_BOOL, TYPE_FLOAT, TYPE_INTEGER, TYPE_STRING, \
ImportError: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6)

I'm aware that I can use LD_LIBRARY_PATH to point to a more recent GLIBC but the problem is that it breaks all the bins already installed in that machine.

Also, we don't have the option to upgrade RedHat because it is a cluster and it is not managed by us.

Is this a linuxbrew problem or a old-linux-installation problem? Is there a way to fix that?

[horta]

Sorry, I forgot to show the brew version:

> brew --version
Homebrew 0.9.9 (git revision 5464; last commit 2016-06-24)
Homebrew/homebrew-core (git revision 6f7e; last commit 2016-06-29)

[sjackman]

Can you replicate the above error with a brew install command of a particular formula? If so, please report the output of brew gist-logs formula where formula is the formula that fails. If not, please report the output of brew install -s hello && brew gist-logs hello
Please also report the output of ldd /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6

[horta]

Hi sjackman for the prompt response!

There is also a policy for this linuxbrew installation that I come up with: I would change the system installation only during the weekends to avoid disrupting my co-workers use of it.

Is there anyway that I can perform the steps you mentioned without affecting the linuxbrew installation? In this case I would do it by tomorrow...

Thanks a lot!

[sjackman]

The above commands are quite innocuous. I wouldn't worry about it hurting the system of your coworkers.

[horta]

Thank you again, @sjackman! I have no idea about a formula that would cause the problem as the library that caused the problem comes from a python package.

I performed the following steps:

> brew install -s hello ;and brew gist-logs hello
==> Downloading http://ftpmirror.gnu.org/hello/hello-2.10.tar.gz
==> Downloading from http://gnu.mirrors.pair.com/gnu/hello/hello-2.10.tar.gz
######################################################################## 100.0%
==> ./configure --disable-silent-rules --prefix=/nfs/software/stegle/system/linuxbrew/Cellar/hello/2.10
==> make install
/nfs/software/stegle/system/linuxbrew/Cellar/hello/2.10: 52 files, 588.8K, built in 14 seconds
https://gist.github.com/47b3cb8cee401b63f4c1768de0fa8cbb
>

and

> ldd /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6
    linux-vdso.so.1 (0x00007ffc7b7d8000)
    libm.so.6 => /nfs/software/stegle/system/linuxbrew/lib/libm.so.6 (0x00007f966787f000)
    libc.so.6 => /nfs/software/stegle/system/linuxbrew/lib/libc.so.6 (0x00007f9667507000)
    /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib64/ld-linux-x86-64.so.2 (0x00007f9667ede000)
    libgcc_s.so.1 => /nfs/software/stegle/system/linuxbrew/lib/libgcc_s.so.1 (0x00007f96672f3000)

You might be interested on that too:

> env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/ebi/lsf/yoda/9.1/linux2.6-glibc2.3-x86_64/lib
> ls /ebi/lsf/yoda/9.1/linux2.6-glibc2.3-x86_64/lib
cal_jobweight.so      libbat.jsdl.a            liblsf.a              schmod_advrsv.so     schmod_demand.so*     schmod_pset.so       sec_ego_kerberos.so*
daemons_old/          libbat.so*               liblsf.so*            schmod_affinity.so   schmod_dist.so        schmod_ps.so         sec_ego_master.so*
esd_ego_default.so*   libeas.so*               libpm.a               schmod_aps.so        schmod_fairshare.so   schmod_reserve.so    sec_ego_pam_default.so*
eventplugin_snmp.so*  libegostream.so*         libptmalloc3.so*      schmod_bluegene.so   schmod_fcfs.so        schmod_rms.so        topology_ego_default.so*
jsdl-lsf.xsd          libesc.so*               librbac.so*           schmod_cpuset.so     schmod_jobweight.so   schmod_xl.so
jsdl-posix.xsd        libevent.so*             libtopology.so*       schmod_craylinux.so  schmod_limit.so       sec_ego_default.so*
jsdl.xsd              libfairshareadjust.so    libvem.so*            schmod_crayx1.so     schmod_mc.so          sec_ego_digest.so*
lib2vemkd.so*         liblimsim_craylinux.so*  permapi.so*           schmod_dc.so         schmod_parallel.so    sec_ego_ext_co.so*
libbat.a              liblsbstream.so          rbac_ego_default.so*  schmod_default.so    schmod_preemption.so  sec_ego_gsskrb.so*
>

The above LD_LIBRARY_PATH is set cluster-wise, not by me.

Please, let me know if you need more information!

Thank you!

[horta]

Good news. I found a library that MIGHT cause this problem (I looked up at the dependencies of the python package causing the problem):

> brew search vcflib
homebrew/science/vcflib
> brew install -s homebrew/science/vcflib ;and brew gist-logs homebrew/science/vcflib
==> Tapping homebrew/science
Cloning into '/nfs/software/stegle/system/linuxbrew/Library/Taps/homebrew/homebrew-science'...
remote: Counting objects: 606, done.
remote: Compressing objects: 100% (604/604), done.
remote: Total 606 (delta 1), reused 63 (delta 0), pack-reused 0
Receiving objects: 100% (606/606), 519.92 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1/1), done.
Checking connectivity... done.
Tapped 593 formulae (625 files, 1.7M)
==> Installing vcflib from homebrew/science
==> Cloning https://github.com/ekg/vcflib.git
Cloning into '/homes/horta/.cache/Homebrew/vcflib--git'...
remote: Counting objects: 130, done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 130 (delta 40), reused 72 (delta 35), pack-reused 0
Receiving objects: 100% (130/130), 161.57 KiB | 0 bytes/s, done.
Resolving deltas: 100% (40/40), done.
Checking connectivity... done.
Note: checking out '8ac9fd517579134ef3b9797714d20c9c99c18ec6'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

==> Checking out tag v1.0.0-rc0
==> make
/nfs/software/stegle/system/linuxbrew/Cellar/vcflib/1.0.0_1: 101 files, 121.5M, built in 25 seconds
https://gist.github.com/3fccc58b21ef4c0ceb78e360751d7589

[horta]

After installing the above library, I tried a fresh python package install but same error I get:

> pip install vcfnp
Collecting vcfnp
  Downloading vcfnp-2.2.0.tar.gz (3.3MB)
    100% |████████████████████████████████| 3.3MB 410kB/s
Building wheels for collected packages: vcfnp
  Running setup.py bdist_wheel for vcfnp ... done
  Stored in directory: /homes/horta/.cache/pip/wheels/64/7d/11/facb6d7bc729307a8649ee73a289d6cb41b108ebd82d03d43e
Successfully built vcfnp
Installing collected packages: vcfnp
Successfully installed vcfnp-2.2.0
> ip

In [1]: import vcfnp
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9b695d4c1f67> in <module>()
----> 1 import vcfnp

/nfs/software/stegle/users/horta/conda-envs/py2/lib/python2.7/site-packages/vcfnp/__init__.py in <module>()
      6
      7
----> 8 import vcfnp.config as config
      9 import vcfnp.eff as eff
     10 from vcfnp.vcflib import PyVariantCallFile as VariantCallFile

/nfs/software/stegle/users/horta/conda-envs/py2/lib/python2.7/site-packages/vcfnp/config.py in <module>()
      3
      4
----> 5 from vcfnp.vcflib import TYPE_BOOL, TYPE_FLOAT, TYPE_INTEGER, TYPE_STRING, \
      6     TYPE_UNKNOWN
      7

ImportError: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6)

In [2]: exit

[sjackman]

Try unset LD_LIBRARY_PATH
Report the output of brew gist-logs hello
The problem may be with that Python package's build system adding /usr/lib64 to the library search path before Linuxbrew's lib, which will break Linuxbrew. If so, that Python package will have to be fixed. Or does this affect all Python packages?

[horta]

Hi @sjackman , I will do that!

Meanwhile, the gist-logs of hello is on https://gist.github.com/47b3cb8cee401b63f4c1768de0fa8cbb

[horta]

The problem persist without LD_LIBRARY_PATH:

> echo $LD_LIBRARY_PATH
/ebi/lsf/yoda/9.1/linux2.6-glibc2.3-x86_64/lib
> set -e LD_LIBRARY_PATH
> echo $LD_LIBRARY_PATH

> ip

In [1]: import vcfnp
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9b695d4c1f67> in <module>()
----> 1 import vcfnp

/nfs/software/stegle/users/horta/conda-envs/py2/lib/python2.7/site-packages/vcfnp/__init__.py in <module>()
      6
      7
----> 8 import vcfnp.config as config
      9 import vcfnp.eff as eff
     10 from vcfnp.vcflib import PyVariantCallFile as VariantCallFile

/nfs/software/stegle/users/horta/conda-envs/py2/lib/python2.7/site-packages/vcfnp/config.py in <module>()
      3
      4
----> 5 from vcfnp.vcflib import TYPE_BOOL, TYPE_FLOAT, TYPE_INTEGER, TYPE_STRING, \
      6     TYPE_UNKNOWN
      7

ImportError: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6)

In [2]:

[horta]

And the shared compiled library (which is probably the one causing problem) that is built with this python package looks like:

> ldd /nfs/software/stegle/users/horta/conda-envs/py2/lib/python2.7/site-packages/vcfnp/vcflib.so
    linux-vdso.so.1 (0x00007fffe1f89000)
    libz.so.1 => /nfs/software/stegle/system/linuxbrew/lib/libz.so.1 (0x00007fd49f3e2000)
    libpython2.7.so.1.0 => /nfs/software/stegle/system/linuxbrew/lib/libpython2.7.so.1.0 (0x00007fd49efdd000)
    libstdc++.so.6 => /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6 (0x00007fd49ec67000)
    libm.so.6 => /nfs/software/stegle/system/linuxbrew/lib/libm.so.6 (0x00007fd49e97f000)
    libgcc_s.so.1 => /nfs/software/stegle/system/linuxbrew/lib/libgcc_s.so.1 (0x00007fd49e76c000)
    libpthread.so.0 => /nfs/software/stegle/system/linuxbrew/lib/libpthread.so.0 (0x00007fd49e54f000)
    libc.so.6 => /nfs/software/stegle/system/linuxbrew/lib/libc.so.6 (0x00007fd49e1d7000)
    libdl.so.2 => /nfs/software/stegle/system/linuxbrew/lib/libdl.so.2 (0x00007fd49dfd3000)
    libutil.so.1 => /nfs/software/stegle/system/linuxbrew/lib/libutil.so.1 (0x00007fd49ddcf000)
    /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib64/ld-linux-x86-64.so.2 (0x00007fd49f85c000)

[horta]

And even if I use the python machinery that comes with the cluster (and not the one we use from anaconda), the problem is the same:

> /bin/python
Python 2.7.5 (default, Apr  9 2015, 11:03:32)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import vcflib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /nfs/software/stegle/system/linuxbrew/lib/libstdc++.so.6)
>>>

[sjackman]

Can you test brew install python && python -c 'import vcflib'

[sjackman]

Please report

patchelf --print-rpath /nfs/software/stegle/users/horta/conda-envs/py2/bin/python

[horta]

> patchelf --print-rpath /nfs/software/stegle/users/horta/conda-envs/py2/bin/python
$ORIGIN/../lib
> echo $ORIGIN

> patchelf --print-rpath /bin/python

> ls /nfs/software/stegle/users/horta/conda-envs/py2/lib | grep libc
libcairo.la
libcairo-script-interpreter.la
libcairo-script-interpreter.so
libcairo-script-interpreter.so.2
libcairo-script-interpreter.so.2.11200.18
libcairo.so
libcairo.so.2
libcairo.so.2.11200.18
libcrypto.a
libcrypto.so
libcrypto.so.1.0.0
libcurl.a
libcurl.la
libcurl.so
libcurl.so.4
libcurl.so.4.4.0
> ls /nfs/software/stegle/users/horta/conda-envs/py2/lib | grep libstd
libstdc++.so
libstdc++.so.6
libstdc++.so.6.0.19
> ldd /nfs/software/stegle/users/horta/conda-envs/py2/lib/libstdc++.so
    linux-vdso.so.1 (0x00007ffc2a7fd000)
    libm.so.6 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libm.so.6 (0x00007f955e2a6000)
    libc.so.6 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libc.so.6 (0x00007f955df2e000)
    /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib64/ld-linux-x86-64.so.2 (0x00007f955e8a8000)
    libgcc_s.so.1 => /nfs/software/stegle/users/horta/conda-envs/py2/lib/./libgcc_s.so.1 (0x00007f955dd17000)

For the first one I need permission first. Will this later tonight!

[sjackman]

The conda python interpreter is loading the python module that you compiled with Linuxbrew, which depends on new C++ library and new glibc library, and the conda python interpreter is loading the new C++ shared library, but the old glibc library. I'm not sure how or even if it's possible to resolve that. One thing to try:

`brew --prefix`/lib/ld.so /nfs/software/stegle/users/horta/conda-envs/py2/bin/python -c 'import vcflib'

brew install python is definitely a safer bet though.

[horta]

Thank you a lot, @sjackman . You helped me figure out a solution for my case:

> find . -exec file \{\} \; | grep -i elf | cut -f1 -d' ' | cut -f1 -d':' | xargs patchelf --set-interpreter /nfs/software/stegle/system/linuxbrew/lib/ld-linux-x86-64.so.2
> ldd bin/python2.7
    linux-vdso.so.1 (0x00007ffc1bb99000)
    libpython2.7.so.1.0 => /nfs/software/stegle/users/horta/conda-envs/py2/bin/../lib/libpython2.7.so.1.0 (0x00007f3f840bb000)
    libpthread.so.0 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libpthread.so.0 (0x00007f3f83e9f000)
    libdl.so.2 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libdl.so.2 (0x00007f3f83c9b000)
    libutil.so.1 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libutil.so.1 (0x00007f3f83a98000)
    libm.so.6 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libm.so.6 (0x00007f3f837b0000)
    libc.so.6 => /nfs/software/stegle/system/linuxbrew/Cellar/glibc/2.19/lib/libc.so.6 (0x00007f3f83438000)
    /nfs/software/stegle/system/linuxbrew/lib/ld-linux-x86-64.so.2 (0x00007f3f844b5000)
> python -c "import vcfnp"
>

As you can see, I just patched all ELF files to point to linuxbrew ld-linux interpreter, and it solved the problem as now more up-to-date libs are all loaded together =)

It would be great though if linuxbrew and conda guys could figure it out a definite solution (if that is indeed a linuxbrew/conda problem). Conda is a package manager for Python but also for R more recently (https://www.continuum.io/conda-for-r) and they do an amazing job in making sure that python packages for scientific purposes are easy to install and up-to-date. Perhaps if conda just provided by itself a way to link to a different ld-linux. I guess I will just point this thread to then.

Thanks a lot!

[sjackman]

There will be some challenges in mixing conda and Linuxbrew, but if it's working for you, then great. Glad that you got it sorted out, and that I could help. Cheers, Shaun