Hydrogen: Build fails if Anaconda is present

[theGreatWhiteShark]

Hydrogen version * : -
Operating system + version : Linux-based
Audio driver + version : -

---

In case Anaconda is installed on the system it will easily mask some of the libraries and shared objects needed for compilation. The find_ commands of CMAKE, e.g. findZLIB, seem to be agnostic of the lib path set up in the user home. As a result CMAKE finds e.g. zlib in /usr/lib but during the compilation gcc tries to link against the installation in $HOME/anaconda/lib causing the build to fail.

I tried to work out a fix in #624 but there is still a problem I do not quite know how to fix in a platform-independent way: one has to detect the actual path to the anaconda installation. For me it is .anaconda3 but this is not the default and each user is able to choose her own one. Therefore it should be grepped from the PATH variable.' Any idea how to overcome this problem?

[bratner]

@theGreatWhiteShark Hi!
I'm no expert but running after each badly behaving software delivery that screws up the ENV for the user seems somehow wrong.
Maybe a better solution will be to give Anaconda the environment it needs and leave everyone else alone? I'm not sure how because I don't know the product.
Also, if I understand your patch correctly, you fix the issue by making sure that the compilation is against whatever version of zlib Anacondian's decided to bring in. IMHO a bad practice firstly because you switch your security updates/trust from OS vendor to Anaconda's vendor.

Can you please paste the compilation line that fails? Would you mind attaching the output of printenv?
I'm interested if you have pre-defined CFLAGS or LDFLAGS set for some reason.
Also CMakeCache.txt might be an interesting read to see what ZLIB finder found.

[theGreatWhiteShark]

Hey @bratner.
Yes, using a possibly outdated version of a package provided by Anaconda is a bad idea. Having it locked on a specific version is even worse. But when developing some medium size projects in a language as fragile as Python one has to bite the pain.

The reason I suggested this patch is becomes people using Anaconda and having one of the required libraries installed in their custom environments won't be able to build Hydrogen from source. As someone new to the software you will most probably just bin it and try a different one. I'm not sure how many people, if any, will encounter this problem. But right now there isn't an even remotely connected solution I could find online so I think it's unlikely the Anacondians will just solve it themselves.

The issue itself is pretty straight forward (as far as I can tell).
When running cmake for configuration the system-wide library is found (most likely since findZLIB does not considers the Anaconda path).

-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8")

But at the same time it recognizes that the found library is masked by the one installed using Anaconda.

-- Configuring done
CMake Warning at src/core/CMakeLists.txt:8 (ADD_LIBRARY):
Cannot generate a safe runtime search path for target hydrogen-core-1.0.0
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/phil/.anaconda3/lib

Some of these libraries may not be found correctly.

CMake Warning at src/tests/CMakeLists.txt:12 (add_executable):
Cannot generate a safe runtime search path for target tests because files
in some directories may conflict with libraries in implicit directories:

runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/phil/.anaconda3/lib

Some of these libraries may not be found correctly.

CMake Warning at src/cli/CMakeLists.txt:12 (ADD_EXECUTABLE):
Cannot generate a safe runtime search path for target h2cli because files
in some directories may conflict with libraries in implicit directories:

runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/phil/.anaconda3/lib

Some of these libraries may not be found correctly.

CMake Warning at src/player/CMakeLists.txt:10 (ADD_EXECUTABLE):
Cannot generate a safe runtime search path for target h2player because
files in some directories may conflict with libraries in implicit
directories:

runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/phil/.anaconda3/lib

Some of these libraries may not be found correctly.

CMake Warning at src/gui/CMakeLists.txt:46 (ADD_EXECUTABLE):
Cannot generate a safe runtime search path for target hydrogen because
files in some directories may conflict with libraries in implicit
directories:

runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/phil/.anaconda3/lib

Some of these libraries may not be found correctly.

During compilation this results in

[ 53%] Linking CXX executable h2cli
/home/phil/.anaconda3/lib/libpng16.so.16: undefined reference to inflateValidate@ZLIB_1.2.9' collect2: error: ld returned 1 exit status src/player/CMakeFiles/h2player.dir/build.make:140: recipe for target 'src/player/h2player' failed make[2]: *** [src/player/h2player] Error 1 CMakeFiles/Makefile2:541: recipe for target 'src/player/CMakeFiles/h2player.dir/all' failed make[1]: *** [src/player/CMakeFiles/h2player.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... [ 53%] Building CXX object src/tests/CMakeFiles/tests.dir/assertions/file.cpp.o /home/phil/.anaconda3/lib/libpng16.so.16: undefined reference toinflateValidate@ZLIB_1.2.9'
collect2: error: ld returned 1 exit status
src/cli/CMakeFiles/h2cli.dir/build.make:142: recipe for target 'src/cli/h2cli' failed
make[2]: * [src/cli/h2cli] Error 1
CMakeFiles/Makefile2:452: recipe for target 'src/cli/CMakeFiles/h2cli.dir/all' failed
make[1]: [src/cli/CMakeFiles/h2cli.dir/all] Error 2
[ 54%] Building CXX object src/tests/CMakeFiles/tests.dir/automation_path_serializer_test.cpp.o
[ 54%] Building CXX object src/tests/CMakeFiles/tests.dir/automation_path_test.cpp.o
[ 55%] Building CXX object src/tests/CMakeFiles/tests.dir/functional_tests.cpp.o
[ 55%] Building CXX object src/tests/CMakeFiles/tests.dir/instrument_list_test.cpp.o
[ 56%] Building CXX object src/tests/CMakeFiles/tests.dir/main.cpp.o
[ 57%] Building CXX object src/tests/CMakeFiles/tests.dir/midi_note_test.cpp.o
[ 57%] Building CXX object src/tests/CMakeFiles/tests.dir/note_test.cpp.o
[ 58%] Building CXX object src/tests/CMakeFiles/tests.dir/pattern_test.cpp.o
[ 58%] Building CXX object src/tests/CMakeFiles/tests.dir/rubberband.cpp.o
[ 59%] Building CXX object src/tests/CMakeFiles/tests.dir/test_helper.cpp.o
[ 59%] Building CXX object src/tests/CMakeFiles/tests.dir/xml_test.cpp.o
[ 60%] Building CXX object src/tests/CMakeFiles/tests.dir/tests_automoc.cpp.o
[ 61%] Linking CXX executable tests
[ 61%] Built target tests
Makefile:149: recipe for target 'all' failed
make: ** [all] Error 2

It is not just a problem of zlib, having a Qt version installed using Anaconda the compilation would fail as well.

I don't have CFLAGS of LDFLAGS set but I can give you the lines of printenv, which might be relevant

PATH=/home/phil/.anaconda3/bin:/home/phil/.gem/ruby/2.1.0/bin:/home/phil/.luarocks/bin:/home/phil/.go/bin:/home/phil/.go1.9.2/bin:/home/phil/.cabal/bin:/home/phil/git/configurations-and-scripts/bash:/home/phil/git/configurations-and-scripts/awk:/home/phil/git/configurations-and-scripts/python:/home/phil/git/configurations-and-scripts/java:/home/phil/bin:/home/phil/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin:/home/phil/software/android-sdk/tools:/home/phil/software/android-sdk/platform-tools:/home/phil/software/android-sdk/build-tools/25.0.3:/home/phil/software/android-sdk/tools/bin
_=/usr/bin/printenv

But if you all vote for this issue being minor and effectively affecting no one, I can of course drop it :)

[bratner]

@theGreatWhiteShark Hi!

So the problem with zlib is secondary. The primary problem seems to be that your build links against libpng from the private Anaconda libs directory, and that libpng is compiled against their version of zlib.
So your compilation links the system zlib but Anaconda's libpng.
I have just noticed that there is no way you can attach a file here. I would like to take a peek at your CMakeCache.txt.
I found out that running make VERBOSE=1 will print out the full (gcc) command lines used by the compiler. It might be interesting to see the offending compilation command line.
Also cmake warnings are weird. Are you sure there is no LD_LIBRARY_PATH set in your environment?
Try running unset LD_LIBRARY_PATH before running cmake ../ in a fresh build directory.
If that is not the case then maybe Anaconda written itself into /etc/ld.so.conf or included itself in a separate file in /etc/ld.so.conf.d/.
You can find out by running ldconfig -v and then searching through output to see if the Anaconda directory is present in the listing.

I'm not voting for anything. It's easy to see that this is an environmental issue, not a software bug.
IMHO it just might be beneficial to localize, find a solution, and keep on record.

[theGreatWhiteShark]

If you want I can mail the CMakeCache.txt to your redbend email address. Same with the outputs of both the make VERBOSE=1 and ldconfig -v commands. They are very long and it might be a good idea to search them with an editor right away.

Yes, I'm sure there is no LD_LIBRARY_PATH set in my environment and unsetting the variable beforehand doesn't change anything. Also in /etc/ld.so.conf.d/ there is now trace Anaconda. The only environmental variable containing any trace of their custom environment is the PATH one and thus only executables should be accessible. Those executables, e.g. qmake, are linked against different (versions) of shared objects. Now, when those binaries are called during the compilation process and cmake wants to use the system's and not Anaconda's versions of the libraries, things go south and the build fails. At least this is what I'm thinking is going on in here. But my Linux-level might not be high enough to see the bigger picture.

[bratner]

My gmail address is best. I'm not sure I have access to redbend address. The only reason the later is visible in the logs is because of me being noob at working a public git.
You can also paste in pastebin.com (unlisted with expiration date if that is a concern) also I have subscribed to the mailing list so attachments can be sent there.

Seems that there is a way to attach files right here as well by clicking below the dotted line below the comment entry or drag-and-drop. I'll give it a try here:
Hmm... has strict extension requirement. Had to rename to .txt to attach
ldconfig.txt

Regarding the issue at hand. This is still a weird behavior. Here is my h2cli link line from verbose make.
It doesn't even link to png and I don't see in the source of anything that would.

[ 59%] Linking CXX executable h2cli
cd /home/bratner/projects/hydrogen/build_debug/src/cli && /usr/bin/cmake -E cmake_link_script CMakeFiles/h2cli.dir/link.txt --verbose=1
/usr/bin/g++    -O0 -std=gnu++11 -fno-implement-inlines -g    CMakeFiles/h2cli.dir/main.cpp.o CMakeFiles/h2cli.dir/h2cli_automoc.cpp.o  -o h2cli -rdynamic ../core/libhydrogen-core-1.0.0.so /home/bratner/Qt5.9.2/5.9.2/gcc_64/lib/libQt5Widgets.so.5.9.2 -llo -ltar -larchive -lsndfile -lasound -ljack -lpulse -llrdf -lz -lpthread -llo /home/bratner/Qt5.9.2/5.9.2/gcc_64/lib/libQt5Xml.so.5.9.2 /home/bratner/Qt5.9.2/5.9.2/gcc_64/lib/libQt5XmlPatterns.so.5.9.2 /home/bratner/Qt5.9.2/5.9.2/gcc_64/lib/libQt5Network.so.5.9.2 /home/bratner/Qt5.9.2/5.9.2/gcc_64/lib/libQt5Gui.so.5.9.2 /home/bratner/Qt5.9.2/5.9.2/gcc_64/lib/libQt5Core.so.5.9.2 -Wl,-rpath,/home/bratner/projects/hydrogen/build_debug/src/core:/home/bratner/Qt5.9.2/5.9.2/gcc_64/lib: 
make[2]: Leaving directory '/home/bratner/projects/hydrogen/build_debug'
[ 59%] Built target h2cli

[theGreatWhiteShark]

Ah, indeed.
All I had to do was to activate some additional JavaScript stuff.

CMakeCache.txt
ldconfig_output.txt
make_verbose_output.txt

[elpescado]

Hi, it looks like your make output doesn't contain error messages - it seems you captured only standard output, while error messages were printed to standard error. Append 2>&1 to command line to capture both.

BTW, I think we're not the only ones bitten (constricted?) by Anaconda:
https://github.com/pism/pism/issues/356
https://stackoverflow.com/questions/29930280/how-to-resolve-cmake-directory-libz-so-1-conflict-with-library-in-implicit-direc
https://stackoverflow.com/questions/48447728/cmake-runtime-library-hidden-files?rq=1
https://gitlab.com/morpheus.lab/morpheus/issues/3
https://askubuntu.com/questions/928465/cmake-library-directory-conflict-warning-and-error
https://cmake.org/pipermail/cmake/2009-November/033295.html

I feel solution presented in first issue most sensible:

Solution

When you run cmake, pass it the PATH variable with conda removed:

PISM_INSTALL_PREFIX=~/pism PATH=$NOCONDA_PATH cmake ..

you can set the NOCONDA_PATH variable in your .bashrc (or similar) just before you add conda to your path like so:

export NOCONDA_PATH=$PATH
export PATH=$HOME/anaconda3/bin:$PATH

Maybe we should document it in FAQ or something.

I think that the root cause is that Anaconda breaks OS setup by modifying environment. Trying to modify environment even further in hope to undo changes made by Anaconda seems fragile (futile?) to me.

[bratner]

@theGreatWhiteShark
Here is what happened in your case. It all starts with QT distro being provided by Anaconda.
This is where the PATH precedence of Anaconda takes place.
You can run which qmake and qmake -v to see if this conclusion is correct and it points to Anaconda's QT5, not the system one.

Now the QT5 distro of Anaconda is compiled against libpng which is included in Anaconda itself and that libpng was compiled with version 1.2.9 of libz which is also brought in by anaconda but the linker tries to link against the system version. Which is why your fix works.

This is how you can see it in your CMakeCache.txt, QT related paths are in Anaconda distribution.

....
//The directory containing a CMake configuration file for Qt5Core.
Qt5Core_DIR:PATH=/home/phil/.anaconda3/lib/cmake/Qt5Core
....

Are you using Anaconda provided QT5 to compile hydrogen on purpose?

@elpescado suggested to remove Anaconda binary directory from PATH before running cmake on a fresh build dir and see what changed in CMakeCache.txt.

[theGreatWhiteShark]

@elpescado
Yes, adding a dedicated question in the FAQ with the solution you found sounds like a better option than modifying the cmake configuration.

@bratner

Are you using Anaconda provided QT5 to compile hydrogen on purpose?

No. I was using a Python-based framework, which was built upon Qt5, in another project.

[bratner]

So does removing anaconda from path before running cmake solves the issue?

[theGreatWhiteShark]

Yes, everything works just fine since Anaconda is out of the way.

@elpescado
I wonder if the wiki of this Github page will be a better place for note about the Anaconda build. After all developers failing to compile the code might search in here first.

[theGreatWhiteShark]

I've added a pull request containing an additional question in the FAQ and close this issue as well as the corresponding pull request

[trebmuh]

_(see here)_