Spack: Compiler in spec ignored for build dependencies

Created on 9 Dec 2016 · 37Comments · Source: spack/spack

I've never noticed this before, but I don't think it's the intended behavior??

$ spack spec eigen %intel
Input spec
--------------------------------
eigen%intel

Normalized
--------------------------------
eigen%intel
    ^cmake

Concretized
--------------------------------
[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"~debug+fftw+metis+mpfr+scotch+suitesparse arch=linux-centos6-x86_64
    ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-centos6-x86_64
        ^[email protected]%[email protected] arch=linux-centos6-x86_64
        ^[email protected]%[email protected] arch=linux-centos6-x86_64
            ^[email protected]%[email protected]+pic arch=linux-centos6-x86_64
    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"+float+long_double+mpi+openmp~pfft_patches~quad arch=linux-centos6-x86_64
        ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"~debug~gforker+hydra~mrail~nemesis~nemesisib~nemesisibtcp+psm~remshell~slurm~sock arch=linux-centos6-x86_64
            ^[email protected]%[email protected] arch=linux-centos6-x86_64
                ^[email protected]%[email protected]+sigsegv arch=linux-centos6-x86_64
                    ^[email protected]%[email protected] arch=linux-centos6-x86_64
            ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64
                ^[email protected]%[email protected] arch=linux-centos6-x86_64
                ^[email protected]%[email protected]+internal_glib arch=linux-centos6-x86_64
                ^[email protected]%[email protected] arch=linux-centos6-x86_64
    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64
    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"~debug~gdb~idx64~real64+shared arch=linux-centos6-x86_64
    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64
    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"+compression~esmumps+metis~mpi+shared arch=linux-centos6-x86_64
        ^[email protected]%[email protected] arch=linux-centos6-x86_64
            ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos6-x86_64
                ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64
                ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"~python arch=linux-centos6-x86_64
                    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64
                ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64
            ^[email protected]%[email protected] arch=linux-centos6-x86_64
    ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"+fpic+tbb arch=linux-centos6-x86_64
        ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"+fpic+openmp+shared arch=linux-centos6-x86_64
        ^[email protected]%[email protected] cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64

It looks like all of the build dependencies are building with GCC even though Intel was specified. @mathstuf Is this how things are supposed to work?

concretization documentation

Source

adamjstewart

Most helpful comment

It seems like we should just be concretizing the DAG of things reachable via link edges, then concretize the build deps node by node per-build.

tgamblin on 9 Dec 2016

👍2

All 37 comments

do you by any chance specify which compiler to prefer in packages.yaml?

davydden on 9 Dec 2016

@davydden Nope. Can you not reproduce this? Mac defaults to clang now right? Try running spack spec eigen %gcc

adamjstewart on 9 Dec 2016

yes, i think i see similar effects:

$ spack spec eigen%gcc
Input spec
--------------------------------
eigen%gcc

Normalized
--------------------------------
eigen%gcc
    ^cmake

Concretized
--------------------------------
[email protected]%[email protected]~debug+fftw+metis+mpfr+scotch+suitesparse arch=darwin-sierra-x86_64
    ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=darwin-sierra-x86_64
        ^[email protected]%[email protected] arch=darwin-sierra-x86_64
        ^[email protected]%[email protected] arch=darwin-sierra-x86_64
            ^[email protected]%[email protected]+pic arch=darwin-sierra-x86_64
    ^[email protected]%[email protected]+float+long_double~mpi~openmp~pfft_patches~quad arch=darwin-sierra-x86_64
    ^[email protected]%[email protected] arch=darwin-sierra-x86_64
        ^[email protected]%[email protected]+sigsegv arch=darwin-sierra-x86_64
            ^[email protected]%[email protected] arch=darwin-sierra-x86_64
    ^[email protected]%[email protected]~debug~gdb~idx64~real64+shared arch=darwin-sierra-x86_64
    ^[email protected]%[email protected] arch=darwin-sierra-x86_64
    ^[email protected]%[email protected]+compression~esmumps+metis~mpi+shared arch=darwin-sierra-x86_64
        ^[email protected]%[email protected] arch=darwin-sierra-x86_64
        ^[email protected]%[email protected] arch=darwin-sierra-x86_64
            ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-sierra-x86_64
                ^[email protected]%[email protected] arch=darwin-sierra-x86_64
                ^[email protected]%[email protected]~python arch=darwin-sierra-x86_64
                    ^[email protected]%[email protected] arch=darwin-sierra-x86_64
                ^[email protected]%[email protected] arch=darwin-sierra-x86_64
            ^[email protected]%[email protected] arch=darwin-sierra-x86_64
    ^[email protected]%[email protected]+fpic+tbb arch=darwin-sierra-x86_64
        ^[email protected]%[email protected]+fpic~openmp+shared arch=darwin-sierra-x86_64
        ^[email protected]%[email protected] arch=darwin-sierra-x86_64

davydden on 9 Dec 2016

It looks like all of the build dependencies are building with GCC even though Intel was specified.

That was me who added this behavior (and it is intentional): build deps now default to frontend compiler. If this is not desirable alternatives can be discussed.

PR was: https://github.com/LLNL/spack/pull/2292

scheibelp on 9 Dec 2016

This part appears incorrect though:

        ^[email protected]%[email protected] arch=darwin-sierra-x86_64
            ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-sierra-x86_64
                ^[email protected]%[email protected] arch=darwin-sierra-x86_64
                ^[email protected]%[email protected]~python arch=darwin-sierra-x86_64
                    ^[email protected]%[email protected] arch=darwin-sierra-x86_64
                ^[email protected]%[email protected] arch=darwin-sierra-x86_64
            ^[email protected]%[email protected] arch=darwin-sierra-x86_64

bzip2 and others should have been built with clang in this case - I'll need to check into this (ASAP)

scheibelp on 9 Dec 2016

Interesting. I don't _think_ it will be a problem for me? And it will lead to less duplication of build-dependencies, which no one really cares that much about. Is there a way to specify what compiler the build dependencies are built with, or do you have to do them one at a time on the command line? Do they still reflect preferences chosen inpackages.yaml? I think there are a lot of questions still remaining about how things work now.

adamjstewart on 9 Dec 2016

Do they still reflect preferences chosen in packages.yaml?

Yes - with the caveat that it filters out compilers that aren't defined for the frontend architecture.

Is there a way to specify what compiler the build dependencies are built with, or do you have to do them one at a time on the command line?

At the moment one-at-a-time is the only way. There could be an option to disable the behavior entirely

scheibelp on 9 Dec 2016

That was me who added this behavior (and it is intentional):

is it documented? If not, I am adding the label to remind ourselves about it.

davydden on 9 Dec 2016

is it documented? If not, I am adding the label to remind ourselves about it.

No, and it should be come to think of it.

scheibelp on 9 Dec 2016

Btw, why was this change made exactly? Is it only necessary for Cray/BGQ clusters?

adamjstewart on 9 Dec 2016

@adamjstewart It's supposed to allow the build dependencies to be recycled better and to avoid building them with "weird" compilers. It's useful not only on Cray and BG/Q, but also on clusters like the new ppc64le machines at LLNL and ORNL, where we want to build codes with compilers like XL, but nobody wants to port, say, cmake or Qt (which cmake depends on) to XL. It would even be useful on regular Linux clusters when building with things like PGI.

IBM is working on packaging math libraries and tools with Spack for us and ORNL, and they were running into these issues, so we told them to focus on getting the HPC stuff building with the fancy compilers and not to worry about build deps.

tgamblin on 9 Dec 2016

👍1

Would it be useful to separate the build deps from the rest of the DAG in the output of spack spec?

tgamblin on 9 Dec 2016

Ah, that makes sense. I agree that it is nice to recycle build dependencies. I was just really surprised when I saw that change out of the blue. I think it's fine as is.

adamjstewart on 9 Dec 2016

Would it be useful to separate the build deps from the rest of the DAG in the output of spack spec?

@mathstuf suggested to resolve them separately https://github.com/LLNL/spack/issues/839 , which i would say is good to have.

davydden on 9 Dec 2016

👍1

It seems like we should just be concretizing the DAG of things reachable via link edges, then concretize the build deps node by node per-build.

tgamblin on 9 Dec 2016

👍2

It seems like we should just be concretizing the DAG of things reachable via link edges, then concretize the build deps node by node per-build.

right, there is no reason to have build-only dependencies in DAG as they won't influence anything, they are auxiliary packages needed to build things and that's it.

davydden on 9 Dec 2016

OK one complexity in the example in this thread:

The build dependency flex links to packages which are also linked (transitively) by the root. The bug where bzip2 gets a different compiler than gettext comes up because at the time of compiler concretization for gettext, the bzip2 dependency has not been materialized (I suspect because it is activated by a variant); therefore at the time when the compiler is concretized for gettext it does not perceive that one of its dependencies is linked from the root. At the moment my only thought for fixing this is to retroactively update compilers (which I think is doable - I'll work on a PR for this today).

Basically, the question comes up: how to handle build dependencies when one of their own link dependencies is linked from the root. If only one instance of each package may exist in the concretization then the build dependency must also be built with the same compiler as the root. If it's possible to build a package multiple times (I don't immediately see how to make this happen fast) then it can use a different compiler.

scheibelp on 9 Dec 2016

right, there is no reason to have build-only dependencies in DAG as they won't influence anything, they are auxiliary packages needed to build things and that's it.

I think they should eventually be in the DAG for provenance (in case their affect the build somehow) but right now we don't include them to avoid having to rebuild things JUST because of a build dep. Once we can consider installed things during concretization I was planning to put them back.

But yes -- I think their resolution should be entirely separate. We could even consider resolving them one-by-one so that if they have conflicting link dependencies, they do not interfere. The "normalization" restriction we use in concretization (i.e., only one instance of any dependency in the DAG, ever) stems from the conditions needed to get the dynamic linker to work consistently. It only has one symbol namespace. But since the build deps are really run as separate processes they don't have that restriction.

tgamblin on 9 Dec 2016

👍1

@tgamblin One thing I want to avoid is running spack spec, seeing that a package A depends on B and C, then running spack install and being very confused why Spack is building D. It's good to know what will be built, but yes we should ignore them in the hashing part of the process.

adamjstewart on 9 Dec 2016

Build dependencies do need to resolve against the DAG subset where they are used, so if a project links Python itself and one of its build dependencies uses Python, the Python libraries must match otherwise you may get the wrong Python library loaded (even if we didn't use LD_LIBRARY_PATH now, PATH on Windows would require things to be set up this way).

mathstuf on 9 Dec 2016

I'm taking a look into the feasibility of resolving build dependencies separately - specifically if it can be done by late next week. If I assess that it cannot be done in that time (or if people voice an opinion on it) then I'll create a PR to revert #2292. In the meantime, concretization for eigen (and likely other complex packages) is broken. A temporary fix would be:

spack spec eigen%intel+scotch ^gettext%intel

Or making intel a preferred compiler.

scheibelp on 9 Dec 2016

Build dependencies do need to resolve against the DAG subset where they are used, so if a project links Python itself and one of its build dependencies uses Python, the Python libraries must match otherwise you may get the wrong Python library loaded

@mathstuf If I understand correctly this is a situation where package X links to python and

X -> Y -> python

Python is a run dependency of Y and Y is a build dependency of X. If python were simply a build dependency of Y then I presume the python used by Y and by X could be different.

scheibelp on 9 Dec 2016

That won't work because if X needs Python3 for linking and Y needs Python2 for running, what does one set PYTHONPATH to?

mathstuf on 9 Dec 2016

Y needs Python2 for running

but i think we discuss the case of build-only dependencies, that is the case when the edge from a node to python is neither link nor run.

davydden on 9 Dec 2016

That won't work because if X needs Python3 for linking and Y needs Python2 for running, what does one set PYTHONPATH to?

I'm confused because my comment was an attempt to create an example of what you were saying didn't work (I wasn't asserting that in fact it should work):

this is a situation where package X links to python and
X -> Y -> python
Python is a run dependency of Y and Y is a build dependency of X.

I went on to say that if Python was a build dependency of Y and not a run dependency, then you could build two separate pythons.

Does that make sense?

scheibelp on 9 Dec 2016

That won't work because if X needs Python3 for linking and Y needs Python2 for running, what does one set PYTHONPATH to?

A couple additional points:

Perhaps most importantly - this sounds like a difficult problem with or without Spack. Does it appear in general? How is it handled there?
For the purposes of building X perhaps it just needs PYTHONPATH to refer to Python2, when it comes to running X it could refer to Python3. I'm likely overlooking something about how library finding would work

scheibelp on 9 Dec 2016

@scheibelp This is a very common problem. Most packages that rely on Python as a build dependency rely on python@2. If you are trying to build a Python 3 module and one of its dependencies relies on Python 2 as a build dependency, Spack is dead in the water.

adamjstewart on 9 Dec 2016

How is it handled there?

All modules are found in the default site-packages directories and no PYTHONPATH is needed at all. Spack gets into trouble because it needs to construct a PYTHONPATH.

I had more, but @adamjstewart said it more concisely :) .

mathstuf on 9 Dec 2016

This is a very common problem. Most packages that rely on Python as a build dependency rely on python@2. If you are trying to build a Python 3 module and one of its dependencies relies on Python 2 as a build dependency, Spack is dead in the water.

I think the key difference between this and the case I mentioned is that Python2 is a build dependency in this case and not a run dependency. Given a case like:

(python3 module) -> (dependency X) -> (python2 as a build dependency)

In this case I imagine Spack could (although it doesn't right now) omit Python2 setup when building the python3 module.

If the following were instead the case:

(python3 module) -> (dependency X) -> (python2 as a __run__ dependency)

This would seem difficult, but I'm guessing it isn't common.

scheibelp on 9 Dec 2016

We're on the same page :) . In any case, that is #839.

mathstuf on 9 Dec 2016

I'm taking a look into the feasibility of resolving build dependencies separately - specifically if it can be done by late next week. If I assess that it cannot be done in that time (or if people voice an opinion on it) then I'll create a PR to revert #2292.

@tgamblin

2548 is the start of an attempt at this. It occurs to me that even if it doesn't take much time, this feature would override #2292. Therefore I think it should be reverted. The PR for that is at: https://github.com/LLNL/spack/pull/2549

scheibelp on 10 Dec 2016

I am not sure if this is related, please feel free to disregard. But when this happened to me it was from a spack install eigen %[email protected], which ultimately failed trying to build openblas because I was dumb and didn't specify a fortran compiler in my compilers.yaml.

At the time of the failed installation I had an openblas %[email protected]. So if the decision is to keep the re-use feature, openblas was missed (via the suite-sparse dependency).

Happy to provide more information if it is relevant / useful.

svenevs on 10 Dec 2016

@svenevs This isn't a re-use feature, it is a "use the default compiler for build dependencies" feature. One side-effect of using the default compiler is you are more likely to reuse a single installation if you are trying to build the same software stack with multiple compilers. But if you don't have that build dependency built with the default compiler, Spack would still rebuild it. In your case, openblas was reinstalled because it isn't a build dependency, so we actually care about what compiler it was built with and want to avoid mixing compilers. It generally isn't a problem with C, but I've heard that mixing C++ compilers with different dependencies can cause problems.

adamjstewart on 12 Dec 2016

but I've heard that mixing C++ compilers with different dependencies can cause problems

C++ compilers use completely different ABIs since there is no standard (every MSVC version crossed with release and debug build modes and the calling conventions/mangling of GCC exist). Those two are the main ABIs and the standard library in use pretty much dictates it.

mathstuf on 12 Dec 2016

🎉1

I haven't read through all the comments but maybe someone has suggested this.
Is it possible to put global configuration in installation command?
For instance