Spack: Multiple providers found for blas

[davydden]

on a fresh spack ae9a9e019a73cb951d4d2a2585ac71a53f351c81 I suddenly see the following issue:

$ spack install dealii@develop%[email protected] ^openblas
==> Error: Multiple providers found for 'blas': ['[email protected]%[email protected]~all+daal~ilp64+ipp+mkl+mpi~newdtags~openmp+rpath+shared+tools arch=linux-ubuntu16-x86_64', '[email protected]%[email protected]~openmp+pic+shared arch=linux-ubuntu16-x86_64']

Yes, there are multiple providers for blas, but I don't understand why is it a problem now... Note that the problem is there even though there is ^openblas.

p.s. this is without any packages.yaml file, only compilers.yaml

---

TLDR: currently a workaround is to specify both mpi and lapack providers, that is do

spack install dealii@develop%[email protected]+mpi ^openmpi ^openblas

---

MWE of the problem to be added to the unit tests is https://github.com/LLNL/spack/issues/4107#issuecomment-298889480

[alalazo]

Was able to reproduce this. I vaguely remember a similar issue open in the past, but couldn't find it. @adamjstewart Does this ring a bell?

[davydden]

funny enough, bisection led to multi-valued variant PR https://github.com/LLNL/spack/pull/2386

[davydden]

@alalazo :

$ git bisect bad
9e4b0eb34a66927ca92df79dedc68d35c9fbd4ae is the first bad commit
commit 9e4b0eb34a66927ca92df79dedc68d35c9fbd4ae
Author: Massimiliano Culpo

https://github.com/LLNL/spack/commit/9e4b0eb34a66927ca92df79dedc68d35c9fbd4ae
:smile:

[alalazo]

Ouch

[davydden]

i looked through the PR and did not notice anything that could cause this behaviour...

[alalazo]

So, the problem seems to be trilinos, and it does work for simpler packages that depend on blas:

$ spack spec py-numpy
Input spec
--------------------------------
py-numpy

Normalized
--------------------------------
py-numpy
    ^py-setuptools
        ^[email protected]:2.8,3.4:
            ^bzip2
            ^ncurses
                ^pkg-config
            ^openssl
                ^zlib
            ^readline
            ^sqlite

Concretized
--------------------------------
[email protected]%[email protected]+blas+lapack arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~openmp+pic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]+shared~tk~ucs4 arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected]+shared arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected]~symlinks arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected]+internal_glib arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected]+pic+shared arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 

$ spack spec trilinos
Input spec
--------------------------------
trilinos

Normalized
--------------------------------
trilinos
    ^blas
    ^cmake
    ^glm
    ^lapack
    ^matio
    ^mpi
    ^netcdf+mpi
        ^hdf5
            ^[email protected]:
        ^m4

Concretized
--------------------------------
==> Error: Multiple providers found for 'lapack': ['[email protected]%[email protected]~all+daal~ilp64+ipp+mkl+mpi~newdtags~openmp+rpath+shared+tools arch=linux-ubuntu14-x86_64', '[email protected]%[email protected]~openmp+pic+shared arch=linux-ubuntu14-x86_64']

[davydden]

that's strange... because things work for petsc, BUT trilinos uses netcdf which became multi-variant... maybe the issue is in this direction?

[davydden]

but other packages that need netscdf and use blas are ok.... I think one just has to debug it with some python specific tool similar to GDB...

[davydden]

@alalazo if you still have your branch around, can you try bisecting individual commits? That should result in a more specific patch that broke it as opposed to the whole PR squashed in a single commit.

[alalazo]

@davydden It's an issue with concretization caused by:

depends_on('mpi')

in trilinos. That brings in for some reason intel-parallel-studio which also provides lapack and blas, hence the error. Now if you do for instance:

$ spack spec trilinos ^mpich
Input spec
--------------------------------
trilinos
    ^mpich

Normalized
--------------------------------
trilinos
    ^blas
    ^cmake
    ^glm
    ^lapack
    ^matio
    ^mpich
    ^netcdf+mpi
        ^hdf5
            ^[email protected]:
        ^m4

Concretized
--------------------------------
[email protected]%[email protected]+boost~debug~exodus+hdf5+hypre+metis+mumps~python+shared+suite-sparse~superlu+superlu-dist+tpetra~xsdkflags arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+atomic+chrono+date_time~debug+filesystem~graph~icu+iostreams+locale+log+math~mpi+multithreaded+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer+wave arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+pic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~symlinks arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]+internal_glib arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+hydra+pmi+romio~verbs arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~int64~internal-superlu+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~openmp+pic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+hdf5+shared+zlib arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~debug~gdb~int64~real64+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~fpic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~cdmremote~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+sigsegv arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~debug~gdb+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+pic~tbb arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~int64 arch=linux-ubuntu14-x86_64 

[alalazo]

Also, if you comment the single line:

depends_on('mpi')

Spack is also able to concretize:

$ spack spec trilinos
Input spec
--------------------------------
trilinos

Normalized
--------------------------------
trilinos
    ^blas
    ^cmake
    ^glm
    ^lapack
    ^matio
    ^netcdf+mpi
        ^hdf5
            ^[email protected]:
        ^m4

Concretized
--------------------------------
[email protected]%[email protected]+boost~debug~exodus+hdf5+hypre+metis+mumps~python+shared+suite-sparse~superlu+superlu-dist+tpetra~xsdkflags arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+atomic+chrono+date_time~debug+filesystem~graph~icu+iostreams+locale+log+math~mpi+multithreaded+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer+wave arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+pic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~symlinks arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]+internal_glib arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]~cuda+libxml2+pci arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                        ^[email protected]%[email protected]+sigsegv arch=linux-ubuntu14-x86_64 
                            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected]~python arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~int64~internal-superlu+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~openmp+pic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+hdf5+shared+zlib arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~debug~gdb~int64~real64+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~fpic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~cdmremote~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~debug~gdb+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+pic~tbb arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~int64 arch=linux-ubuntu14-x86_64 

[davydden]

@alalazo interesting... did you have a chance restoring your branch on GitHub to see which part of the changes uncover/produce such behavior?

[alalazo]

I have been able to reduce the interesting behavior to a package with just:

variant('mumps',  default=True, description='Compile with support for MUMPS solvers')

depends_on('mpi')
depends_on('scalapack', when='+mumps')

This gives:

$ spack spec trilinos
Input spec
--------------------------------
trilinos

Normalized
--------------------------------
trilinos
    ^cmake
    ^mpi

Concretized
--------------------------------
[email protected]%[email protected]+mumps arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~symlinks arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]+internal_glib arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]+pic+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~ilp64~openmp+shared arch=linux-ubuntu14-x86_64 # <--- Where does this come from?
    ^[email protected]%[email protected]~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~cuda+libxml2+pci arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected]+sigsegv arch=linux-ubuntu14-x86_64 
                        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]~python arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 

If you remove any of the above (even if you make the second variant unconditional) everything works as expected and you get openblas and openmpi.

[davydden]

    ^[email protected]%[email protected]~ilp64~openmp+shared arch=linux-ubuntu14-x86_64 # <--- Where does this come from?

maybe it became default provider somehow? strange...

[alalazo]

@davydden I took the freedom of adding a workaround to the issue description.

[adamjstewart]

maybe it became the default provider somehow

@svenevs and I noticed some really strange behavior the other day where py-pil was being chosen as the default pil provider even though it should be py-pillow according to etc/spack/defaults/packages.yaml. It may be worthwhile to make modifications to this file and make sure it is actually being used.

[adamjstewart]

@alalazo Yes, I have run into something similar before. See #3764. The problem boiled down to:

1. I was trying to build something that depended on MPI and BLAS
2. I wanted to use mvapich2 and intel-parallel-studio+mkl~mpi
3. Spack correctly resolved the spec (both mvapich2 and intel-parallel-studio were installed)
4. But when I called spec['mpi'].mpicc it would use intel-parallel-studio instead of mvapich2.

My current workaround is to comment out provides('mpi') in intel-parallel-studio

[davydden]

i don't think the issue is due to @alalazo 's PR, but rather that it uncovered some bug elsewhere during refactoring. I still think it would be good to find out which part of this PR actually led to such behaviour so that one can have a better guess of what is broken.

It may be worthwhile to make modifications to this file and make sure it is actually being used.

yes, i agree. Somehow it is not being used...

[davydden]

I still think it would be good to find out which part of this PR actually led to such behaviour so that one can have a better guess of what is broken.

i tried bisecting this PR but unfortunately you can't really do this because some commits are not functional. So I guess we can only look at it as a whole...

[KineticTheory]

I'm also seeing this when trying to build trilinos:

spack install [email protected]%[email protected] +boost +hypre [email protected] ^hypre~internal-superlu ^netlib-lapack
==> Error: Multiple providers found for lapack...

I had to comment out

provides('lapack')
provides('blas')
provides('scalapack')

from intel-mkl and intel-parallel-studio to get trilinos to build.

[alalazo]

@KineticTheory So the workaround in the description (specify both mpi and lapack) didn't work at all?

[KineticTheory]

@alalazo Working on this last night, somehow I missed the part about specifying mpi. I was only specifying lapack. I've tried it again this morning, and it appears that specifying both lapack and mpi avoids the problem.

[davydden]

@alalazo did you have a chance to try to debug this issue?

[alalazo]

Not really. Might be related to the things solved in #4129 (the semantic of could vs. can) but I still need to have a look at it.

[davydden]

FYI, this issue happens even if you have

packages:
  all:
    providers:
      mpi: [openmpi]
      blas: [openblas]
      lapack: [openblas]

which, together with the fact that default providers are ignored, points to the direction that these settings are not _ignored_, but rather somehow overwritten during concretization.

@alalazo do you think you could try to debug this issue?

[adamjstewart]

Ran into this bug during my Spack Tutorial yesterday in front of 25-30 people considering using Spack. This needs to get fixed.

[davydden]

Ran into this bug during my Spack Tutorial yesterday in front of 25-30 people considering using Spack. This needs to get fixed.

(sad)

[jrood-nrel]

I'm experiencing this as well.

[user@machine spack]$ pwd
/home/user/spack/etc/spack
[user@machine spack]$ cat packages.yaml 
packages:
  all:
    compiler: [[email protected], [email protected]]
    providers:
      mpi: [openmpi]
      blas: [netlib-lapack]
      lapack: [netlib-lapack]
[user@machine spack]$ spack stage trilinos
==> Error: Multiple providers found for 'lapack': ['[email protected]%[email protected]~all+daal~ilp64+ipp+mkl+mpi~newdtags~openmp+rpath+shared+tools arch=linux-centos7-x86_64', '[email protected]%[email protected]~debug~external-blas+lapacke+shared arch=linux-centos7-x86_64']
[user@machine spack]$ git log
commit 541496dfe150f50a11531156a108f7741c5d1177
Author: becker33 <email>
Date:   Fri Jun 16 12:31:56 2017 -0700

[adamjstewart]

@alalazo I can reproduce this as well with the minimalist packages.yaml of:

packages:
  all:
    providers:
      blas: [netlib-lapack]

It looks like we missed a case. Switching netlib-lapack to openblas makes things work again, so it may be specific to that package. Tried testing with intel-mkl as well but got a completely different error I've never seen before:

==> Error: Class constructor failed for package 'builtin.intel-mkl'.

Caused by:
AttributeError: can't set attribute
  File "/home/ajstewart/spack/lib/spack/spack/repository.py", line 579, in get
    self._instances[key] = package_class(copy)
  File "/home/ajstewart/spack/lib/spack/spack/package.py", line 573, in __init__
    self.license_required = False

[alalazo]

Right now (246c07f864d2db571f7e3ab55e875a4e75b93281):

$ spack spec  dealii@develop ^openblas
Input spec
--------------------------------
dealii@develop
    ^openblas

Normalized
--------------------------------
dealii@develop
    ^bzip2
    ^cmake
    ^muparser
    ^openblas
    ^suite-sparse
    ^tbb
    ^zlib

Concretized
--------------------------------
dealii@develop%[email protected]+arpack~doc+gsl+hdf5~int64+metis+mpi+netcdf+oce~optflags+p4est+petsc+python+slepc+trilinos arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+mpi+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]~symlinks arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected]+internal_glib arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected]+pic+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~openmp+pic+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]~cuda+libxml2+pci arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                        ^[email protected]%[email protected]+sigsegv arch=linux-ubuntu14-x86_64 
                            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
                ^[email protected]%[email protected]~python arch=linux-ubuntu14-x86_64 
                    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+atomic+chrono+date_time~debug+filesystem~graph~icu+iostreams+locale+log+math+mpi+multithreaded+program_options+python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer+wave arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+shared~tk~ucs4 arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~debug~gdb~int64+real64+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~cdmremote~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~X11+tbb arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+boost~complex~debug+double+hdf5+hypre~int64+metis+mpi~mumps+shared+superlu-dist~trilinos arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~int64~internal-superlu+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~debug~gdb+shared arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]~int64 arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]+arpack arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~cuda~openmp+pic~tbb arch=linux-ubuntu14-x86_64 
    ^[email protected]%[email protected]~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost~debug~dtk+epetra+epetraext+exodus+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2+instantiate~instantiate_cmplx+metis+ml+muelu+mumps~openmp~pnetcdf~python+sacado+shared~stk+suite-sparse~superlu+superlu-dist+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected] arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+hdf5+shared+zlib arch=linux-ubuntu14-x86_64 
        ^[email protected]%[email protected]+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu14-x86_64 
            ^[email protected]%[email protected]~fpic+shared arch=linux-ubuntu14-x86_64

Can we consider the issue closed?

[alalazo]

@davydden I am going to close this. Feel free to reopen if still an issue.

[RemiLacroix-IDRIS]

Hello there,

It seems like I can reproduce this issue with the latest version of Spack when trying to install Trilinos with ESSL as the BLAS provider:

$ bin/spack spec -I trilinos
Input spec
--------------------------------
     trilinos

Concretized
--------------------------------
==> Error: Multiple providers found for 'blas': ['[email protected]%[email protected]~cuda~ilp64 threads=openmp arch=linux-rhel7-ppc64le', '[email protected]%[email protected]+shared arch=linux-rhel7-ppc64le']

Here is how my packages.yaml file looks like:

packages:
  all:
    providers:
      blas: [essl]
      mpi: [spectrum-mpi]
  essl:
    paths:
      [email protected]: /opt/ibmmath/essl/5.5
    buildable: False
  spectrum-mpi:
    paths:
      [email protected]: /opt/ibm/spectrum_mpi
    buildable: False

---

On a side note, I am seeing something weird if I don't define a spectrum-mpi external package and set it as provider for MPI.

That is, if my packages.yaml file looks like:

packages:
  all:
    providers:
      blas: [essl]
  essl:
    paths:
      [email protected]: /opt/ibmmath/essl/5.5
    buildable: False

then I get the following error:

$ bin/spack spec -I trilinos
Input spec
--------------------------------
     trilinos

Concretized
--------------------------------
==> Error: There are no valid versions for spectrum-mpi that match ':'

I don't understand this error as I would expect Spack to use another provider for MPI (even if that means building OpenMPI or MPICH for example). I could not find any explicit dependency with Spectrum MPI in the ESSL package.

Note that I am not sure whether it is related or not with the issue tackled by this ticket and I would happy open another support request if you like.

[RemiLacroix-IDRIS]

Any news about this? I am also getting similar errors with other virtual dependencies, for example jpeg:

bin/spack spec -It mxnet               
Input spec
--------------------------------
     [    ]  mxnet

Concretized
--------------------------------
==> Error: Multiple providers found for 'jpeg': ['libjpeg', '[email protected]%[email protected] arch=linux-rhel7-ppc64le  ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-rhel7-ppc64le  ^nasm']

[scheibelp]

Regarding https://github.com/spack/spack/issues/4107#issuecomment-402758462

I can reproduce this issue with the latest version of Spack when trying to install Trilinos with ESSL as the BLAS provider
Error: Multiple providers found for 'blas': ['[email protected]%[email protected]~cuda~ilp64 threads=openmp arch=linux-rhel7-ppc64le', '[email protected]%[email protected]+shared arch=linux-rhel7-ppc64le']

If there is a dependency which requires lapack (e.g. py-numpy), then the concretization process will pull in a lapack-provider. Many lapack providers also provide blas, which Spack considers a conflict (it doesn't want two packages which provide the same virtual). I don't currently see any lapack-only providers in Spack, so building with essl will not be possible for the time being.

Based on https://github.com/spack/spack/issues/4107#issuecomment-405515169 though, you have updated your packages.yaml to build without essl. That is a different issue which is resolved by providing hints to the concretizer, I can get it to concretize with the following hints:

spack spec mxnet ^libjpeg ^[email protected] ^vtk+osmesa

More details on this are found on various related bug reports under item 5 of https://github.com/spack/spack/wiki/Issue-Disambiguation.

Also, I'm pretty sure you edited the mxnet package description to replace depends_on(openblas) with depends_on(blas) but I didn't see a mention of that in https://github.com/spack/spack/issues/4107#issuecomment-402758462 - FYI that's worth adding.

[RemiLacroix-IDRIS]

Thanks for your answer.

If there is a dependency which requires lapack (e.g. py-numpy), then the concretization process will pull in a lapack-provider. Many lapack providers also provide blas, which Spack considers a conflict (it doesn't want two packages which provide the same virtual).

Got it, that's an annoying bug. I guess Spack should be able to use only the blas or lapack part of a package that provides both. Being that such package has a blas_libs and a lapack_libs functions it seems like it should be doable.

Based on #4107 (comment) though, you have updated your packages.yaml to build without essl.

In fact I didn't, I commented out the provides(blas) call from the packages that provides lapack as a workaround but I forgot to mention that.

There is however another issue with the essl package as it doesn't provide a full BLAS but I will create a new issue to discuss that since it is unrelated to this one.

That is a different issue which is resolved by providing hints to the concretizer, I can get it to concretize with the following hints.

Ok, good to know this one can be worked around pretty easily. I hope there are plans to improve virtual dependencies a bit. It seems like a powerful feature that is still kind of buggy.

Also, I'm pretty sure you edited the mxnet package description to replace depends_on(openblas) with depends_on(blas) but I didn't see a mention of that in #4107 (comment) - FYI that's worth adding.

I didn't modify the mxnet package in any way. I noticed that it wasn't using a virtual dependency for blas but looking at the Makefile, I wasn't so sure it could work with all the other blas providers (as far as I could tell, the library name is hardcoded).

[tjfulle]

I ran in to this same problem today. Attempting to install mumps with

spack install mumps %clang 6.0.0 ^openmpi /dzyw6ng

results in

==> Error: Multiple providers found for 'blas': ['[email protected]%[email protected]~advisor~clck+daal~gdb~ilp64~inspector+ipp~itac+mkl+mpi~newdtags+rpath+shared+tbb threads=none ~vtune arch=darwin-highsierra-x86_64', '[email protected]%[email protected] build_type=RelWithDebInfo ~external-blas+lapacke+shared~xblas arch=darwin-highsierra-x86_64']

NOTE: I do not have intel-parallel-studio installed on my machine nor does spack find find any blas associated with it:

spack find blas
==> 2 installed packages
-- darwin-highsierra-x86_64 / [email protected] -----------------------
[email protected]

-- darwin-highsierra-x86_64 / [email protected] -------------------------
[email protected]

In packages.yaml I have:

 netlib-lapack:
    paths:
      [email protected]: /usr
    buildable: False
  all:
    providers:
      blas: [netlib-lapack]
      lapack: [netlib-lapack]

If I modify the installation command to:

spack install mumps %clang 6.0.0 ^openmpi @3.1.2

(replace explicit hash with version) spack creates a valid concretization for mumps and installation is successful.