Spack: filter_compiler_wrappers() does not wok on Cray systems as advertised

Created on 30 Mar 2019  路  33Comments  路  Source: spack/spack

My understanding is filter_compiler_wrappers() can be used to change spack compilers back to system compilers - aka mpicc wrappers [or similar use - user makefiles]. And this should work from both spack env [for dependent packages] - and outsite spack [for user compilers]

This is related to #10365

Here petsc creates a user-makefile that a dependant package pflotran uses. Also petsc users who are not in spack env also use this user-makefile for compiling user applications. [aka mpicc wrapper - which internally uses system compiler]

This works for me on linux - but not Cray systems [Theta/Cori]

On cray systems - when spack attempts to install dependent package using this system compiler - the build errors out

Steps to reproduce the issue

balay@cori05:~/spack> git describe
v0.8.17-10445-ge3f00750e
balay@cori05:~/spack> spack install --dirty -j8 pflotran ^hdf5+hl

Error Message

     9     ftn -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/petsc-3.11.0-rgzfigpowxqofxo5wtu5rgt6omjoxnwz/include -I/global/u2/
           b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/hypre-2.15.1-b7sedjuhnjsi4vjw3nqhjirxeysfekvh/include -I/global/u2/b/balay/spack/opt/spack/cray-c
           nl9-haswell/intel-18.0.3.222/superlu-dist-6.1.0-jcfujt5n7tfseukklffr5jkci3j3hkfb/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3
           .222/hdf5-1.10.5-hk3pedgeh6zokqckjxnspqwa6ijaqodq/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/parmetis-4.0.3-kqtzwywdfbc
           plnrv62g7o6dcimvt623q/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/metis-5.1.0-2n3dm3mnxr2pq4ipm77l2vgesdue4rtm/include -
           I/usr/include   -I.  -o derivatives_utilities.o derivatives_utilities.F90
<snip>
     12    Error:
     13      cray-libsci/18.07.1 is not available for the INTEL compiler.
     14    /global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/petsc-3.11.0-rgzfigpowxqofxo5wtu5rgt6omjoxnwz/lib/petsc/conf/petscrules:31: recipe for
            target 'derivatives_utilities.o' failed

Information on your system

Attaching config files and build logs from NERSC/Cori

console.txt
petsc-build.env.txt
petsc-build.out.txt
petscvariables.txt
pflotran-spack-build.env.txt
pflotran-spack-build.out.txt

bug xSDK
Source
picture balay

All 33 comments

cc: @tgamblin

balay picture balay on 30 Mar 2019

compilers.yaml.txt
packages.yaml.txt

balay picture balay on 30 Mar 2019

This is the same as https://github.com/spack/spack/issues/11010 (so I'm inclined to close this in favor of continuing discussion there - feel free to reopen if this seems distinct).

https://github.com/spack/spack/issues/11010#issuecomment-476729900 includes some suggestions. For debugging purposes it would be great to have the console output when running the install with spack -d install pflotran - this will include all the module load/unload commands.

scheibelp picture scheibelp on 1 Apr 2019

Well I think this report is the primary issue with spack [or some conflict on cray] with using filter_compiler_wrappers()

11010 is a symptom of this issue. So will keep this one open.

balay picture balay on 1 Apr 2019
👍1

The final compiler invocation I see before the error in pflotran-spack-build.out (and the PR description) appears to be using the system fortran directly rather than the Spack compiler wrapper:

ftn -c -fPIC -I

So is there a reason you think this results from an error in filter_compiler_wrappers?

petsc creates a user-makefile that a dependant package pflotran uses

Is the problem that this Makefile doesn't appear to be properly updated?

scheibelp picture scheibelp on 1 Apr 2019

Sorry - for the ambiguity in my report.

The issue is not directly with filter_compiler_wrappers() modification of files [i.e setting up wrong compilers in the modified files] - but this feature - as usable from both within spack env - and out of spack env.

What I need [wrt petsc] is something that works in both spack env [for dependent packages] - and out of spack env - for users who are not in spack mode.

@tgamblin mentioned that I could use filter_compiler_wrappers() - and it should work.

  • within spack env - even if system compilers are invoked - spack traps them and invokes spack compilers. [as this this is how it works with mpich compilers wrappers - both in spack build env - to build petsc etc - and outside for users directly invoking mpich installed by spack]

This is the part that is not working on cray machines - and I need help with.

[I don't see this issue on linux machines - where it appears to work in both cases - within spack build env as well as outside for users]

balay picture balay on 2 Apr 2019

And to clarify the above wrt: ftn -c -fPIC -I I understood spack would trap ftn here and switch to spack compiler wrapper - and the compile would succeed [but it did not]

balay picture balay on 2 Apr 2019

I understood spack would trap ftn here and switch to spack compiler wrapper - and the compile would succeed [but it did not]

The ftn variable is not defined in pflotran-spack-build.env so I wouldn't expect it to redirect to a Spack compiler wrapper. Spack defines "FC" and "F90" but not "ftn" by default.

Another benefit of running the pflotran installation with -d (like spack -d install pflotran) is that in addition to the module loading/unloading you get a spack-cc-pflotran-...in.log and ...out.log which shows the input and output for each invocation of the Spack compiler wrappers. As a sanity check could you attach those?

One possible issue (not with ftn but possibly with cc/CC) is that Spack will e.g. define export CC="/global/u2/b/balay/spack/lib/spack/env/intel/icc". I'll note that many entries in the attached compilers.yaml point the environment variables to relative paths:

- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - PrgEnv-cray
    - cce/8.6.0
    operating_system: cnl9
    paths:
      cc: cc              <----   normally this is a full path
      cxx: CC
      f77: ftn
      fc: ftn
    spec: [email protected]
    target: any

Note that Spack uses the paths entries in filter_compiler_wrappers, so will replace a spack compiler wrapper path with cc, while still defining cc. So, even with wrapper filtering, cc invocations specifically could accidentally be redirected back to using the Spack wrappers.

scheibelp picture scheibelp on 2 Apr 2019

On Mon, 1 Apr 2019, Peter Scheibel wrote:

I understood spack would trap ftn here and switch to spack compiler wrapper - and the compile would succeed [but it did not]

The ftn variable is not defined in pflotran-spack-build.env

I'm not sure why you are referring to ftn as a variable.

when petsc invokes filter_compiler_wrappers(petscvariables)- it sets in this makefile correctly (previously attached):

FC = ftn

now pflotran uses this makefile so invokes a compile with ${FC} - i.e ftn

Will regenerate petscvariables - without filter_compiler_wrappers - and attach later [so that you can do a diff - to see what filter_compiler_wrappers() is changing. I think this part is correct]

so I wouldn't expect it to redirect to a Spack compiler wrapper. Spack defines "FC" and "F90" but not "ftn" by default.

I'm assuming you are referring to "ftn" as a make/env variable - but thats not the issue here.

Another benefit of running the pflotran installation with -d (like spack -d install pflotran) is that in addition to the module loading/unloading you get a spack-cc-pflotran-...in.log and ...out.log which shows the input and output for each invocation of the Spack compiler wrappers. As a sanity check could you attach those?

Ok - attaching to this reply [hopefully it comes through]. This is from spack -d install --dirty -j8 xsdk |& tee spack-build.log - with the build continuing from pflotran

One possible issue (not with ftn but possibly with cc/CC) is that Spack will e.g. define export CC="/global/u2/b/balay/spack/lib/spack/env/intel/icc".

pflotran build is not using env variables. Its using the makefile created by petsc that has compilers and compile targets setup.
[in this case - I'm using filter_compiler_wrappers() to switch the compilers listed in this makefile from spack compilers to system compilers]

I'll note that many entries in the attached compilers.yaml point the environment variables to relative paths:

- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - PrgEnv-cray
    - cce/8.6.0
    operating_system: cnl9
    paths:
      cc: cc              <----   normally this is a full path
      cxx: CC
      f77: ftn
      fc: ftn
    spec: [email protected]
    target: any

Not sure how this is related to this issue - but the above is what spack setup for me. Ok redoing setup now:

balay@cori04:~> mv .spack .spack.orig
balay@cori04:~> spack compiler find
==> Added 25 new compilers to /global/homes/b/balay/.spack/cray/compilers.yaml
    [email protected]          [email protected]  [email protected]  [email protected]  [email protected]  [email protected]  [email protected]  [email protected]  [email protected]
    [email protected]         [email protected]  [email protected]  [email protected]  [email protected]         [email protected]  [email protected]  [email protected]
    [email protected]  [email protected]  [email protected]  [email protected]  [email protected]         [email protected]  [email protected]    [email protected]
==> Compilers are defined in the following files:
    /global/homes/b/balay/.spack/cray/compilers.yaml
balay@cori04:~> grep -B14 -A1 [email protected] /global/homes/b/balay/.spack/cray/compilers.yaml
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - PrgEnv-intel
    - !!python/unicode
      'intel/18.0.3.222'
    operating_system: cnl9
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    spec: [email protected]
    target: any
balay@cori04:~>

Note that Spack uses the paths entries in filter_compiler_wrappers, so will replace a spack compiler wrapper path with cc, while still defining cc. So, even with wrapper filtering, cc invocations specifically could accidentally be redirected back to using the Spack wrappers.

balay picture balay on 2 Apr 2019

Ah - github messes up e-mail attachment :(

Reattaching..
pflotran-spack-build.out.txt

balay picture balay on 2 Apr 2019

Ok - I tried the following work-around for 'ftn' - and this works.

Here - I'm overriding system compiler set in petscvariables/makefile [i.e 'ftn'] with the spack compiler from env.

But I'm not having to do the same work-around for 'cc'. Spack is automatically handling 'cc' but not 'ftn' here? [pflotran is fortran only - but slepc uses both c (or c++) and fortran]

diff --git a/var/spack/repos/builtin/packages/pflotran/package.py b/var/spack/repos/builtin/packages/pflotran/package.py
index 5ad70f3a6..1208de1b6 100644
--- a/var/spack/repos/builtin/packages/pflotran/package.py
+++ b/var/spack/repos/builtin/packages/pflotran/package.py
@@ -31,3 +31,7 @@ class Pflotran(AutotoolsPackage):
     def parallel(self):
         return (self.spec.satisfies('@develop') or
                 self.spec.satisfies('@xsdk-0.4.0'))
+
+    def build(self, spec, prefix):
+        make('FC={0}'.format(self.spec['mpi'].mpifc),
+            'FC_LINKER={0}'.format(self.spec['mpi'].mpifc))
diff --git a/var/spack/repos/builtin/packages/slepc/package.py b/var/spack/repos/builtin/packages/slepc/package.py
index 95d5d430c..01a3b0b15 100644
--- a/var/spack/repos/builtin/packages/slepc/package.py
+++ b/var/spack/repos/builtin/packages/slepc/package.py
@@ -87,7 +87,9 @@ class Slepc(Package):

         configure('--prefix=%s' % prefix, *options)

-        make('MAKE_NP=%s' % make_jobs, parallel=False)
+        make('MAKE_NP=%s' % make_jobs,
+             'FC={0}'.format(self.spec['mpi'].mpifc),
+             parallel=False)
         if self.run_tests:
             make('test', parallel=False)
balay picture balay on 2 Apr 2019

Attaching the makefile (petscvariables) created by petsc without
filter_compiler_wrappers() action - and with filter_compiler_wrappers() action.

petscvariables-orig.txt
petscvariables-mod.txt

balay picture balay on 2 Apr 2019

Regarding https://github.com/spack/spack/issues/11070#issuecomment-479072530:

Spack is automatically handling 'cc' but not 'ftn' here? [pflotran is fortran only - but slepc uses both c (or c++) and fortran]

Spack prepends the wrapper directories for the chosen compiler (which include a cc executable) to PATH, so if you call cc it is likely that you will call the compiler wrapper. The intel wrapper directories don't define a ftn executable (although thecceones do), so callingftnwill execute whateverwhich ftn` normally gives you on your system.

As a sanity check, could you define a ftn symlink in spack/lib/spack/env/intel/ which points to ../cc (the same as exists in spack/lib/spack/env/cce/)?

scheibelp picture scheibelp on 10 Apr 2019

As a sanity check, could you define a ftn symlink in spack/lib/spack/env/intel/ which points to ../cc (the same as exists in spack/lib/spack/env/cce/)?

With this change pflotran and slepc build fine [without the above patch]

balay picture balay on 12 Apr 2019

Thanks for trying that, and for submitting the PR with the change!

However, I'd like to understand the problem a bit better: could you run the failing pflotran build, and when you get to the failure, run:

spack build-env pflotran bash
spack cd pflotran
(change directory to the folder containing "derivatives_utilities.F90")
# execute one of the commands reporting an error:
ftn -c -fPIC -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/petsc-3.11.0-rgzfigpowxqofxo5wtu5rgt6omjoxnwz/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/hypre-2.15.1-b7sedjuhnjsi4vjw3nqhjirxeysfekvh/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/superlu-dist-6.1.0-jcfujt5n7tfseukklffr5jkci3j3hkfb/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/hdf5-1.10.5-hk3pedgeh6zokqckjxnspqwa6ijaqodq/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/parmetis-4.0.3-kqtzwywdfbcplnrv62g7o6dcimvt623q/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-18.0.3.222/metis-5.1.0-2n3dm3mnxr2pq4ipm77l2vgesdue4rtm/include -I/usr/include   -I.  -o derivatives_utilities.o derivatives_utilities.F90

And post all of the terminal output that gives?

If you don't have time for that, I am setting up Cori access and am hoping to be able to investigate this more next week.

I am thinking the error message that was reported was a red herring, and I want to confirm that to improve future Spack error reporting):

cray-libsci/18.07.1 is not available for the INTEL compiler.

Since directing the build to use the Spack compiler wrapper fixed the issue, I would have expected the error message to be something about an undefined symbol (which the compiler wrapper would fix by adding a -I/-L).

scheibelp picture scheibelp on 12 Apr 2019

The build is slightly different - so copy/pasting the current broken compile command:

balay@cori05:~/spack> spack build-env pflotran@develop^hdf5+hl bash
balay@cori05:~/spack> spack cd pflotran@develop^hdf5+hl
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran> cd src/pflotran/
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ftn -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include   -I.  -o pflotran_provenance.o pflotran_provenance.F90
Error:
  cray-libsci/18.07.1 is not available for the INTEL compiler.
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> which ftn
/opt/cray/pe/craype/2.5.15/bin/ftn
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran>
balay picture balay on 13 Apr 2019

Thanks for adding that! It confirms that definitely is the error (although unfortunately that means I don't yet fully understand what is going wrong).

Do you have cray-libsci defined in packages.yaml? spack config blame packages would aggregate all the config files and display them together so you don't have to dig through all of them.

scheibelp picture scheibelp on 13 Apr 2019 
balay@cori05:~/spack> grep cray-libsci ~/.spack/packages.yaml
balay@cori05:~/spack> spack config blame packages |grep cray-libsci
balay@cori05:~/spack>

Attaching the complete list.
spack-blame-packages.txt

balay picture balay on 13 Apr 2019

Comparing the spack-env with the regular shell - the modules loaded is almost the same - except for altd and darshan
[spack-env]

  1) modules/3.2.10.6
  2) nsg/1.2.0
  3) craype-network-aries
  4) craype/2.5.15
  5) cray-libsci/18.07.1
  6) udreg/2.3.2-6.0.7.1_5.13__g5196236.ari
  7) ugni/6.0.14.0-6.0.7.1_3.13__gea11d3d.ari
  8) pmi/5.0.14
  9) dmapp/7.1.1-6.0.7.1_5.45__g5a674e0.ari
 10) gni-headers/5.0.12.0-6.0.7.1_3.11__g3b1768f.ari
 11) xpmem/2.2.15-6.0.7.1_5.11__g7549d06.ari
 12) job/2.2.3-6.0.7.1_5.43__g6c4e934.ari
 13) dvs/2.7_2.2.118-6.0.7.1_10.1__g58b37a2
 14) alps/6.6.43-6.0.7.1_5.45__ga796da32.ari
 15) rca/2.2.18-6.0.7.1_5.47__g2aa4f39.ari
 16) atp/2.1.3
 17) PrgEnv-intel/6.0.4
 18) intel/19.0.3.199
 19) craype-haswell
 20) cray-mpich/7.7.4

vs [regular-env]

  1) modules/3.2.10.6
  2) nsg/1.2.0
  3) intel/19.0.3.199
  4) craype-network-aries
  5) craype/2.5.15
  6) cray-libsci/18.07.1
  7) udreg/2.3.2-6.0.7.1_5.13__g5196236.ari
  8) ugni/6.0.14.0-6.0.7.1_3.13__gea11d3d.ari
  9) pmi/5.0.14
 10) dmapp/7.1.1-6.0.7.1_5.45__g5a674e0.ari
 11) gni-headers/5.0.12.0-6.0.7.1_3.11__g3b1768f.ari
 12) xpmem/2.2.15-6.0.7.1_5.11__g7549d06.ari
 13) job/2.2.3-6.0.7.1_5.43__g6c4e934.ari
 14) dvs/2.7_2.2.118-6.0.7.1_10.1__g58b37a2
 15) alps/6.6.43-6.0.7.1_5.45__ga796da32.ari
 16) rca/2.2.18-6.0.7.1_5.47__g2aa4f39.ari
 17) atp/2.1.3
 18) PrgEnv-intel/6.0.4
 19) craype-haswell
 20) cray-mpich/7.7.3
 21) altd/2.0
 22) darshan/3.1.4

And the same compile command works fine on the regular-env

balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ftn -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include   -I.  -o pflotran_provenance.o pflotran_provenance.F90
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran>
balay picture balay on 13 Apr 2019

In spack-env - If I change path with intel compilers upfront - the compile with ftn succeeds.

export PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/bin/intel64:$PATH

balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> echo $PATH
/global/u2/b/balay/spack/lib/spack/env/intel:/global/u2/b/balay/spack/lib/spack/env/case-insensitive:/global/u2/b/balay/spack/lib/spack/env:/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/bin:/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/bin:/opt/cray/pe/mpt/7.7.4/gni/bin:/opt/intel/compilers_and_libraries_2019.3.199/linux/bin/intel64:/opt/cray/rca/2.2.18-6.0.7.1_5.47__g2aa4f39.ari/bin:/opt/cray/alps/6.6.43-6.0.7.1_5.45__ga796da32.ari/sbin:/opt/cray/job/2.2.3-6.0.7.1_5.43__g6c4e934.ari/bin:/opt/cray/pe/craype/2.5.15/bin:/global/u2/b/balay/spack/bin:/global/u2/b/balay/spack/lib/spack/env/intel:/global/u2/b/balay/spack/lib/spack/env/case-insensitive:/global/u2/b/balay/spack/lib/spack/env:/global/homes/b/balay/spack/bin:/usr/common/software/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/syscom/nsg/sbin:/usr/syscom/nsg/bin:/opt/cray/pe/modules/3.2.10.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> export PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/bin/intel64:$PATH
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ftn -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include   -I.  -o pflotran_constants.o pflotran_constants.F90
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran>
balay picture balay on 13 Apr 2019

Here is something interesting. I'm running ftn in verbose mode - it shows the ifort command its invoking.
If I copy/paste and invoke the ifort command directly - there is no error.

balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ftn -craype-verbose -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include   -I.  -o pflotran_constants.o pflotran_constants.F90
ifort -xCORE-AVX2 -D__CRAYXC -D__CRAY_HASWELL -D__CRAYXT_COMPUTE_LINUX_TARGET -c -fPIC -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include -I. -o pflotran_constants.o pflotran_constants.F90 -I/opt/cray/pe/libsci/18.07.1/INTEL/16.0/x86_64/include -I/opt/cray/pe/mpt/7.7.4/gni/mpich-intel/16.0/include -I/opt/cray/rca/2.2.18-6.0.7.1_5.47__g2aa4f39.ari/include -I/opt/cray/alps/6.6.43-6.0.7.1_5.45__ga796da32.ari/include -I/opt/cray/xpmem/2.2.15-6.0.7.1_5.11__g7549d06.ari/include -I/opt/cray/gni-headers/5.0.12.0-6.0.7.1_3.11__g3b1768f.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14.0-6.0.7.1_3.13__gea11d3d.ari/include -I/opt/cray/udreg/2.3.2-6.0.7.1_5.13__g5196236.ari/include -I/opt/cray/wlm_detect/1.3.3-6.0.7.1_5.6__g7109084.ari/include -I/opt/cray/krca/2.2.4-6.0.7.1_5.43__g8505b97.ari/include -I/opt/cray-hss-devel/9.0.0/include 
Error:
  cray-libsci/18.07.1 is not available for the INTEL compiler.
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> which ifort
/global/u2/b/balay/spack/lib/spack/env/intel/ifort
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ifort -xCORE-AVX2 -D__CRAYXC -D__CRAY_HASWELL -D__CRAYXT_COMPUTE_LINUX_TARGET -c -fPIC -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include -I. -o pflotran_constants.o pflotran_constants.F90 -I/opt/cray/pe/libsci/18.07.1/INTEL/16.0/x86_64/include -I/opt/cray/pe/mpt/7.7.4/gni/mpich-intel/16.0/include -I/opt/cray/rca/2.2.18-6.0.7.1_5.47__g2aa4f39.ari/include -I/opt/cray/alps/6.6.43-6.0.7.1_5.45__ga796da32.ari/include -I/opt/cray/xpmem/2.2.15-6.0.7.1_5.11__g7549d06.ari/include -I/opt/cray/gni-headers/5.0.12.0-6.0.7.1_3.11__g3b1768f.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14.0-6.0.7.1_3.13__gea11d3d.ari/include -I/opt/cray/udreg/2.3.2-6.0.7.1_5.13__g5196236.ari/include -I/opt/cray/wlm_detect/1.3.3-6.0.7.1_5.6__g7109084.ari/include -I/opt/cray/krca/2.2.4-6.0.7.1_5.43__g8505b97.ari/include -I/opt/cray-hss-devel/9.0.0/include 
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran>
balay picture balay on 13 Apr 2019

If I remove cray-libsci module - the error goes away

balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ftn  -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include   -I.  -o pflotran_constants.o pflotran_constants.F90
Error:
  cray-libsci/18.07.1 is not available for the INTEL compiler.
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> module remove cray-libsci
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran> ftn  -c -fPIC     -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hypre-develop-idifcjovhx7fr43x4ocujd6ryaf2m6qc/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/superlu-dist-develop-rdwnne2jqboyhnsa5y5d6cede2urgls4/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/hdf5-1.10.5-znrt4i3qf3kj66odmxvzqvhojycylbhl/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/parmetis-4.0.3-6poytjxeyxuraqubz6fwet6hgiokaduo/include -I/global/u2/b/balay/spack/opt/spack/cray-cnl9-haswell/intel-19.0.3.199/metis-5.1.0-mo7bfaydesyrutuiznzhy6vjy7q2r3s5/include -I/usr/include   -I.  -o pflotran_constants.o pflotran_constants.F90
balay@cori05:/global/u2/b/balay/spack/var/spack/stage/pflotran-develop-z5e4e6wq6qlvxoe6praybit5ismuk347/pflotran/src/pflotran>
balay picture balay on 13 Apr 2019

Ok - maybe this is what you are looking for:

balay@cori05:~> strings /opt/cray/pe/craype/2.5.15/bin/ftn |grep 'is not available for'
 is not available for the %s
balay@cori05:~>

So for some reason - ftn is printing this message.

balay picture balay on 13 Apr 2019

That is useful for sure, I should have access to Cori by Monday to investigate that script myself.

scheibelp picture scheibelp on 13 Apr 2019

As for several of the prior messages, the short of it is that your tweaks affect the interleaving of Spack wrappers and Cray wrappers, but it is hard to tell why this results in a failure or not:

If I copy/paste and invoke the ifort command directly - there is no error.

While Spack doesn't wrap ftn by default (as noted in https://github.com/spack/spack/issues/11070#issuecomment-481882637), it does wrap ifort (when the compiler spec is intel).

So, invoking ifort directly is like invoking ftn after adding the Spack ftn wrapper path (like in https://github.com/spack/spack/issues/11070#issuecomment-482341314): In both cases, you get the command line -I/-L manipulations from Spack's compiler wrapper, and then a redirection to Cray's ftn wrapper (Spack strips its own wrappers out of PATH when it is called, so when it calls ftn, as specified in compilers.yaml, that will use the Cray ftn executable).

In spack-env - If I change path with intel compilers upfront - the compile with ftn succeeds.

In this case, when you invoke ftn, you are invoking Cray's wrapper, and then because of the PATH update, the Cray ifort executable is prioritized over Spack's ifort wrapper.

To summarize - things that work:

  • Invoking Cray-ftn -> Cray-ifort (https://github.com/spack/spack/issues/11070#issuecomment-482757102)
  • Invoking Spack-ftn -> Cray-ftn -> Cray-ifort (https://github.com/spack/spack/issues/11070#issuecomment-482341314)
  • Invoking Cray-ftn -> Spack-ifort -> Cray-ftn without cray-libsci loaded (https://github.com/spack/spack/issues/11070#issuecomment-482760549)

Things that don't:

  • Invoking Cray-ftn -> Spack-ifort -> Cray-ftn with cray-libsci loaded (what happens without any changes)
scheibelp picture scheibelp on 13 Apr 2019

Also, Spack generally unloads cray-libsci module, so it's odd that it ends up in spack build-env pflotran and that you have to unload it manually.

I didn't see a mention of that module in your packages.yaml or compilers.yaml I want to check if it gets loaded indirectly via some other module (another thing I should be able to look at on Monday).

scheibelp picture scheibelp on 13 Apr 2019

I have access to Cori so I should have more luck investigating this now.

I'm curious if when you do spack -d build-env pflotran whether there is a mention of unloading the cray-libsci module. If it does appear to be unloaded, then based on https://github.com/spack/spack/issues/11070#issuecomment-482760549 I am surprised that this fails in Spack (though I see how it might not work outside of Spack).

scheibelp picture scheibelp on 16 Apr 2019

Here it is

balay@cori05:~/spack> script
Script started, file is typescript
balay@cori05:~/spack> spack -d build-env pflotran@develop^hdf5+hl bash
<snip>
balay@cori05:~/spack> module list |& grep libsci
  5) cray-libsci/18.07.1
balay@cori05:~/spack> exit
balay@cori05:~/spack> exit
Script done, file is typescript
balay@cori05:~/spack> grep libsci typescript
==> [2019-04-15-20:38:26.422220] Unloading module: cray-libsci
==> [2019-04-15-20:38:26.450186] '/opt/cray/pe/modules/3.2.10.6/bin/modulecmd' 'python' 'unload' 'cray-libsci'
balay@cori05:~/spack> module list |& grep libsci
  5) cray-libsci/18.07.1
balay@cori05:~/spack>

Attaching the complete log from script

typescript.txt

balay picture balay on 16 Apr 2019
👍1

It turns out that while Spack unloads cray-libsci, it later loads PrgEnv-intel which itself loads cray-libsci.

scheibelp picture scheibelp on 17 Apr 2019

The combination of the following items leads pflotran build failure in Spack. None of these actions are necessarily individually wrong. Choosing how to proceed will take some thought.

  1. Spack defines cc/CC/FC as "cc"/"CC"/"ftn" for all Cray compilers, i.e. the compiler variables always point to Cray wrappers vs. directly using executables (e.g. Spack could point FC to /opt/intel/compilers_and_libraries_2019.3.199/linux/bin/intel64/ifort but chooses to point it to ftn instead).
  2. Cray wrappers can call unqualified names like "ifort"
  3. Even when Spack calls filter_compiler_wrappers in a dependency build (like https://github.com/spack/spack/commit/b8684b318c8b39ceff958ffc778da07fdaaa07a8), the dependent build will add Spack compiler wrappers to PATH: petsc updates the petscvariables file in this manner, which is used by pflotran, but during the pflotran build the Spack compiler wrappers are prioritized in PATH, which in combination with [1/2] means that if the Cray "ftn" wrapper calls "ifort" it will call Spack's "ifort" wrapper
  4. Spack tries to unload cray-libsci but currently other modules loaded for Cray builds override that (namely PrgEnv-intel)
  5. Cray's "ftn" wrapper cannot use Spack's "ifort" wrapper if cray-libsci is loaded

As an additional note for [1]: earlier I mentioned that there was an issue with full paths not being used, but even if the full path to the "ftn" wrapper is used, there would still be an issue.

scheibelp picture scheibelp on 17 Apr 2019

@scheibelp I understand why using cray ftn over spack ifort wrapper should work - and currently does not.

However this is not the mode for for c/c++ compilers. i.e cc and CC in PATH are spack compiler wrappers - not cray wrappers. So why should not ftn in PATH also be a spack wrapper - and be consistent across languages?

balay@cori05:~/spack> egrep '(CC|CXX|FC) =' opt/spack/cray-cnl9-haswell/intel-19.0.3.199/petsc-develop-m5aq2zu3x4ppuescyj4azinr4i6brbf2/lib/petsc/conf/petscvariables
PCC = cc
CC = cc
CXX = CC
FC = ftn
balay@cori05:~/spack>

Sure pflotran uses only fortran - but slepc uses both C and fortran compilers [and likely c++] - and PETSc sets up all 3 compilers.

balay@cori05:~/spack> spack build-env pflotran@devel bash balay@cori05:~/spack> which cc /global/u2/b/balay/spack/lib/spack/env/cc balay@cori05:~/spack> which CC /global/u2/b/balay/spack/lib/spack/env/case-insensitive/CC balay@cori05:~/spack> which ftn /opt/cray/pe/craype/2.5.15/bin/ftn balay@cori05:~/spack> ls -F /global/u2/b/balay/spack/lib/spack/env arm/ c89@ case-insensitive/ cce/ cpp@ f90@ fc@ intel/ nag/ xl/ c++@ c99@ cc* clang/ f77@ f95@ gcc/ ld@ pgi/ xl_r/ balay@cori05:~/spack>
So shouldn't be the correct fix here be to add ftn in lib/spack/env?

Sure - my patch adds it incorrectly at lib/spack/env/intel/ instead of lib/spack/env

balay picture balay on 17 Apr 2019

So shouldn't be the correct fix here be to add ftn in lib/spack/env?

Yes - I agree

Sure - my patch adds it incorrectly at lib/spack/env/intel/ instead of lib/spack/env

Could you update https://github.com/spack/spack/pull/11180 and I'll merge that?

scheibelp picture scheibelp on 17 Apr 2019

@scheibelp, PR is now updated with 'ftn' in lib/spack/env

balay picture balay on 17 Apr 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tgamblin picture tgamblin  路  39Comments
rajeeja picture rajeeja  路  39Comments
alfredo-gimenez picture alfredo-gimenez  路  34Comments
davydden picture davydden  路  36Comments
DavidPoliakoff picture DavidPoliakoff  路  63Comments