spack crashing during openmpi installation due to fortran issues
@jwillenbring @tgamblin
My name is Alicia, and I'm a postdoc at SNL. I've been trying to install Trilinos via spack on my RHEL 6 machine with little success. It crashes during the openmpi installation, and reports that it's because my fortran compiler is unable to produce working executables.
[amklinv@s995692 bin]$ ./spack install trilinos
==> Installing trilinos
==> Installing superlu-dist
==> Installing openmpi
==> hwloc is already installed in /home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/hwloc-1.11.4-npuz6xtizklh6v5ckrk6wdv4wo7kfulq
==> Already downloaded /home/amklinv/spack/var/spack/stage/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol/openmpi-2.0.1.tar.bz2
==> Already staged openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol in /home/amklinv/spack/var/spack/stage/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol
==> No patches needed for openmpi
==> Building openmpi
==> WRITE LOCK: /home/amklinv/spack/opt/spack/.spack-db/prefix_lock[898272675117208261:1] [Released]
==> WRITE LOCK: /home/amklinv/spack/var/spack/stage/.lock[6797382048177427243:1] [Released]
==> Error: Command exited with status 1:
'./configure' '--prefix=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol' '--with-hwloc=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/hwloc-1.11.4-npuz6xtizklh6v5ckrk6wdv4wo7kfulq' '--enable-shared' '--enable-static' '--enable-mpi-cxx' '--without-tm' '--without-slurm' '--without-psm' '--without-psm2' '--without-mxm' '--disable-mpi-thread-multiple' '--without-pmi' '--without-sqlite3' '--enable-vt' '--without-verbs'
See build log for details:
/tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1/spack-build.out
/home/amklinv/spack/var/spack/repos/builtin/packages/openmpi/package.py:207, in install:
163 def install(self, spec, prefix):
164 config_args = ["--prefix=%s" % prefix,
165 "--with-hwloc=%s" % spec['hwloc'].prefix,
166 "--enable-shared",
167 "--enable-static"]
168
169 # for Open-MPI 2.0:, C++ bindings are disabled by default.
170 if self.spec.satisfies('@2.0:'):
171 config_args.extend(['--enable-mpi-cxx'])
172
173 if getattr(self, 'config_extra', None) is not None:
174 config_args.extend(self.config_extra)
175
176 # Variant based arguments
177 config_args.extend([
178 # Schedulers
179 '--with-tm' if '+tm' in spec else '--without-tm',
180 '--with-slurm' if '+slurm' in spec else '--without-slurm',
181 # Fabrics
182 '--with-psm' if '+psm' in spec else '--without-psm',
183 '--with-psm2' if '+psm2' in spec else '--without-psm2',
184 '--with-mxm' if '+mxm' in spec else '--without-mxm',
185 # Other options
186 ('--enable-mpi-thread-multiple' if '+thread_multiple' in spec
187 else '--disable-mpi-thread-multiple'),
188 '--with-pmi' if '+pmi' in spec else '--without-pmi',
189 '--with-sqlite3' if '+sqlite3' in spec else '--without-sqlite3',
190 '--enable-vt' if '+vt' in spec else '--disable-vt'
191 ])
192 if '+verbs' in spec:
193 path = _verbs_dir()
194 if path is not None and path not in ('/usr', '/usr/local'):
195 config_args.append('--with-%s=%s' % (self.verbs, path))
196 else:
197 config_args.append('--with-%s' % self.verbs)
198 else:
199 config_args.append('--without-%s' % self.verbs)
200
201 # TODO: use variants for this, e.g. +lanl, +llnl, etc.
202 # use this for LANL builds, but for LLNL builds, we need:
203 # "--with-platform=contrib/platform/llnl/optimized"
204 if self.version == ver("1.6.5") and '+lanl' in spec:
205 config_args.append("--with-platform=contrib/platform/lanl/tlcc2/optimized-nopanasas") # NOQA: ignore=E501
206
>> 207 configure(*config_args)
208 make()
209 make("install")
210
211 self.filter_compilers()
==> Error: Installation process had nonzero exit code : 256
My log file reads
[amklinv@s995692 bin]$ cat /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1/spack-build.out
==> './configure' '--prefix=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol' '--with-hwloc=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/hwloc-1.11.4-npuz6xtizklh6v5ckrk6wdv4wo7kfulq' '--enable-shared' '--enable-static' '--enable-mpi-cxx' '--without-tm' '--without-slurm' '--without-psm' '--without-psm2' '--without-mxm' '--disable-mpi-thread-multiple' '--without-pmi' '--without-sqlite3' '--enable-vt' '--without-verbs'
configure: WARNING: unrecognized options: --without-sqlite3, --enable-vt
============================================================================
== Configuring Open MPI
============================================================================
*** Startup tests
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for gcc... /home/amklinv/spack/lib/spack/env/gcc/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc accepts -g... yes
checking for /home/amklinv/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc understands -c and -o together... yes
checking how to run the C preprocessor... /home/amklinv/spack/lib/spack/env/gcc/gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make... GNU
checking whether make supports nested variables... yes
checking whether UID '85185' is supported by ustar format... yes
checking whether GID '29684' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/gcc... gcc3
checking whether make supports nested variables... (cached) yes
*** Checking versions
checking for repo version... v2.0.0-257-gee86e07
checking Open MPI version... 2.0.1
checking Open MPI release date... Sep 02, 2016
checking Open MPI repository version... v2.0.0-257-gee86e07
checking for repo version... v2.0.0-257-gee86e07
checking Open MPI Run-Time Environment version... 2.0.1
checking Open MPI Run-Time Environment release date... Sep 02, 2016
checking Open MPI Run-Time Environment repository version... v2.0.0-257-gee86e07
checking for repo version... v2.0.0-257-gee86e07
checking Open SHMEM version... 2.0.1
checking Open SHMEM release date... Sep 02, 2016
checking Open SHMEM repository version... v2.0.0-257-gee86e07
checking for repo version... v2.0.0-257-gee86e07
checking Open Portable Access Layer version... 2.0.1
checking Open Portable Access Layer release date... Sep 02, 2016
checking Open Portable Access Layer repository version... v2.0.0-257-gee86e07
checking for bootstrap Autoconf version... 2.69
checking for bootstrap Automake version... 1.15
checking for boostrap Libtool version... 2.4.6
*** Initialization, setup
configure: builddir: /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1
configure: srcdir: /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1
installing to directory "/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol"
*** OPAL Configuration options
checking if want to run code coverage... no
checking if want to compile with branch probabilities... no
checking if want to debug memory usage... no
checking if want to profile memory usage... no
checking if want developer-level compiler pickyness... no
checking if want developer-level debugging code... no
checking if want to developer-level timing framework... no
checking if want to install project-internal header files... no
checking if want pretty-print stacktrace... yes
checking if want pty support... yes
checking if want weak symbol support... yes
checking if want dlopen support... yes
checking if want heterogeneous support... no
checking if word-sized integers must be word-size aligned... no
checking if want IPv6 support... no
checking if want package/brand string... Open MPI [email protected] Distribution
checking if want ident string... 2.0.1
checking if want to use an alternative checksum algo for messages... no
checking maximum length of processor name... 256
checking maximum length of error string... 256
checking maximum length of object name... 64
checking maximum length of info key... 36
checking maximum length of info val... 256
checking maximum length of port name... 1024
checking maximum length of datarep string... 128
checking if want getpwuid support... yes
checking __NetBSD__... no
checking __FreeBSD__... no
checking __OpenBSD__... no
checking __DragonFly__... no
checking __386BSD__... no
checking __bsdi__... no
checking __APPLE__... no
checking __linux__... yes
checking __sun__... no
checking __sun... no
checking netdb.h usability... yes
checking netdb.h presence... yes
checking for netdb.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking netinet/tcp.h usability... yes
checking netinet/tcp.h presence... yes
checking for netinet/tcp.h... yes
checking for struct sockaddr_in... yes
checking if --with-cuda is set... not set (--with-cuda=)
checking whether CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is declared... no
checking whether cuPointerGetAttributes is declared... no
checking if have cuda support... no
*** ORTE Configuration options
checking if want orterun "--prefix" behavior to be enabled by default... no
checking if want orte static ports... yes
*** OMPI Configuration options
checking if want compile-time warnings inside of mpi.h... yes
checking if want sparse process groups... no
checking if want peruse support... no
checking if want Fortran MPI bindings... (try)
checking if want C++ bindings... yes
checking if want MPI::SEEK_SET support... yes
checking if want run-time MPI parameter checking... runtime
checking which 'use mpi_f08' implementation to use... regular (no subarray support)
*** OSHMEM Configuration options
checking if want oshmem... yes
checking if want SGI/Quadrics compatibility mode... yes
checking if want OSHMEM API parameter checking... always
checking for on_exit... yes
checking if want pshmem... yes
checking if want to build OSHMEM fortran bindings... yes
checking if want wrapper compiler rpath support... yes
============================================================================
== Compiler and preprocessor tests
============================================================================
*** C compiler and preprocessor
checking for gcc... (cached) /home/amklinv/spack/lib/spack/env/gcc/gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc accepts -g... (cached) yes
checking for /home/amklinv/spack/lib/spack/env/gcc/gcc option to accept ISO C89... (cached) none needed
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc understands -c and -o together... (cached) yes
checking for /home/amklinv/spack/lib/spack/env/gcc/gcc option to accept ISO C99... none needed
checking for the C compiler vendor... gnu
checking for ANSI C header files... (cached) yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports -finline-functions... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports -fno-strict-aliasing... yes
configure: WARNING: -fno-strict-aliasing has been added to CFLAGS
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports __builtin_expect... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports __builtin_prefetch... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports __builtin_clz... yes
checking for C optimization flags... -O3 -DNDEBUG -finline-functions -fno-strict-aliasing
checking for Interix environment... no
checking for C ident string support... #ident
checking for int8_t... yes
checking for uint8_t... yes
checking for int16_t... yes
checking for uint16_t... yes
checking for int32_t... yes
checking for uint32_t... yes
checking for int64_t... yes
checking for uint64_t... yes
checking for int128_t... no
checking for __int128... yes
checking for uint128_t... no
checking for long long... yes
checking for __float128... yes
checking for long double... yes
checking complex.h usability... yes
checking complex.h presence... yes
checking for complex.h... yes
checking for float _Complex... yes
checking for double _Complex... yes
checking for long double _Complex... yes
checking for intptr_t... yes
checking for uintptr_t... yes
checking for mode_t... yes
checking for ssize_t... yes
checking for ptrdiff_t... yes
checking size of char... 1
checking size of short... 2
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking size of float... 4
checking size of double... 8
checking size of long double... 16
checking size of __float128... 16
checking size of float _Complex... 8
checking size of double _Complex... 16
checking size of long double _Complex... 32
checking size of void *... 8
checking size of size_t... 8
checking size of ssize_t... 8
checking size of ptrdiff_t... 8
checking size of wchar_t... 4
checking size of pid_t... 4
checking alignment of _Bool... 1
checking alignment of int8_t... 1
checking alignment of int16_t... 2
checking alignment of int32_t... 4
checking alignment of int64_t... 8
checking alignment of char... 1
checking alignment of short... 2
checking alignment of wchar_t... 4
checking alignment of int... 4
checking alignment of long... 8
checking alignment of long long... 8
checking alignment of float... 4
checking alignment of double... 8
checking alignment of long double... 16
checking alignment of __float128... 16
checking alignment of float _Complex... 4
checking alignment of double _Complex... 8
checking alignment of long double _Complex... 16
checking alignment of void *... 8
checking alignment of size_t... 8
checking for C bool type... no
checking size of _Bool... 1
checking for weak symbol support... yes
checking for macro weak symbol support... yes
checking for functional offsetof macro... yes
*** C++ compiler and preprocessor
checking whether we are using the GNU C++ compiler... yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/g++ accepts -g... yes
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/g++... gcc3
checking how to run the C++ preprocessor... /home/amklinv/spack/lib/spack/env/gcc/g++ -E
checking for the C++ compiler vendor... gnu
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports -finline-functions... yes
configure: WARNING: -finline-functions has been added to CXXFLAGS
checking if C and C++ are link compatible... yes
checking for C++ optimization flags... -O3 -DNDEBUG -finline-functions
checking size of bool... 1
checking alignment of bool... 1
*** C++ compiler and preprocessor
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/g++ accepts -g... (cached) yes
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/g++... (cached) gcc3
checking how to run the C++ preprocessor... /home/amklinv/spack/lib/spack/env/gcc/g++ -E
checking if C++ compiler works... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports -finline-functions... yes
configure: WARNING: -finline-functions has been added to CXXFLAGS
checking if C and C++ are link compatible... (cached) yes
checking for C++ optimization flags... -O3 -DNDEBUG -finline-functions
checking size of bool... (cached) 1
checking alignment of bool... (cached) 1
checking if able to build the MPI C++ bindings... yes
checking for the C++ compiler vendor... (cached) gnu
checking if want C++ exception handling... no
checking for C++ template_repository_directory... not used
checking for C++ compiler template parameters... none needed
checking for C++ ident string support... #ident
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports __builtin_expect... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports __builtin_prefetch... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports const_cast<> properly... yes
*** Compiler characteristics
checking for __attribute__... yes
checking for __attribute__(aligned)... yes
checking for __attribute__(always_inline)... yes
checking for __attribute__(cold)... yes
checking for __attribute__(const)... yes
checking for __attribute__(deprecated)... yes
checking for __attribute__(deprecated_argument)... yes
checking for __attribute__(format)... yes
checking for __attribute__(format_funcptr)... yes
checking for __attribute__(hot)... yes
checking for __attribute__(malloc)... yes
checking for __attribute__(may_alias)... yes
checking for __attribute__(no_instrument_function)... yes
checking for __attribute__(noinline)... yes
checking for __attribute__(nonnull)... yes
checking for __attribute__(noreturn)... yes
checking for __attribute__(noreturn_funcptr)... yes
checking for __attribute__(packed)... yes
checking for __attribute__(pure)... yes
checking for __attribute__(sentinel)... yes
checking for __attribute__(unused)... yes
checking for __attribute__(visibility)... yes
checking for __attribute__(warn_unused_result)... yes
checking for __attribute__(weak_alias)... yes
checking for __attribute__(destructor)... yes
checking for compiler familyid... 1
checking for compiler familyname... GNU
checking for compiler version... 328448
checking for compiler version_str... 5.3.0
*** Java compiler
checking OSX locations... not found
checking Linux locations... not found
checking Solaris locations... not found
checking Java support available... no
*** Java MPI bindings
checking if want Java bindings... no
*** OpenSHMEM profiling
checking if pshmem will be enabled... yes (weak symbols supported)
*** Assembler
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/gcc... gcc3
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking for fgrep... /bin/grep -F
checking if .proc/endp is needed... no
checking directive for setting text section... .text
checking directive for exporting symbols... .globl
checking for objdump... objdump
checking if .note.GNU-stack is needed... yes
checking suffix for labels... :
checking prefix for global symbol labels...
checking prefix for lsym labels... .L
checking prefix for function in .type... @
checking if .size is needed... yes
checking if .align directive takes logarithmic value... no
checking if processor supports x86_64 16-byte compare-and-exchange... yes
checking if compiler correctly handles volatile 128bits... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports GCC inline assembly... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports DEC inline assembly... no
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports XLC inline assembly... no
checking for assembly format... default-.text-.globl-:--.L-@-1-0-1-1-1
checking for assembly architecture... AMD64
checking for builtin atomics... BUILTIN_NO
checking for perl... perl
checking for pre-built assembly file... yes (atomic-amd64-linux.s)
checking for atomic assembly filename... atomic-amd64-linux.s
*** Fortran compiler
checking whether we are using the GNU Fortran compiler... yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/gfortran accepts -g... yes
configure: WARNING: Open MPI now ignores the F77 and FFLAGS environment variables; only the FC and FCFLAGS environment variables are used.
checking whether ln -s works... yes
checking if Fortran compiler works... no
**********************************************************************
* It appears that your Fortran compiler is unable to produce working
* executables. A simple test application failed to properly
* execute. Note that this is likely not a problem with Open MPI,
* but a problem with the local compiler installation. More
* information (including exactly what command was given to the
* compiler and what error resulted when the command was executed) is
* available in the config.log file in the Open MPI build directory.
**********************************************************************
configure: error: Could not run a simple Fortran program. Aborting.
I would appreciate any help you could provide in resolving this issue and would be happy to provide any additional information that might be helpful.
amklinv
All 39 comments
could you please copy-paste your ~/compilers.yaml file to see what is there for f77 and fc. You can also check if the fortran compiler found by Spack can create executables outside of Spack.
davydden
on 17 Oct 2016
[amklinv@s995692 ~]$ cat .spack/compilers.yaml
compilers:
- compiler:
modules: []
operating_system: redhat6
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
spec: [email protected]
- compiler:
modules: [sems-gcc/5.3.0]
operating_system: redhat6
paths:
cc: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/gcc
cxx: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/g++
f77: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/gfortran
fc: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/gfortran
spec: [email protected]
- compiler:
modules: [sems-gcc/5.1.0]
operating_system: redhat6
paths:
cc: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.1.0/base/bin/gcc
cxx: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.1.0/base/bin/g++
f77: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.1.0/base/bin/gfortran
fc: /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.1.0/base/bin/gfortran
spec: [email protected]
My fortran compiler is able to create the fortran 90 hello world executable.
[amklinv@s995692 fortran]$ cat hello.f90
program hello
print *, "Hello World!"
end program hello
[amklinv@s995692 fortran]$ which gfortran
/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/gfortran
[amklinv@s995692 fortran]$ gfortran -o hello hello.f90
[amklinv@s995692 fortran]$ ./hello
Hello World!
all that looks good. At this points I don't have anything else to advice but to dig into config.log in the build directory to see why OpenMPI thinks that Fortran compilers are not functional...
davydden
on 17 Oct 2016
Here's the configuration log (renamed because github got grumpy about a .log file):
config.log.txt
Relevant chunk:
configure:32866: checking whether /home/amklinv/spack/lib/spack/env/gcc/gfortran accepts -g
configure:32877: /home/amklinv/spack/lib/spack/env/gcc/gfortran -c -g conftest.f >&5
configure:32877: $? = 0
configure:32885: result: yes
configure:32983: WARNING: Open MPI now ignores the F77 and FFLAGS environment variables; only the FC and FCFLAGS environment variables are used.
configure:32993: checking whether ln -s works
configure:32997: result: yes
configure:33181: checking if Fortran compiler works
configure:33210: /home/amklinv/spack/lib/spack/env/gcc/gfortran -o conftest conftest.f >&5
configure:33210: $? = 0
configure:33210: ./conftest
./conftest: error while loading shared libraries: libquadmath.so.0: cannot open shared object file: No such file or directory
configure:33210: $? = 127
configure: program exited with status 127
configure: failed program was:
| program main
|
| end
configure:33226: result: no
configure:33240: error: Could not run a simple Fortran program. Aborting.
It didn't find the gcc quad-precision math library, which is in /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/lib64
amklinv
on 17 Oct 2016
@amklinv the error is
./conftest: error while loading shared libraries: libquadmath.so.0: cannot open shared object file: No such file or directory This may be related to https://www.centos.org/forums/viewtopic.php?t=3237. You need to check that this library exists, since it is not on your system by default.
Rombur
on 17 Oct 2016
This library seems to exist on my machine and is in my LD_LIBRARY_PATH:
LD_LIBRARY_PATH="/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/lib64:/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/lib:/projects/sems/install/rhel6-x86_64/sems/compiler/python/2.7.9/lib"
[amklinv@s995692 lib64]$ pwd
/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/lib64
[amklinv@s995692 lib64]$ ls -lh libquadmath.*
-rw-rw-r-- 1 nobody nobody 1.9M Oct 16 15:24 libquadmath.a
-rwxrwxr-x 1 nobody nobody 1.1K Oct 16 15:24 libquadmath.la
lrwxrwxrwx 1 nobody nobody 20 May 13 12:58 libquadmath.so -> libquadmath.so.0.0.0
lrwxrwxrwx 1 nobody nobody 20 May 13 12:58 libquadmath.so.0 -> libquadmath.so.0.0.0
-rwxrwxr-x 1 nobody nobody 1.1M Oct 16 15:24 libquadmath.so.0.0.0
@amklinv :
Spack by default sanitises the environment, so those LD_LIBRARY_PATH won't be there when you build with Spack. IIRC there is a way to add extra LD_LIBRARY_PATH, maybe @tgamblin or @adamjstewart can comment on this.
There is also this PR https://github.com/LLNL/spack/pull/943 which would solve your problem.
davydden
on 17 Oct 2016
@amklinv: If the problem is as @davydden suggests, you can tell Spack NOT to sanitize your environment by using spack install --dirty. This is a common problem on systems that require modules to be loaded for compilers to work. You probably need this for more than just OpenMPI.
If you want to skip installing OpenMPI altogether (might save you some time) you can look at the section of the manual on external packages and just point Spack to a system OpenMPI install.
tgamblin
on 17 Oct 2016
spack install --dirty trilinos
ought to do the trick. The --dirty flag prevents Spack from sanitizing the build environment. Obviously this isn't ideal since you'll need to do it with every installation. I can't wait for #943 to be merged 馃槂
Edit: @tgamblin, you beat me to it.
adamjstewart
on 17 Oct 2016
p.s. if you have modules, then there was a way to add modules for compilers, i think there is something on that here http://spack.readthedocs.io/en/latest/configuration.html
davydden
on 17 Oct 2016
i just realised that you already set modules: [sems-gcc/5.3.0]. @tgamblin @adamjstewart : should not this already resolve the issue?
davydden
on 17 Oct 2016
@davydden: it should, unless there's something else missing from the environment. If spack install --dirty fixes this we should look at why the module setting doesn't.
tgamblin
on 17 Oct 2016
spack install --dirty seems to fix this. Thanks! I'm now stuck on the download of gmp, which I assume is related to the great Sandia firewall...so I think I have to resolve that one on my own.
==> Installing gmp
==> m4 is already installed in /home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/m4-1.4.17-vmfebizhu4v7oceybq4p2prxp5rydvxd
==> Trying to fetch from file:///home/amklinv/spack/var/spack/cache/gmp/gmp-6.1.1.tar.bz2
curl: (37) Couldn't open file /home/amklinv/spack/var/spack/cache/gmp/gmp-6.1.1.tar.bz2
==> Fetching from file:///home/amklinv/spack/var/spack/cache/gmp/gmp-6.1.1.tar.bz2 failed.
==> Trying to fetch from https://gmplib.org/download/gmp/gmp-6.1.1.tar.bz2
curl: (35) SSL connect error
==> Fetching from https://gmplib.org/download/gmp/gmp-6.1.1.tar.bz2 failed.
==> Error: All fetchers failed for gmp-6.1.1-v6p627gesea5lgthrj3rpl7ht4a2wmru
==> Error: Installation process had nonzero exit code : 256
[amklinv@s995692 bin]$ ls /home/amklinv/spack/var/spack/cache/gmp
ls: cannot access /home/amklinv/spack/var/spack/cache/gmp: No such file or directory
If this is a RHEL6 system, the system curl may be too old to negotiate with gmplib.org. You might be able to fix this with spack install curl && spack load curl
tgamblin
on 17 Oct 2016
Otherwise if there are proxy options needed, we can try to figure out how to shoehorn them into curl. FYI, the curl invocation is in lib/spack/spack/fetch_strategy.py, in URLFetchStrategy. If you need to tweak that, we can use whatever you come up with to add some options for adjusting proxy settings.
tgamblin
on 17 Oct 2016
spack install curl and module load curl-7.50.3-gcc-5.3.0-7wdmk2t worked for me. (No clue why spack load curl didn't.)
I now crash during the openblas build, so that's progress! Would you like the configure and build logs for openblas?
amklinv
on 17 Oct 2016
spack install curl and module load curl-7.50.3-gcc-5.3.0-7wdmk2t worked for me. (No clue why spack load curl didn't.)
You need to do spack module refresh first then you can use spack load
Rombur
on 17 Oct 2016
@tgamblin @davydden :
i just realised that you already set modules: [sems-gcc/5.3.0]. @tgamblin @adamjstewart : should not this already resolve the issue?
@davydden: it should, unless there's something else missing from the environment. If spack install --dirty fixes this we should look at why the module setting doesn't.
I am reading this through and would like to clarify about modules: if I specify modules in compilers.yaml, does all env variables from those modules will be set and preserved during build? Or this also requires --dirty ?
pramodk
on 17 Oct 2016
@pramodk: if you specify modules in the compilers.yaml file, they should be imported into the build environment even without --dirty. Spack has (as of the Cray support) the ability to load modules just in the build environment.
@mamelara @becker33: do you know why that might not be working? This is on a Linux machine -- did you test there?
tgamblin
on 17 Oct 2016
Openblas will be using fortran as well, do you install with "dirty" flag?
davydden
on 18 Oct 2016
Openblas will be using fortran as well, do you install with "dirty" flag?
Yes.
You need to do
spack module refreshfirst then you can usespack load
Thanks!
amklinv
on 18 Oct 2016
@tgamblin : thanks for information! In my testing it's not working as expected. Here is detailed example (x86_redhat_cluster):
I have gcc module which looks like this:
module show gcc/4.9.0
-------------------------------------------------------------------
/home/modulefiles/gcc/4.9.0:
module-whatis Loads gcc 4.9.0 to your environment variables
module load mpc/1.0.2
module load mpfr/3.1.2
module load gmp/5.1.3
prepend-path PATH /home//tools/gcc/gcc-4.9.0/install/bin
prepend-path LD_LIBRARY_PATH /home//tools/gcc/gcc-4.9.0/install/lib64
prepend-path LD_RUN_PATH /home//tools/gcc/gcc-4.9.0/install/lib64
prepend-path CPATH /home//tools/gcc/gcc-4.9.0/install/include
prepend-path SEARCH_DIR /home//tools/gcc/gcc-4.9.0/install/lib64
If I use that above compiler directly (without loading module), I get following error:
/home//tools/gcc/gcc-4.9.0/install/bin/gcc example.c
/home/tools/gcc/gcc-4.9.0/install/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.0/cc1: error while loading shared libraries: libmpc.so.3: cannot open shared object file: No such file or directory
So LD_LIBRARY_PATH is not set for mpc and hence the error which is expected.
Now I add entry into compilers.yaml as:
- compiler:
modules: [gcc/4.9.0, mpc/1.0.2, mpfr/3.1.2, gmp/5.1.3]
operating_system: redhat6
paths:
cc: /home/tools/gcc/gcc-4.9.0/install/bin/gcc
cxx: /home/tools/gcc/gcc-4.9.0/install/bin/g++
f77: /home/tools/gcc/gcc-4.9.0/install/bin/gfortran
fc: /home/tools/gcc/gcc-4.9.0/install/bin/gfortran
spec: [email protected]
(I also tried adding only gcc/4.9.0).
If I try to build a package, I see following error in spack-build.out:
$ spack install -v testpkg %[email protected]
/home/tools/gcc/gcc-4.9.0/install/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.0/cc1:
error while loading shared libraries: libmpc.so.3: cannot open shared
object file: No such file or directory
I haven't loaded any module or set any path yet. But if I just run with --dirty flag:
spack install -v --dirty testpkg %[email protected]
Now the package build successfully! My understanding was that the --dirty flag will be only useful if I load any modules or set any env variables.
If I check spack-build.env in above two cases, LD_LIBRARY_PATH is set properly / preserved if I use --dirty flag (even without loading any module or setting path explicitly).
(let me know if I should move this discussion to separate issue)
pramodk
on 18 Oct 2016
@tgamblin Just checked on genepool. It's able to load the gcc compiler module. I even unloaded module gcc/4.6.3 prior to installing with spack. Could it be that we set up the compiler environment before the build environment (which gets rid of LD_LIBRARY_PATH unless --dirty is used)?
mamelara
on 18 Oct 2016
The compiler wrappers themselves delete LD_LIBRARY_PATH during the final stage. That's probably the source of the problem.
adamjstewart
on 18 Oct 2016
@amklinv: @Rombur
You need to do spack module refresh first then you can use spack load
That, or maybe @amklinv didn't source share/spack/setup-env.sh
tgamblin
on 18 Oct 2016
Would you like the configure and build logs for openblas?
@amklinv: sure
tgamblin
on 18 Oct 2016
amklinv
on 18 Oct 2016
openblas errors are
../kernel/x86_64/dgemm_kernel_4x8_haswell.S: Assembler messages:
../kernel/x86_64/dgemm_kernel_4x8_haswell.S:1759: Error: no such instruction: `vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x8_haswell.S:1759: Error: no such instruction: `vpermpd $ 0x1b,%ymm0,%ymm0'
and alike.
Unless someone here knows what to do, i would open an issue upstream https://github.com/xianyi/OpenBLAS/issues .
davydden
on 18 Oct 2016
@amklinv : see https://github.com/xianyi/OpenBLAS/issues/493 , it looks similar, try updating binutils.
davydden
on 18 Oct 2016
I updated my binutils using spack. Now it crashes during the netlib-scalapack installation. It looks like it's still failing to find the libraries in my LD_LIBRARY_PATH (even though I'm using the dirty flag).
amklinv
on 18 Oct 2016
i think this time you face some other issues:
../../lib/libscalapack.so: undefined reference to `_gfortran_transfer_integer_write@GFORTRAN_1.4'
../../lib/libscalapack.so: undefined reference to `_gfortran_transfer_character_write@GFORTRAN_1.4'
../../lib/libscalapack.so: undefined reference to `_gfortran_transfer_real_write@GFORTRAN_1.4'
../../lib/libscalapack.so: undefined reference to `_gfortran_transfer_complex_write@GFORTRAN_1.4'
I think for whatever reason -lgfortran is missing in linking line. I would try submitting this as a bug to Scalapack forum.
davydden
on 20 Oct 2016
@amklinv: I think your OpenMPI issue should be fixed by #2072. Can you pull and try again?
tgamblin
on 22 Oct 2016
@amklinv: I mean your LD_LIBRARY_PATH issue.
tgamblin
on 22 Oct 2016
I pulled again and am now seeing an issue in the TBB installation.
[amklinv@s995692 bin]$ cat /tmp/amklinv/spack-stage/spack-stage-fxPZo5/tbb44_20160413oss/spack-build.out
==> 'make' '-j48' 'compiler=gcc'
Created ./build/linux_intel64_gcc_cc5.3.0_libc2.12_kernel2.6.32_release and ..._debug directories
make -C "./build/linux_intel64_gcc_cc5.3.0_libc2.12_kernel2.6.32_debug" -r -f ../../build/Makefile.tbb cfg=debug
make[1]: Entering directory `/tmp/amklinv/spack-stage/spack-stage-fxPZo5/tbb44_20160413oss/build/linux_intel64_gcc_cc5.3.0_libc2.12_kernel2.6.32_debug'
../../build/Makefile.tbb:31: CONFIG: cfg=debug arch=intel64 compiler=gcc target=linux runtime=cc5.3.0_libc2.12_kernel2.6.32
/home/amklinv/spack/lib/spack/env/gcc/g++ -o x86_rtm_rw_mutex.o -c -MMD -DTBB_USE_DEBUG -DDO_ITT_NOTIFY -g -O0 -DUSE_PTHREAD -m64 -mrtm -fPIC -D__TBB_BUILD=1 -Wall -Wno-parentheses -Wno-non-virtual-dtor -I../../src -I../../src/rml/include -I../../include ../../src/tbb/x86_rtm_rw_mutex.cpp
/tmp/ccuwkgaO.s: Assembler messages:
/tmp/ccuwkgaO.s:672: Error: no such instruction: `xtest'
/tmp/ccuwkgaO.s:691: Error: no such instruction: `xend'
/tmp/ccuwkgaO.s:859: Error: no such instruction: `xbegin .L58'
/tmp/ccuwkgaO.s:875: Error: no such instruction: `xabort $255'
/tmp/ccuwkgaO.s:1037: Error: no such instruction: `xbegin .L76'
/tmp/ccuwkgaO.s:1055: Error: no such instruction: `xabort $255'
make[1]: *** [x86_rtm_rw_mutex.o] Error 1
make[1]: Leaving directory `/tmp/amklinv/spack-stage/spack-stage-fxPZo5/tbb44_20160413oss/build/linux_intel64_gcc_cc5.3.0_libc2.12_kernel2.6.32_debug'
make: *** [tbb] Error 2
is that with the intel compiler?
tgamblin
on 27 Oct 2016
oh never mind -- I missed the g++ line.
tgamblin
on 27 Oct 2016
This look similar to this issue. Is your system really old? You may need an updated version of binutils to build things... I would like to handle that better, but right now we can't detect this kind of thing.
tgamblin
on 27 Oct 2016
I had spack install an updated version of binutils, and I thought I loaded the module correctly, but I guess I didn't. Loading the binutils module fixed the issue, and I have a spack install of Trilinos now. Thanks all!
amklinv
on 27 Oct 2016
Great!
It'd be nice if we could figure out better how to include binutils as a dependency! I doubt you're the only one who would have this issue.
Keep filing issues if you have more problems!
tgamblin
on 27 Oct 2016
Related issues
DavidPoliakoff
路
63Comments
davydden
路
36Comments
citibeth
路
72Comments
rajeeja
路
39Comments
adamjstewart
路
48Comments
Most helpful comment
I had spack install an updated version of binutils, and I thought I loaded the module correctly, but I guess I didn't. Loading the binutils module fixed the issue, and I have a spack install of Trilinos now. Thanks all!