Trilinos: Epetra CMake doesn't find dggsvd3_ in netlib LAPACK?

[brian-kelley]

Question

@trilinos/epetra
I'm trying to a non-CUDA build of Trilinos on white with these modules.
I loaded them by running source .../cmake/std/atdm/load-env.sh gnu:

  1) cuda/9.2.88
  2) binutils/2.30.0
  3) gcc/7.2.0
  4) papi/5.5.1
  5) java/ibm/sdk/8.0.0
  6) openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
  7) pnetcdf-exo/1.9.0/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
  8) zlib/1.2.8
  9) hdf5/1.8.20/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
 10) netcdf/4.6.1/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
 11) cgns/20180521/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
 12) netlib/3.8.0/gcc/7.2.0
 13) superlu/4.3.0/gcc/7.2.0
 14) boost/1.65.1/gcc/7.2.0
 15) metis/5.0.1/gcc/7.2.0
 16) parmetis/4.0.3/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
 17) cmake/3.9.6
 18) git/2.10.1
 19) valgrind/3.12.0
 20) devpack/20180521/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88

and I used this configure script:

rm -rf CMakeFiles
rm -f CMakeCache.txt

cmake \
  -DTPL_ENABLE_MPI=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DTrilinos_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_EXAMPLES=ON \
  -DTrilinos_ENABLE_Tpetra:BOOL=ON \
  -DTrilinos_ENABLE_Epetra:BOOL=ON \
  -DTrilinos_ENABLE_Panzer:BOOL=ON \
  -DTpetra_INST_INT_INT=ON \
  -DTPL_ENABLE_Matio=OFF \
  -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
  -DBUILD_SHARED_LIBS=ON \
  -DBLAS_LIBRARY_DIRS=$BLAS_ROOT/lib \
  -DLAPACK_LIBRARY_DIRS=$LAPACK_ROOT/lib \
  -DNetcdf_INCLUDE_DIRS=$NETCDF_ROOT/include \
  -DNetcdf_LIBRARY_DIRS=$NETCDF_ROOT/lib \
  -DBoost_INCLUDE_DIRS=$BOOST_ROOT/include \
  -DBoostLib_INCLUDE_DIRS=$BOOST_ROOT/include \
  -DBoostLib_LIBRARY_DIRS=$BOOST_ROOT/lib \
  ../Trilinos

The netlib module sets the env variable LAPACK_ROOT = /home/projects/ppc64le-pwr8/netlib/3.8.0/gcc/7.2.0. Trilinos finds liblapack.a during configuration, but later Epetra says that it's not a "new" LAPACK with dggsvd3:

Processing enabled TPL: LAPACK (enabled by TeuchosNumerics, disable with -DTPL_ENABLE_LAPACK=OFF)
-- LAPACK_LIBRARY_NAMES='lapack lapack_win32'
-- Searching for libs in LAPACK_LIBRARY_DIRS='/ascldap/users/projects/ppc64le-pwr8/netlib/3.8.0/gcc/7.2.0/lib'
-- Searching for a lib in the set "lapack lapack_win32":
--   Searching for lib 'lapack' ...
--     Found lib '/ascldap/users/projects/ppc64le-pwr8/netlib/3.8.0/gcc/7.2.0/lib/liblapack.a'
-- TPL_LAPACK_LIBRARIES='/ascldap/users/projects/ppc64le-pwr8/netlib/3.8.0/gcc/7.2.0/lib/liblapack.a'
...
Processing enabled package: Epetra (Libs, Tests, Examples)
-- Looking for dggsvd3
-- Looking for dggsvd3 - not found
-- Looking for dggsvd3_
-- Looking for dggsvd3_ - not found
-- Looking for DGGSVD3
-- Looking for DGGSVD3 - not found
-- Looking for DGGSVD3_
-- Looking for DGGSVD3_ - not found
Did not find new version of lapack. dggsvd3 is not available.

But if I look at the library, it does have a symbol named "dggsvd3_":

[bmkelle@white11 lib]$ nm /ascldap/users/projects/ppc64le-pwr8/netlib/3.8.0/gcc/7.2.0/lib/liblapack.a | grep dggsvd3
dggsvd3.o:
0000000000000000 T dggsvd3_

During Epetra's build, this causes linker errors - Epetra expects LAPACK to have a symbol of a different name, "dggsvd_":

...
[ 30%] Building CXX object packages/epetra/test/Map/CMakeFiles/Epetra_Map_test.dir/cxx_main.cpp.o
../../src/libepetra.so.12.17: undefined reference to `dggsvd_'
../../src/libepetra.so.12.17: undefined reference to `sggsvd_'
collect2: error: ld returned 1 exit status
make[2]: *** [packages/epetra/test/CrsGraph/Epetra_CrsGraph_test.exe] Error 1

Could this be an Epetra cmake bug, or should I change something in my configuration?

white's login node (white11) has a system LAPACK in /usr/lib64 which does work, but the compute nodes don't have it.

[jewatkins]

I ran into the same issue on a different machine and came across #480. I was able to add -D HAVE_dggsvd3_POST=1 to override the check and force the use of dggsvd3.

[brian-kelley]

@jewatkins Thanks for the workaround, that worked.

[gahansen]

I just ran into this same issue on a Trilinos build on the "vortex" machine. I can add some more information.

It appears to me that the CMake CHECK_HAVE_EPETRA_LAPACK_GSSVD3 function in:

Trilinos/packages/epetra/cmake/CheckHaveEpetraLapackGSSVD3.cmake

is not working properly. I placed a message command in that function to print out the values of ${CMAKE_REQUIRED_LIBRARIES} that is being searched.

In my case, this is the CMake output:

Processing enabled package: Epetra (Libs)
/usr/tcetmp/packages/lapack/lapack-3.9.0-gcc-7.3.1/lib/liblapack.so;/usr/tcetmp/packages/lapack/lapack-3.9.0-gcc-7.3.1/lib/libblas.so
Did not find new version of lapack. dggsvd3 is not available.

But, I used "nm" to check liblapack.so and it returns:

[gahanse@vortex60 lib]$ nm liblapack.so | grep dggsvd3
00000000001e1ae0 T dggsvd3_

I will move forward with @jewatkins suggestion above for now. I can reproduce this issue on demand so please let me know if you would like more info.