Dealii: Linker error 9.2.0 dmg on MacOS Catalina: argument list too long

[anne-glerum]

As I mentioned in the closed issue #10656, I'm experiencing linker errors when trying to install ASPECT. Could this be related to the very long directory names?

With the current dealii image on Catalina, after module load dealii, I try to install ASPECT and get the following linker error:
[114/114] Linking CXX executable aspect
FAILED: aspect
: && /Applications/deal.II.app/Contents/Resources/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/openmpi-3.1.6-ch3xlssbdriiio4hygk3xhnxclncjvze/bin/mpic++ + ...a very long list...
clang: error: unable to execute command: posix_spawn failed: Argument list too long
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

_Originally posted by @anne-glerum in https://github.com/dealii/dealii/issues/10656#issuecomment-712726901_

[drwells]

I think no one commented in the previous thread because few of us use the mac bundle and fewer of us use it with ASPECT (speaking for myself, at least, I don't use spack or the bundle).

posix_spawn is a low-level enough function that I suspect there is no easy fix. @luca-heltai do you have any hints?

[drwells]

I would much rather fix the mac bundle but a possible workaround is to use https://github.com/dealii/candi - does that at least get things working for you?

[anne-glerum]

Unfortunately I had compiler issues with candi that I couldn't sort out with the help of @tjhei, so I decided to try the bundle.

[drwells]

@luca-heltai even if you don't have time to fix this, could you pass along the code you have for setting up the mac package so we can take a look?

[drwells]

Timo, JP, and I are talking about this now and some possible fixes. We think we can fix the very long package names by modifying config:install_tree:projections:all. See

https://spack.readthedocs.io/en/latest/config_yaml.html

[luca-heltai]

There is nothing special to do: install the package, then call spack uninstall *, modify the parameters, and call spack install dealii again (maybe after updating spack to the last version). If this works, spack clean all, and then copy the deal.II.app into a copy of the dmg file. If you don't know/don't want to do these last steps, I can easily do them for you, once you have a working deal.II.app. Just send me a compressed copy of the spack tree.

[luca-heltai]

I'll try to do this later today.

[tjhei]

Great, thank you @luca-heltai . Can I ask you to consider adding module load dealii by default when opening the deal.II terminal? This has been difficult for many new users as things "mostly" work until you try to link ASPECT.

Finally, it would be great to have a script to generate the .dmg that is reproducible and available somewhere. But one step at a time. :-)

[luca-heltai]

The "script" is

rm -rf /Volumes/deal.II.dmg/deal.II.app/; cp -a /Applications/deal.II.dmg/deal.II.app/ /Volumes/deal.II.dmg/deal.II.app/

:)

btw, I'm having difficulties in building scalapack. It simply won't build with the latest XCode.

I did not add the module load dealii by default because it takes a few seconds to complete and I found it annoying, but sure, no problem. That can be done (once compilation of scalapack succeeds).

Did any of you succeeded in building scalapack and/or petsc without it?

[tjhei]

The "script" is

I doubt that this is it, what about creating directories, cloning spack, installing deal.II, adding the files, etc.?

I did not add the module load dealii by default because it takes a few seconds to complete and I found it annoying, but sure, no problem.

I don't see why the deal.II terminal is useful without running the module load command. Annoying that this is slow...

btw, I'm having difficulties in building scalapack.

If you want to post details, @jppelteret might be able to help.

[jppelteret]

This is my current package.yaml file. I see that I have scalapack disabled, so I think that I must have also hit difficulties with it (and ADOL-C too) :-/ I'll try to install it again to see what's going on and if there's anything that can be done within the spack ecosystem to fix it.

packages:
  all:
    compiler: [apple-clang, gcc, clang]
    providers:
      mpi: [openmpi]
      blas: [openblas]
      lapack: [openblas]
  dealii:
    version: [master]
    variants: ~adol-c+arpack+assimp~cuda~doc~examples~ginkgo+gmsh+gsl+hdf5~int64+metis+mpi+muparser~nanoflann~netcdf+oce+optflags+p4est+petsc~python~scalapack+slepc+sundials+symengine+threads+trilinos
  boost:
    version: [1.72.0]
  gcc:
    version: [9.3.0]
  llvm:
    version: [9.0.1]
  openmpi:
    version: [3.1.6]
  trilinos:
    variants: ~exodus~netcdf~pnetcdf~mumps gotype=int
  symengine:
    variants: +llvm

[jppelteret]

@luca-heltai, I guess you get errors like this, right? I see this for both scalapack 2.0.2 and 2.1.0.

/<PATH>/spack-stage-netlib-scalapack-2.1.0-mnjgrebekoi6272ut6i3zyldgorynvcp/spack-src/BLACS/SRC/sgsum2d_.c:154:7: error: implicit declaration of function 'BI_smvcopy' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      BI_smvcopy(Mpval(m), Mpval(n), A, tlda, bp->Buff);
      ^
/<PATH>/spack-stage-netlib-scalapack-2.1.0-mnjgrebekoi6272ut6i3zyldgorynvcp/spack-src/BLACS/SRC/dgsum2d_.c:154:7: error: implicit declaration of function 'BI_dmvcopy' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      BI_dmvcopy(Mpval(m), Mpval(n), A, tlda, bp->Buff);
      ^

[jppelteret]

Unbelievably, I've found a Debian build log from 2013 that has the same warning emitted. The only difference is that the -Werror flag now promotes this warning to an error. I'm going to check to see if we can somehow circumvent it.

[jppelteret]

@luca-heltai Would you like to give https://github.com/spack/spack/pull/19824 a go?

If Scalapack builds, then these lines can be amended in the packages.yaml that I shared above

  dealii:
    version: [master]
    variants: ~adol-c+arpack+assimp~cuda~doc~examples~ginkgo+gmsh+gsl+hdf5~int64+metis+mpi+muparser~nanoflann~netcdf+oce+optflags+p4est+petsc~python+scalapack+slepc+sundials+symengine+threads+trilinos
  trilinos:
    variants: ~exodus~netcdf~pnetcdf gotype=int

I disable the netcdf variants because I want to build with ADOL-C, with which it conflicts.

[luca-heltai]

I was able to compile deal.II, with this packages.yaml:

~yaml
packages:
perl:
externals:
- spec: [email protected]
prefix: /usr/
buildable: False
version: [5.18.4]
curl:
externals:
- spec: [email protected]
prefix: /usr/
buildable: False
version: [7.64.1]
trilinos:
variants: ~netcdf~exodus
dealii:
variants: ~examples~adol-c~ginkgo
~
ADOL-C compiled fine, but then there were issues when trying to build deal.II about unresolved symbols.

I'm now packaging a new image.

[luca-heltai]

Can you try if this image works for you?

https://github.com/dealii/dealii/releases/download/v9.2.0/dealii-9.2.0-v3.dmg

See the readme for a summary.log, and for the instructions. I made it in such a way that by default you don't need to use modules, but you can if you set DEAL_II_USE_MODULES (in which case, DYLD_LIBRARY_PATH and CMAKE_PREFIX_PATH are not set in the shell).

With this, in the default terminal you can also compile deal.II, and it would detect all installed libraries.

The default path names have been shortened a lot, reducing the probability of issues when linking.

@tjhei : everytime I create an image, I have to manually check several libraries for correct install names. I doubt this is easily scriptable.

With this image, I was able to compile and run step-40 by clicking on the app, and running, within the shell, cmake ..; make; mpirun -np 4 ./step-40.

[luca-heltai]

If this works, @tamiko could copy it in his mirror, and we could advertise in the mailing list.

[anne-glerum]

Ha, this is great! With this new image, I could install dealii step 40, aspect and shared libs for aspect, and aspect runs as well. Thanks a lot for the quick fix!

[luca-heltai]

Great! Happy that this worked. :)

[tjhei]

@tamiko can you update the mirror please?
I am happy to update the website, but we would need to update the pgp signature as well. Did you do that last time as well?

[tamiko]

@tjhei On it. Luca did also upload a pgp signature, so we should be set.

[tamiko]

@tjhei @luca-heltai Mirror updated.

[tamiko]

Website updated.

[jppelteret]

I have to manually check several libraries for correct install names.

Which libraries are these @luca-heltai ? I think that we got one of the culprits (suite-sparse) resolved with https://github.com/spack/spack/pull/19507, and in https://github.com/spack/spack/issues/19508 it was ascertained that trilinos, open-cascade and muparser do get embedded rpath locations. I suspect that ginkgo might still be a problem and we can fix that upstream too.

ADOL-C compiled fine, but then there were issues when trying to build deal.II about unresolved symbols.

IIRC I kept on triggering a failure with one of these configure time checks, which is strange because we got that fixed upstream and it certainly used to work for me (on my older MacOS install). But I'll try again to see if I have any more problems, and I'll try to figure out some solutions to them.

[jppelteret]

Ha, this is great! With this new image, I could install dealii step 40, aspect and shared libs for aspect, and aspect runs as well. Thanks a lot for the quick fix!

Yay! Thanks a lot @luca-heltai for fixing the package!

[luca-heltai]

Which libraries are these @luca-heltai ? I think that we got one of the culprits (suite-sparse) resolved with spack/spack#19507, and in spack/spack#19508 it was ascertained that trilinos, open-cascade and muparser do get embedded rpath locations. I suspect that ginkgo might still be a problem and we can fix that upstream too.

I have issues with

• ginkgo
• netcdf (impossible to have a compatible spec with trilinos+exodus+netcdf)
• adol-c

I think adol-c is an issue upstream, ginkgo may be a problem for us.