Spack: Installation error: Python can't find _iconv symbol

Created on 7 Jan 2020 · 32Comments · Source: spack/spack

I have been trying to build packages with spack on mac os Mojave for awhile now and I have been struggling. I've simplified my software stack as much as possible to reproduce one of the issues I keep seeing.

I created an environment (called nexus, here) with the following commands:

$ spack env create nexus nexus.yaml

Where nexus.yaml is:

spack:
  specs:
    - python

From here, I activate, concretize and install:

$ spack env activate nexus
$ spack concretize
==> Warning: [email protected] cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> Warning: [email protected] cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> Concretized python
 -   hpjs6nz  [email protected]%[email protected]+bz2+ctypes+dbm+lzma~nis~optimizations patches=210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 +pic+pyexpat+pythoncmd+readline~shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=darwin-mojave-x86_64
 -   ysmjykg      ^[email protected]%[email protected]+shared arch=darwin-mojave-x86_64
 -   yn2yzi3          ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   jg6ekuh              ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   l4gxq46      ^[email protected]%[email protected]~libbsd arch=darwin-mojave-x86_64
 -   3t4dwsv      ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   h2bjtz2          ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   zbgxvxi              ^[email protected]%[email protected]~symlinks~termlib arch=darwin-mojave-x86_64
 -   f3libie                  ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   msyeqht      ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-mojave-x86_64
 -   w354vrb          ^[email protected]%[email protected]~python arch=darwin-mojave-x86_64
 -   b7n722h              ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   xj36ejk              ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-mojave-x86_64
 -   2e6sifw          ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   7yg7qah      ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   ugvknwc      ^[email protected]%[email protected]+systemcerts arch=darwin-mojave-x86_64
 -   l6fc2zw          ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-mojave-x86_64
 -   hsmq6ou      ^[email protected]%[email protected]~column_metadata+fts~functions~rtree arch=darwin-mojave-x86_64

==> Updating view at /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view

The install process reported no errors whatsoever. Ran for awhile and then afterwards, python executable is active and available, but spack itself is broken:

```Traceback (most recent call last):
File "/Users/corey.adams/spack/spack/bin/spack", line 63, in
import spack.main # noqa
File "/Users/corey.adams/spack/spack/lib/spack/spack/main.py", line 29, in
import spack.architecture
File "/Users/corey.adams/spack/spack/lib/spack/spack/architecture.py", line 69, in
import spack.compiler
File "/Users/corey.adams/spack/spack/lib/spack/spack/compiler.py", line 19, in
import spack.spec
File "/Users/corey.adams/spack/spack/lib/spack/spack/spec.py", line 105, in
import spack.repo
File "/Users/corey.adams/spack/spack/lib/spack/spack/repo.py", line 36, in
import spack.caches
File "/Users/corey.adams/spack/spack/lib/spack/spack/caches.py", line 15, in
import spack.fetch_strategy
File "/Users/corey.adams/spack/spack/lib/spack/spack/fetch_strategy.py", line 42, in
import spack.util.web as web_util
File "/Users/corey.adams/spack/spack/lib/spack/spack/util/web.py", line 18, in
from six.moves.urllib.request import urlopen, Request
File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 92, in __get__
result = self._resolve()
File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 160, in _resolve
module = _import_module(self.mod)
File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 82, in _import_module
__import__(name)
File "/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/urllib/request.py", line 2583, in
from _scproxy import _get_proxy_settings, _get_proxies
ImportError: dlopen(/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _iconv
Referenced from: /usr/lib/libcups.2.dylib
Expected in: /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib
in /usr/lib/libcups.2.dylib

I am using clang11:

$ spack config get compilers
compilers:

compiler:
spec: [email protected]
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77:
fc:
flags: {}
operating_system: mojave
target: x86_64
modules: []
environment: {}
extra_rpaths: []

I note a few things here:
The library that causes the issue doesn't directly link against `libcups`:

$ otool -L /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so
/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so:
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1061.40.2)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1673.126.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

I am not enough of a linking expert to track down where the dependency is coming from.  I also note that the library it finds for libcups that spack has build does *not* link to a library that has `_iconv` symbols defined:

corey.adams@cadams2 : ~/NEXT
$ otool -L /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib
/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib:
/Users/corey.adams/spack/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

corey.adams@cadams2 : ~/NEXT
$ nm /Users/corey.adams/spack/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/lib/libiconv.2.dylib | grep _iconv
0000000000003360 T _iconv_canonicalize

However, the library available at system level does:

$ otool -L /usr/lib/libcups.2.dylib
/usr/lib/libcups.2.dylib:
/usr/lib/libcups.2.dylib (compatibility version 2.0.0, current version 2.13.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
/System/Library/Frameworks/GSS.framework/Versions/A/GSS (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1575.17.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 963.255.3)
/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.255.3)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

corey.adams@cadams2 : ~/NEXT
$ nm /usr/lib/libiconv.2.dylib | grep _iconv
0000000000002360 T _iconv
00000000000f0da0 s _iconv_2VersionNumber
00000000000f0d70 s _iconv_2VersionString
000000000000267a T _iconv_canonicalize
0000000000002382 T _iconv_close
0000000000001049 T _iconv_open
000000000000238f T _iconvctl
0000000000002488 T _iconvlist
```

I don't have anything special on LD_LIBRARY_PATH - it is unset before spack env activate.

Somehow, spack seems to be building a package that links against the system version of libcups (which in turn links to system iconv) while at run time it is picking up it's own libcups (and it's own libiconv) which is missing symbols that it needs.

This is macOS mojave.

Thoughts?

build-error environments macOS

Source

coreyjadams

All 32 comments

That's odd, I'm also using Spack Environments on macOS (Catalina) and using my Spack-installed Python to run Spack. I haven't encountered any issues like this.

@gartung is our macOS linking expert, maybe he has some ideas on how to debug this.

adamjstewart on 7 Jan 2020

@gartung any thoughts? spack on mac has been killing me, I'd really love to get this working.

coreyjadams on 7 Jan 2020

@chissg Have you run into this in the ups environment?

gartung on 7 Jan 2020

I googled the partial error messages and got this
https://github.com/Roche/pyreadstat/issues/2
which suggested seeing what libraries python is loading with

DYLD_PRINT_LIBRARIES=YES python -c "from _scproxy import _get_proxy_settings, _get_proxies"

gartung on 7 Jan 2020

This is another one
https://github.com/openai/universe/issues/139
It suggests that you need to check the scons build to see where it picked up libintl from.

gartung on 7 Jan 2020

It suggests that you need to check the scons build to see where it picked up libintl from.

This surprises me, as Spack doesn't depend on SCons, so I would be surprised to find that is causes your error. If you remove SCons from your environment, can you no longer reproduce the error?

adamjstewart on 7 Jan 2020

The very first thing spack installs in that environment is libiconv

mac-133717:spack gartung$ spack install
==> Installing environment nexus
==> Installing libiconv
==> Searching for binary cache of libiconv
==> No binary for libiconv found: installing from source
==> Fetching https://ftp.gnu.org/gnu/libiconv/libiconv-1.16.tar.gz
######################################################################## 100.0%
==> Staging archive: /var/folders/sb/_59sd1t97k149f3mrk_hqhbr0000gp/T/gartung/spack-stage/spack-stage-libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/libiconv-1.16.tar.gz
==> Created stage in /var/folders/sb/_59sd1t97k149f3mrk_hqhbr0000gp/T/gartung/spack-stage/spack-stage-libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp
==> No patches needed for libiconv
==> Building libiconv [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'

gartung on 7 Jan 2020

Because diffutils depends on it

 -   hpjs6nz  [email protected]%[email protected]+bz2+ctypes+dbm+lzma~nis~optimizations patches=210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 +pic+pyexpat+pythoncmd+readline~shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=darwin-mojave-x86_64
 -   ysmjykg      ^[email protected]%[email protected]+shared arch=darwin-mojave-x86_64
 -   yn2yzi3          ^[email protected]%[email protected] arch=darwin-mojave-x86_64
 -   jg6ekuh              ^[email protected]%[email protected] arch=darwin-mojave-x86_64

gartung on 7 Jan 2020

There is no way to link diffutils without libiconv. To keep libiconv from being linked into the environment you will need to have a packages.yaml that declares diffutils or bzip2 as external picked up from the system.

gartung on 7 Jan 2020

There are a number of libraries that should be picked up from the system on macOS to avoid situations like these.

gartung on 7 Jan 2020

The error is happening when the six package tries to load urllib in this environment.

gartung on 7 Jan 2020

I tried; It didn't work.

I couldn't directly remove scons from the environment, because spack itself will not run with the environment active. I removed the environment, uninstalled scons, and then recreated the environment with only python. It reinstalled python + it's dependencies, but I get the same error.

coreyjadams on 7 Jan 2020

👍1

Awesome, so the issue has nothing to do with SCons, that simplifies things. I'm still surprised you're encountering this issue and I'm not.

adamjstewart on 7 Jan 2020

I am concretizing and building with the following packages.yaml to avoid building libiconv

packages:
  perl:
    buildable: False
    paths:
      perl: /usr
  tar:
    buildable: False
    paths:
      tar: /usr
  libiconv:
    buildable: False
    paths:
      libiconv: /usr

gartung on 7 Jan 2020

👀1

Ok, so it sounds like there is a potential solution which is to make diffutils or bzip2 into external, non-buildable packages?

Here is the dump for python's libraries: https://gist.github.com/coreyjadams/7a5b642d1a1115d729b19d01197863f6

I will try using the system packages as you have above to see if that works.

coreyjadams on 7 Jan 2020

Using the suggested packages.yaml I can at least run spack in the activated environment. However if I run
${prefix}/bin/python3 -c "from _scproxy import _get_proxy_settings, _get_proxies"
I get

dyld: unloaded: /Users/gartung/test/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/python-3.7.4-lzdvkfn3it4xzowyuvrzft2aj3zfpoqi/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/gartung/test/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/python-3.7.4-lzdvkfn3it4xzowyuvrzft2aj3zfpoqi/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _sqlite3_intarray_bind
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  Expected in: /Users/gartung/test/spack/var/spack/environments/nexus/.spack-env/view/lib/libsqlite3.dylib
 in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData

If I prepend /usr/lib to DYLD_LIBRARY_PATH in the environment with

export DYLD_LIBRARY_PATH=/usr/lib:$DYLD_LIBRARY_PATH

then the same command runs without error.

If I deactivate the environment I can also run the command without error.

The apparent function of lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so is to dlopen all of the C libraries referenced in the default python packages checking for missing symbols.

gartung on 7 Jan 2020

The suggestion found following these two link is to use DYLD_FALLBACK_LIBRARY_PATH instead of setting DYLD_LIBRARY_PATH in the environment

https://superuser.com/questions/1446292/anaconda-stopped-working-when-added-dyld-library-path-to-bashrc
https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s

gartung on 7 Jan 2020

@adamjstewart Is there a way to chose which variable is set in the environment?

gartung on 7 Jan 2020

I used the suggested packages from the system and got the same error as before, in that python itself worked but I could not run spack.

Prepending /usr/lib/ to DYLD_LIBRARY_PATH did allow me to run spack, and python still runs, but I have not yet tried anything else on top of this.

coreyjadams on 7 Jan 2020

@hartzell is gonna love this, this is just like #3955. Might also be related to #13245.

@goxberry reported the exact same problem and opened a PR to fix the issue, see #9221

The thing that still confuses me about this issue, and makes it different from #3955 and #9221, is that those issues involve (DY)LD_LIBRARY_PATH causing _system_ software to break, whereas your issue is with _Spack-installed_ software breaking. The RPATHs seem fine, so I'm not sure why you're having these issues. I don't activate my environment or use modules, so that explains why I didn't encounter the same issue with (DY)LD_LIBRARY_PATH though.

adamjstewart on 7 Jan 2020

For what it's worth, as far as I can tell all spack-installed software works. At least, I haven't yet seen it breaking even with these issues. It's spack itself that is failing once I installed these packages.

coreyjadams on 7 Jan 2020

But Spack is just a Python library, it doesn't link to anything. It seems like it's the python executable that is broken, not Spack...

adamjstewart on 7 Jan 2020

@adamjstewart do you run with SIP enabled? This prevents setting DYLD_LIBRARY_PATH to anything other than the system paths. I run with SIP disabled because our build scripts are nested and expect the environment to be set by the parent script.

gartung on 8 Jan 2020

Perhaps DYLD_LIBRARY_PATH should not be set at all by 'spack env activate'

gartung on 8 Jan 2020

I am having flashbacks to building ROOT with spack and using Xquartz for the X11 libraries. Some of the X11 libraries depend on system library that ROOT depends on but I did not declare them as external packages. ROOT happily built against the spack-built libraries but when running interactively it used DYLD_LIBRARY_PATH and would crash with missing symbol errors expected from system libraries.

gartung on 8 Jan 2020

Yes, I run with SIP enabled.

adamjstewart on 8 Jan 2020

Ah, I run with SIP disabled. Perhaps that's the difference in your system's success and mine's failure.

coreyjadams on 8 Jan 2020

Sorry for the bump, but ran into this myself today. Just for info, I have spack installed under my home area (so theoretically everything it installs is outside SIP) on a Catalina system with SIP enabled. Simplest case to reproduce is pretty much as reported:

$ spack env create foo
$ spack env activate -p foo
[foo] $ spack install cmake
...
[foo] $ cmake --version
cmake version 3.15.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[foo] $ spack install python
...
[foo] $ cmake --version
dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /Users/<ME>/Software/spack/var/spack/environments/foo/.spack-env/view/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
zsh: abort      cmake --version
[foo] $ DYLD_LIBRARY_PATH="" cmake --version
cmake version 3.15.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[foo] $

@gartung's Python test python3 -c "from _scproxy import _get_proxy_settings, _get_proxies also fails in the main environment, but works on unsetting DYLD_LIBRARY_PATH.

So the remedy for me seems, at this stage, to be unsetting DYLD_LIBRARY_PATH in the environment. I'll see if I can trace the behaviour of SIP vs DYLD_LIBRARY_PATH a bit better though.

drbenmorgan on 28 Jan 2020

👍1

Yes, environments are basically unusable on mac (without the -V flag) because the DYLD_LIBRARY_PATH setting breaks every non-spack app.

sethrj on 31 Jan 2020

Hi all, sorry to append to this issue. But I am finding identical error messages:

    from _scproxy import _get_proxy_settings, _get_proxies
ImportError: dlopen(/Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/python-3.7.6-pqvyilqugzurz6l7qxcmwoqeqo42ckgu/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/libiconv-1.16-gj3jnj46xug7rpjxwaepe6uwx35jdphz/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib

when I try to run python setup.py install on https://github.com/yt-project/yt git repo.

I tried using and not using spack environments.
I tried updating Xcode and command lines, and then re-installing spack.

But none of the fixes you mentioned in the previous messages (setting the DLYB paths), nor the one described here https://github.com/qgis/QGIS-Mac-Packager/issues/28, helped.

I was wondering if you had any other suggestions of what I could try.
Or if you think that my issue in particular has to do with yt and not spack installation of python.
Thanks

LDAmorim on 9 Mar 2020

Hi all and sorry for my late reply. I hadn't understood your suggestions here fully before.
Currently, with macOS Catalina 10.15.4 and spack installed python I still get the above mentioned issue.
But there is a simple work-around, as now I can understand you had explained above, that is the replacement:

$ cp  /usr/lib/libiconv.2.dylib spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/libiconv-1.16-s3pw3k55ma2ikm3alvpb6ocaqkvzdper/lib/

Cheers,
Diana

LDAmorim on 13 Apr 2020

Is it possible these python and gettext issues are related? #16075 The former depends on the latter and both have problems with libiconv on macOS.

Looks to me like the spack gettext packages forgets to depends_on('iconv'). It does expose a --with-libiconv-prefix and I guess it picks up the system lib along some paths and specifying the spack lib along other packages, crashing things here and there.

I just checked this on ubuntu and even there, libiconv is picked up directly since its provided luckily by a transitive gettext dependency. Will send a PR in a few minutes.