Conda-build: 3.16.0 fails to import libarchive
Actual Behavior
We have a CI pipeline that worked with 3.15.1 -- updating to 3.16.0 crashes with the following:
Traceback (most recent call last):
File "/opt/conda/bin/conda-build", line 7, in <module>
from conda_build.cli.main_build import main
File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 18, in <module>
import conda_build.api as api
File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 22, in <module>
from conda_build.config import Config, get_or_merge_config, DEFAULT_PREFIX_LENGTH as _prefix_length
File "/opt/conda/lib/python3.6/site-packages/conda_build/config.py", line 17, in <module>
from .variants import get_default_variant
File "/opt/conda/lib/python3.6/site-packages/conda_build/variants.py", line 15, in <module>
from conda_build.utils import ensure_list, trim_empty_keys, get_logger
File "/opt/conda/lib/python3.6/site-packages/conda_build/utils.py", line 10, in <module>
import libarchive
File "/opt/conda/lib/python3.6/site-packages/libarchive/__init__.py", line 1, in <module>
from .entry import ArchiveEntry
File "/opt/conda/lib/python3.6/site-packages/libarchive/entry.py", line 6, in <module>
from . import ffi
File "/opt/conda/lib/python3.6/site-packages/libarchive/ffi.py", line 108, in <module>
errno = ffi('errno', [c_archive_p], c_int)
File "/opt/conda/lib/python3.6/site-packages/libarchive/ffi.py", line 95, in ffi
f = getattr(libarchive, 'archive_'+name)
File "/opt/conda/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
func = self.__getitem__(name)
File "/opt/conda/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /opt/conda/bin/python: undefined symbol: archive_errno
We are using the continuumio/miniconda3 docker image and we get the following dependencies:
Solving environment: ...working... done
## Package Plan ##
environment location: /opt/conda
added / updated specs:
- anaconda-client
- conda-build
- conda-verify
- nodejs
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-verify-3.1.1 | py36_0 33 KB
pyyaml-3.13 | py36h14c3975_0 178 KB
libstdcxx-ng-8.2.0 | hdf63c60_1 2.9 MB
zstd-1.3.3 | h84994c4_0 840 KB
jinja2-2.10 | py36_0 184 KB
glob2-0.6 | py36_1 17 KB
certifi-2018.8.24 | py36_1 140 KB
libarchive-3.3.3 | h7d0bbab_1 1.5 MB
future-0.16.0 | py36_0 668 KB
jsonschema-2.6.0 | py36_0 62 KB
beautifulsoup4-4.6.3 | py36_0 138 KB
filelock-3.0.9 | py36_0 13 KB
jupyter_core-4.4.0 | py36_0 63 KB
conda-build-3.16.0 | py36_0 469 KB
pytz-2018.5 | py36_0 232 KB
ipython_genutils-0.2.0 | py36_0 39 KB
openssl-1.0.2p | h14c3975_0 3.5 MB
python-libarchive-c-2.8 | py36_6 20 KB
anaconda-client-1.7.2 | py36_0 141 KB
nodejs-8.12.0 | he6710b0_0 14.6 MB
nbformat-4.4.0 | py36_0 141 KB
tqdm-4.26.0 | py36h28b3542_0 59 KB
lzo-2.10 | h49e0be7_2 313 KB
python-dateutil-2.7.3 | py36_0 260 KB
traitlets-4.3.2 | py36_0 133 KB
bzip2-1.0.6 | h14c3975_5 414 KB
libxml2-2.9.8 | h26e45fe_1 2.0 MB
click-7.0 | py36_0 118 KB
patchelf-0.9 | hf484d3e_2 68 KB
markupsafe-1.0 | py36h14c3975_1 24 KB
conda-4.5.11 | py36_0 1.0 MB
libgcc-ng-8.2.0 | hdf63c60_1 7.6 MB
clyent-1.2.2 | py36_1 18 KB
lz4-c-1.8.1.2 | h14c3975_0 158 KB
pkginfo-1.4.2 | py36_1 39 KB
psutil-5.4.7 | py36h14c3975_0 305 KB
decorator-4.3.0 | py36_0 15 KB
icu-58.2 | h9c2bf20_1 22.5 MB
------------------------------------------------------------
Total: 60.7 MB
The following NEW packages will be INSTALLED:
anaconda-client: 1.7.2-py36_0
beautifulsoup4: 4.6.3-py36_0
bzip2: 1.0.6-h14c3975_5
click: 7.0-py36_0
clyent: 1.2.2-py36_1
conda-build: 3.16.0-py36_0
conda-verify: 3.1.1-py36_0
decorator: 4.3.0-py36_0
filelock: 3.0.9-py36_0
future: 0.16.0-py36_0
glob2: 0.6-py36_1
icu: 58.2-h9c2bf20_1
ipython_genutils: 0.2.0-py36_0
jinja2: 2.10-py36_0
jsonschema: 2.6.0-py36_0
jupyter_core: 4.4.0-py36_0
libarchive: 3.3.3-h7d0bbab_1
libxml2: 2.9.8-h26e45fe_1
lz4-c: 1.8.1.2-h14c3975_0
lzo: 2.10-h49e0be7_2
markupsafe: 1.0-py36h14c3975_1
nbformat: 4.4.0-py36_0
nodejs: 8.12.0-he6710b0_0
patchelf: 0.9-hf484d3e_2
pkginfo: 1.4.2-py36_1
psutil: 5.4.7-py36h14c3975_0
python-dateutil: 2.7.3-py36_0
python-libarchive-c: 2.8-py36_6
pytz: 2018.5-py36_0
pyyaml: 3.13-py36h14c3975_0
tqdm: 4.26.0-py36h28b3542_0
traitlets: 4.3.2-py36_0
zstd: 1.3.3-h84994c4_0
The following packages will be UPDATED:
certifi: 2018.4.16-py36_0 --> 2018.8.24-py36_1
conda: 4.5.4-py36_0 --> 4.5.11-py36_0
libgcc-ng: 7.2.0-hdf63c60_3 --> 8.2.0-hdf63c60_1
libstdcxx-ng: 7.2.0-hdf63c60_3 --> 8.2.0-hdf63c60_1
openssl: 1.0.2o-h20670df_0 --> 1.0.2p-h14c3975_0
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
waxmittmann
All 10 comments
Can you show your package list again, this time with --show-chsnnel-urls please?
mingwandroid
on 10 Oct 2018
conda update --all (or just conda update python) should see you right.
mingwandroid
on 10 Oct 2018
I had to fix our builds with this patch:
diff --git a/libarchive/ffi.py b/libarchive/ffi.py
--- a/libarchive/ffi.py
+++ b/libarchive/ffi.py
@@ -15,6 +15,7 @@
import logging
import mmap
import os
+import sys
from .exception import ArchiveError
@@ -24,8 +24,8 @@
page_size = mmap.PAGESIZE
-libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive')
+libarchive_path = f'{sys.prefix}/lib/libarchive.so'
libarchive = ctypes.cdll.LoadLibrary(libarchive_path)
# Constants
sscherfke
on 10 Oct 2018
That's only going to be needed temporarily but good that you got unstuck.
mingwandroid
on 10 Oct 2018
@mingwandroid do we need to add a run_constrained entry to conda-build so that an update to the latest python package build (same X.Y series, maybe same version, but newer build number) will be forced when someone updates to conda-build 3.16.1+?
msarahan
on 11 Oct 2018
We probably should yes.
mingwandroid
on 11 Oct 2018
I'll have to ask Kale. I'm not sure conda supports any expression to constraint >= build numbers. If it does, I'm pretty sure it's only a very recent addition (no 4.3 support).
msarahan
on 11 Oct 2018
Well, new python 3.6 and 3.7 are coming out real soon now!
mingwandroid
on 11 Oct 2018
Same problem here; happening only in one of our repos; no idea what makes it special. Fixed it by installing conda-build=3.14 instead of latest; will see if conda update --all works too, as suggested above.
EDIT: yup update --all also worked. Tough one to figure out this was; cost me a lot of time! Not quite sure what is going on myself, but would it be hard to get this covered with a regression test?
EelcoHoogendoorn
on 17 Oct 2018
I also hit this issue in a pipeline, and confirm conda update --all works. For many use cases, this should be resolved when the newer versions are bundled within the Miniconda installer and docker image.
scw
on 25 Oct 2018
Related issues
pdp10
路
6Comments
asmeurer
路
3Comments
frol
路
5Comments
ebolyen
路
3Comments
sscherfke
路
4Comments