Staged-recipes: Somebody just broke `gcc` on Linux
isl: 0.16.1-0
Removing old build environment
Removing old work directory
BUILD START: bob.learn.em-2.0.8-py27_0
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ..........
Solving package specifications: .........
The following NEW packages will be INSTALLED:
bob.blitz: 2.0.8-np111py27_4
bob.core: 2.1.2-py27_1
bob.extension: 2.0.11-py27_4
bob.io.base: 2.0.8-py27_0
bob.learn.activation: 2.0.4-py27_2
bob.learn.linear: 2.0.7-py27_2
bob.math: 2.0.3-py27_2
bob.sp: 2.0.4-py27_2
boost: 1.60.0-py27_0
cloog: 0.18.0-0
cmake: 3.5.0-3
gcc: 4.8.5-3
gmp: 6.1.0-2
hdf5: 1.8.16-2
icu: 56.1-2
isl: 0.16.1-0
libblitz: 0.10-1
libgcc: 4.8.5-1
libgfortran: 3.0-0
mkl: 11.3.1-0
mpc: 1.0.3-1
mpfr: 3.1.4-1
numpy: 1.11.0-py27_0
openblas: 0.2.14-4
openssl: 1.0.2g-0
pip: 8.1.1-py27_1
pkg-config: 0.28-1
python: 2.7.11-0
readline: 6.2-2
setuptools: 20.7.0-py27_0
sqlite: 3.9.2-0
tk: 8.5.18-0
wheel: 0.29.0-py27_0
zlib: 1.2.8-0
Linking packages ...
/home/amir/miniconda/envs/_build/gcc/libexec/gcc/x86_64-unknown-linux-gnu/4.8.5/cc1: error while loading shared libraries: libisl.so.10: cannot open shared object file: No such file or directory | 76%
Installation failed: gcc is not able to compile a simple 'Hello, World' program.
Error: post-link failed for: gcc-4.8.5-3
183amir
All 72 comments
That gcc package needs to be eliminated. For now add this isl 0.12.2 to your build requirements.
jakirkham
on 20 Apr 2016
ok thanks.
183amir
on 20 Apr 2016
@pelson, is there some way for us to pin gccs dependencies build time dependencies? This should have no effect on run time dependencies. In particular, they should match the versions below. We already package many of these things and anyone of them has the chance to get upgraded and break the compiler. Apparently none of them are pinned in the compiler package.
cloog 0.18.0
gcc 4.8.5
gmp 5.1.2
isl 0.12.2
mpc 1.0.1
mpfr 3.1.2
Sorry, @183amir, reopening as I would like @pelson to look at this.
jakirkham
on 20 Apr 2016
@jakirkham - is that just an upgrade to the gcc recipe, or does it need to be applied to every single feedstock which depends on any of these?
pelson
on 20 Apr 2016
Any feedstock that depends on gcc. :/
jakirkham
on 20 Apr 2016
Alternatively we can just delete the package from our channel for now.
jakirkham
on 20 Apr 2016
Though there is no telling when somebody redeploys it. We should definitely raise this with Continuum though. Not pinning library dependencies of a compiler is deeply troubling.
jakirkham
on 20 Apr 2016
I see. Continuum have just updated gcc? On anaconda.org it looks like it hasn't been updated for >5 months...
Should that package be pinning isl to a specific version? (it doesn't currently)
pelson
on 20 Apr 2016
No I released isl, but it is a dependency of gcc.
According to the conda-recipe it should never have used this version as it won't allow anything over 0.14.0. We released 0.16.1, which should not have caused an issue, but it did.
jakirkham
on 20 Apr 2016
Thus, I'm not sure if there is any reliable pinning with gcc regarding any of its other dependencies. Many of which we have packaged. For example, gmp, mpfr, mpc (things used by various symbolic algebra packages like SymPy).
jakirkham
on 20 Apr 2016
Honestly, the only thing we haven't packaged is cloog, but I imagine that will be proposed too.
jakirkham
on 20 Apr 2016
According to the conda-recipe it should never have used this version as it won't allow anything over 0.14.0. We released 0.16.1, which should not have caused an issue, but it did.
That isn't the gcc that we are using though. The anaconda one _doesn't_ pin a isl version at all.
pelson
on 20 Apr 2016
This had been the canonical gcc for some time. The reason I know this is @stuarteberg patched to be portable across various Linux distros. These changes were incorporated by @asmeurer in the rebuild of gcc.
jakirkham
on 20 Apr 2016
Click on the i at https://anaconda.org/anaconda/gcc/files. It is clear that the distribution there, _isn't_ from the recipe at conda-recipes.
pelson
on 20 Apr 2016
In fact, we would not even be able to use it here without @stuarteberg's patches. The old compiler didn't work on CentOS 6. I worked with him to fix that very case.
jakirkham
on 20 Apr 2016
Sounds like we need a conda-forge package for gcc...
pelson
on 20 Apr 2016
You're right this doesn't have the isl pinning. This is certainly an error. Sorry I should have checked.
jakirkham
on 20 Apr 2016
...or we could adopt devtoolset. We really shouldn't have something so fundamental be subject to such breakages.
jakirkham
on 20 Apr 2016
I wonder if gcc can even be built within the CIs constraints.
jakirkham
on 20 Apr 2016
Can we ask that the compiler get rebuilt?
jakirkham
on 20 Apr 2016
Adding cloog ( https://github.com/conda-forge/staged-recipes/pull/437 ). This is the only dependency of gcc we don't have.
jakirkham
on 20 Apr 2016
Add gcc ( https://github.com/conda-forge/staged-recipes/pull/438 ) in this PR.
jakirkham
on 20 Apr 2016
@pelson, would you mind taking a look at the cloog PR ( https://github.com/conda-forge/staged-recipes/pull/437 ). This builds quite fast on the CIs and should force icl back to 0.12.2, which should fix this issue.
jakirkham
on 20 Apr 2016
Could you please try dropping the pinning, @183amir?
jakirkham
on 20 Apr 2016
Shouldn't I wait till gcc is available?
183amir
on 20 Apr 2016
Nope. Shouldn't be necessary.
jakirkham
on 20 Apr 2016
We pinned isl via cloog. It was already a dependency, but this should make it stringent enough for the conda solver to pull in the right one.
jakirkham
on 20 Apr 2016
My conda is not downloading any new version:
bob.io.image-feedstock (git)-[master] % conda build -c conda-forge --python=2.7 recipe
Removing old build environment
Removing old work directory
BUILD START: bob.io.image-2.0.5-py27_1
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ..........
Solving package specifications: .........
The following NEW packages will be INSTALLED:
bob.blitz: 2.0.8-np111py27_4
bob.core: 2.1.2-py27_1
bob.extension: 2.0.11-py27_4
bob.io.base: 2.0.8-py27_0
boost: 1.60.0-py27_0
cloog: 0.18.0-0
gcc: 4.8.5-3
giflib: 5.1.2-1
gmp: 6.1.0-2
hdf5: 1.8.16-2
icu: 56.1-2
isl: 0.16.1-0
jpeg: 9b-0
libblitz: 0.10-1
libgcc: 4.8.5-1
libpng: 1.6.17-0
libtiff: 4.0.6-3
mkl: 11.3.1-0
mpc: 1.0.3-1
mpfr: 3.1.4-1
numpy: 1.11.0-py27_0
openssl: 1.0.2g-0
pip: 8.1.1-py27_1
pkg-config: 0.28-1
python: 2.7.11-0
readline: 6.2-2
setuptools: 20.7.0-py27_0
sqlite: 3.9.2-0
tk: 8.5.18-0
wheel: 0.29.0-py27_0
xz: 5.2.2-0
zlib: 1.2.8-0
Linking packages ...
/home/amir/miniconda/envs/_build/gcc/libexec/gcc/x86_64-unknown-linux-gnu/4.8.5/cc1: error while loading shared libraries: libisl.so.10: cannot open shared object file: No such file or directory | 87%
Installation failed: gcc is not able to compile a simple 'Hello, World' program.
Error: post-link failed for: gcc-4.8.5-3
Can you turn on show channel urls and try again. It will help us see where things are coming from.
pelson
on 20 Apr 2016
bob.io.image-feedstock (git)-[master] % conda build -c defaults -c conda-forge --override-channels --python=2.7 recipe :(
Removing old build environment
Removing old work directory
BUILD START: bob.io.image-2.0.5-py27_1
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ........
Solving package specifications: .........
The following packages will be downloaded:
package | build
---------------------------|-----------------
giflib-5.1.2 | 0 160 KB conda-forge
The following NEW packages will be INSTALLED:
bob.blitz: 2.0.8-np111py27_4 conda-forge
bob.core: 2.1.2-py27_1 conda-forge
bob.extension: 2.0.11-py27_4 conda-forge
bob.io.base: 2.0.8-py27_0 file:///home/amir/miniconda/conda-bld/linux-64/
boost: 1.60.0-py27_0 defaults
cloog: 0.18.0-0 defaults
gcc: 4.8.5-3 defaults
giflib: 5.1.2-0 conda-forge
gmp: 6.1.0-2 conda-forge
hdf5: 1.8.16-2 conda-forge
icu: 56.1-2 conda-forge
isl: 0.16.1-0 conda-forge
jpeg: 9b-0 conda-forge
libblitz: 0.10-0 conda-forge
libgcc: 4.8.5-1 defaults
libpng: 1.6.17-0 defaults
libtiff: 4.0.6-3 conda-forge
mkl: 11.3.1-0 defaults
mpc: 1.0.3-1 conda-forge
mpfr: 3.1.4-1 conda-forge
numpy: 1.11.0-py27_0 defaults
openssl: 1.0.2g-0 defaults
pip: 8.1.1-py27_1 defaults
pkg-config: 0.28-1 conda-forge
python: 2.7.11-0 defaults
readline: 6.2-2 defaults
setuptools: 20.7.0-py27_0 defaults
sqlite: 3.9.2-0 defaults
tk: 8.5.18-0 defaults
wheel: 0.29.0-py27_0 defaults
xz: 5.2.2-0 defaults
zlib: 1.2.8-0 defaults
Fetching packages ...
giflib-5.1.2-0 100% |###################################################################################################################################################################| Time: 0:00:00 370.81 kB/s
Extracting packages ...
[ COMPLETE ]|######################################################################################################################################################################################| 100%
Linking packages ...
/home/amir/miniconda/envs/_build/gcc/libexec/gcc/x86_64-unknown-linux-gnu/4.8.5/cc1: error while loading shared libraries: libisl.so.10: cannot open shared object file: No such file or directory | 87%
Installation failed: gcc is not able to compile a simple 'Hello, World' program.
Error: post-link failed for: gcc-4.8.5-3
Sorry guys I need to get a few hours of sleep. Will check back after, but it doesn't look like you are getting our cloog. This could be a channel configuration issue.
jakirkham
on 20 Apr 2016
Agreed. If you swap -c defaults -c conda-forge for -c conda-forge -c defaults, does that make a difference? Also, can you provide conda info
pelson
on 20 Apr 2016
No changing their order doesn't make a difference.
Using Anaconda Cloud api site https://api.anaconda.org
Current conda install:
platform : linux-64
conda version : 4.0.5
conda-build version : 1.20.0
python version : 3.5.1.final.0
requests version : 2.9.1
root environment : /home/amir/miniconda (writable)
default environment : /home/amir/miniconda
envs directories : /home/amir/miniconda/envs
package cache : /home/amir/miniconda/pkgs
channel URLs : https://conda.anaconda.org/t/<TOKEN>/bobpkgs/linux-64/
https://conda.anaconda.org/t/<TOKEN>/bobpkgs/noarch/
https://repo.continuum.io/pkgs/free/linux-64/
https://repo.continuum.io/pkgs/free/noarch/
https://repo.continuum.io/pkgs/pro/linux-64/
https://repo.continuum.io/pkgs/pro/noarch/
config file : /home/amir/.condarc
is foreign system : False
You can also see that in happens also in our CI: https://circleci.com/gh/conda-forge/bob.io.image-feedstock/9
183amir
on 20 Apr 2016
In this pull request: https://github.com/conda-forge/bob.io.image-feedstock/pull/2
183amir
on 20 Apr 2016
Now I know why cloog from default channel is being pulled:
gcc: 4.8.5-3 defaults
Error: The following specifications were found to be in conflict:
- gcc -> cloog 0.18.0
- python 2.7*
The gcc from defaults channel depends on 0.18.0
conda search cloog
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
cloog . 0.18.0 0 defaults
* 0.18.1 1 conda-forge
Give that a shot (once the CI has finished from https://github.com/conda-forge/cloog-feedstock/pull/1)
pelson
on 20 Apr 2016
It's released.
jakirkham
on 20 Apr 2016
@jakirkham - go to sleep! :zzz:
pelson
on 20 Apr 2016
I did. I wasn't kidding when I said a few hours.
jakirkham
on 20 Apr 2016
https://circleci.com/gh/conda-forge/staged-recipes/2306
https://circleci.com/gh/conda-forge/bob.io.image-feedstock/10
183amir
on 20 Apr 2016
I did. I wasn't kidding when I said a few hours.
I guess it is time to add a new rule in the code of conduct: Conda-forge member must sleep at least 8 hours per day :stuck_out_tongue_winking_eye:
ocefpaf
on 20 Apr 2016
I am staying far away from compiler tools.
jakirkham
on 20 Apr 2016
How is this even still a problem. We have cloog 0.18.0 with a higher build number. Maybe we need to build the exact same build number?
jakirkham
on 20 Apr 2016
The only way I could tell conda to get the conda-forge one was pinning the version of cloog to 0.18.0 like this: - cloog 0.18.0 in build requirements in my recipe.
183amir
on 20 Apr 2016
That helped? Geez, what a mess. Did you still have isl pinned too?
jakirkham
on 20 Apr 2016
No, I removed isl. Here is the recipe:
{% set version = "2.0.8" %}
package:
name: bob.learn.em
version: {{ version }}
source:
fn: bob.learn.em-{{ version }}.zip
url: https://pypi.python.org/packages/source/b/bob.learn.em/bob.learn.em-{{ version }}.zip
md5: eaa69afa7b82b3309e5010aeeccbd1fc
build:
number: 0
skip: true # [not linux]
script: python -B setup.py install --single-version-externally-managed --record record.txt
requirements:
build:
- python
- setuptools
- bob.extension
- bob.blitz
- bob.core >2.0.5
- bob.io.base
- bob.sp
- bob.math
- bob.learn.activation
- bob.learn.linear
- cmake
- pkg-config
- cloog 0.18.0
- gcc # [linux]
run:
- python
- bob.extension
- bob.blitz
- bob.core >2.0.5
- bob.io.base
- bob.sp
- bob.math
- bob.learn.activation
- bob.learn.linear
- libgcc # [linux]
test:
commands:
- nosetests -sv bob.learn.em
imports:
- bob
- bob.learn
- bob.learn.em
- bob.learn.em.test
requires:
- nose
about:
home: http://github.com/bioidiap/bob.learn.em
license: Modified BSD License (3-clause)
summary: Bindings for emelaneous machines and trainers of Bob
extra:
recipe-maintainers:
- 183amir
Not really sure why that should matter. @mcg1969, could you explain what is going on with the solver here? It seems to only respect build numbers if we pin a package whose version we would get without pinning.
jakirkham
on 20 Apr 2016
Alright, I'm releasing something that has the exact same build number as defaults. Not really sure it will work, but let's see anyways.
jakirkham
on 20 Apr 2016
Probably a dependency conflict somewhere. But I frankly find it impossible to debug conda solver issues if I have to wade through conda build as well. Do you think you could submit an issue to conda/conda with a reproducible example that doesn't involve conda build?
mcg1969
on 20 Apr 2016
Note that conda 4.1 _will_ have better channel collision handling. Doing -c conda-forge will cause conda to prefer _any_ package from that channel over defaults---assuming its dependencies are satisfied.
mcg1969
on 20 Apr 2016
I think it is fixed now, @183amir, but this is a horrible hack. Basically, we shadow defaults copy and it then bumps the build number. See in your staged-recipes log.
jakirkham
on 20 Apr 2016
Ah, so that got merged, @mcg1969.
jakirkham
on 20 Apr 2016
The first cut of it got merged into master but we discovered an issue with the package cache that needed a rather involved fix. That's in the mgrant/channelfix branch and will be in 4.1.
mcg1969
on 20 Apr 2016
Basically what was happening was this, @mcg1969. gcc depends on cloog amongst other things. The defaults version of cloog is 0.18.0 0. It does depend on isl and gmp, but pins neither in anyway. We added cloog 0.18.0 10. This pins isl and gmp exactly to 5.1.2 and 0.12.2. However, the lower build number was preferred. So, this is the question. What doesn't it pick the higher build number.
Now we have added cloog 0.18.0 0, which is exactly the same as cloog 0.18.0 10, but with a lower build number. This shadows defaults copy and the solver picks cloog 0.18.0 10.
jakirkham
on 20 Apr 2016
Sorry I just can't go by descriptions alone. Of course it should prefer a higher build number. So it's probably a dependency issue, and that requires a reproducible test to tease out.
mcg1969
on 20 Apr 2016
Should add we have isl 0.16.1. I think for some reason it was preferring less restrictions and trying to get the newer version of isl. However, we can't have this or the compiler borks.
jakirkham
on 20 Apr 2016
Doing -c conda-forge will cause conda to prefer any package from that channel over defaults---assuming its dependencies are satisfied.
From conda-forge's perspective, this could be problematic if:
conda install -c conda-forge gdal
Would actually install the gdal from defaults because defaults are prepended to the channels list. @mcg1969 - I guess that has been considered, right?
pelson
on 20 Apr 2016
I'll try to work on a simple case with dummy packages and post it the conda issue tracker later, @mcg1969.
jakirkham
on 20 Apr 2016
@msarahan, I know this is unpopular and undesired, but is there any chance we could get a rebuild of the gcc package? We have a workaround in place, but I'm afraid it is pretty fragile and changes to gcc's dependencies or conda could upset the careful balance here. As we still seem tied to the gcc package solution for the near term, it would be nice if it were pinned correctly when it comes to defaults packages. In particular, I would like to get @frol's version constraint fixes that would be included with build number 4. ( https://github.com/conda/conda-recipes/tree/447b5ee24052e3f892f484a30a0ace9f498d696d/gcc-4.8 ). Alternatively, if we can adopt a different intermediate solution like the devtoolset PR ( https://github.com/pelson/Obvious-CI/pull/61 ), we could avoid these dependency shifts affecting our toolchain.
jakirkham
on 20 Apr 2016
I am not likely to have time to rebuild gcc on the time scale that I think you want/need. I recommend merging your devtoolset PR.
msarahan
on 20 Apr 2016
If you are ok with that, let's do it.
jakirkham
on 20 Apr 2016
@pelson, channels specified with -c take priority over channels in .condarc and/or the defaults.
We still need a way to override the channel priority for single packages. That will come.
mcg1969
on 20 Apr 2016
@jakirkham: before you dump a lot of time to build a test case, look at the dependencies for the packages involved here, and compare those to the dependencies in Anaconda's equivalents. If yours are stricter than ours, that's solid evidence that this is a dependency issue and not a solver bug. (Frankly, I'm pretty darn certain it's not a solver bug.)
mcg1969
on 20 Apr 2016
For instance, this file suggests you're tying cloog 0.18.0 to particular versions of gmp and isl.
mcg1969
on 20 Apr 2016
Unfortunately we have to, the compiler breaks if we don't have that exact version of isl. The gmp pinning might not matter so much as we have had our own gmp package for a long time without issues.
jakirkham
on 20 Apr 2016
I understand you might have _reasons_ for pinning the versions of these packages, but this just makes it all the more likely that conda is going to be _forced_ to select our package over yours.
mcg1969
on 20 Apr 2016
Situations like this point to the need to implement a "strict" channel mode in conda, so that it isn't _permitted_ to reach down to other channels to pull in a package. I'd much rather have a hard failure due to dependency conflicts in this case, that way it spurs you on to find the true cause.
mcg1969
on 20 Apr 2016
Well, IMHO having a broken compiler is a non-starter. Many people are dependent on this gcc package for C++0X or outright C++11 features.
That all being said, this is one of many reasons we are try to move away from packaging it at all. Though we might be stuck on Mac (Fortran and OpenMP support though no Mac native Docker containers). So, we will still need to think a bit about these sorts of problems.
jakirkham
on 20 Apr 2016
@mcg1969, one of the reasons I'm concerned about a solver bug is this comment by @183amir. It doesn't make sense to me why pinning something to a version it already will land on (has to as gcc pinning is exact wrt to cloog version) will cause the build number to go up, but not pinning it to that version results in ignoring the build number.
jakirkham
on 21 Apr 2016
So we placed a workaround. Also we migrated everyone we could off of the gcc package. It broke when channel priority became a thing. We pinged Continuum for a hot-fix to restrict isl correctly. They fixed it and things seem to be working reliably now. Going to go ahead and close this as it is resolved.
jakirkham
on 25 Jul 2016
Related issues
jakirkham
路
7Comments
sachalau
路
8Comments
isuruf
路
4Comments
matham
路
7Comments
sodre
路
4Comments
Most helpful comment
I guess it is time to add a new rule in the code of conduct: Conda-forge member must sleep at least 8 hours per day :stuck_out_tongue_winking_eye: