Conda-forge.github.io: cos7 CDTs and new compilers

[beckermr]

As part of our effort to add support for cos7 to linux-64, we'd like to do some refactoring of the structure of our CDTs. We (@isuruf and myself) are looking for community help in this work.

To do:

1. make sure old CDTs are mutually exclusive with the new ones

   • [x] patch the repo data to make all current CDTs conflict with the sysroot packages (https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/60)
   • [ ] patch the CDT repo data on defaults so they conflict with the sysroot package (https://github.com/AnacondaRecipes/repodata-hotfixes/pull/94)
   • [x] patch repodata so that new sysroots don't interact with old compilers (https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/66, https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/65)

2. build CDTS

   • [x] setup one-off CDT builds on conda-forge (https://github.com/conda-forge/cdt-builds/pull/2)
   • [x] build all of the old-style CDTs
   • [x] build all of the new-style ones too
   • [x] make all CDTs use no_hoist: true in the source sections
   • [x] new-style CDTs should use the new conda sysroot (i.e., {{ arch }}-conda-linux-gnu) sysroot
   • [x] new-style CDTs should depend on the correct version of the sysroot package (i.e., cos6 depends on 2.12 and cos7 depends on 2.17)
   • [x] old-style CDTs need to have the proper run_constrained
   • see https://hackmd.io/T5q0GPa1QfKfQ6irNWAe5Q?edit for notes from the CDT discussion

3. move to only conda-forge CDTs

   • [x] enable strict channel priority in the ci setup scripts (to make sure we always use our CDTs, https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/110)

4. turn on new compilers

5. [x] finally remove the symlinks and CDT-location variants in the new compilers (https://github.com/conda-forge/linux-sysroot-feedstock/pull/26, https://github.com/conda-forge/linux-sysroot-feedstock/pull/27)
6. [x] make sure to add a sysroot run dep to gcc_impl, gxx_impl and gfortran_impl so that the repo data patches above don't break the new compilers (https://github.com/conda-forge/ctng-compilers-feedstock/pull/32)

cc @isuruf @jakirkham @kkraus14 @nehaljwani @mingwandroid @jjhelmus

xref: https://github.com/conda-forge/conda-forge.github.io/issues/900
closes https://github.com/conda-forge/ctng-compilers-feedstock/issues/28

[beckermr]

cc @brianv0 too

[kkraus14]

cc @mike-wendt

[mingwandroid]

I need to rebuild all the COS6 CDTs (I did this a while back, those are on anaconda.org/rdonnelly) due to a bug (missing feature kinda really) in the construction of the old ones (to do with conda-build not having any way to be told not to 'hoist' the top level dir from archives it unpacks).

This is fixed now. Please make sure your CDT meta.yamls have:

source:
  - url: http://mirror.centos.org/centos/6.10/os/x86_64/Packages/mesa-libEGL-11.0.7-4.el6.x86_64.rpm
    sha256: 1b5638595636d26590e1a2140c9d64a452b63d985a507197206d0e85b7796753
    no_hoist: true

[beckermr]

thanks @mingwandroid!

[beckermr]

Hi all. We need a plan to break this work up and get it done. Can you all join the next core meeting? We’ll have an extra one next week at the usual time.

[beckermr]

I've made a few PRs with a pretty hacky script.

https://github.com/conda-forge/staged-recipes/pull/12014
https://github.com/conda-forge/staged-recipes/pull/12015
https://github.com/conda-forge/staged-recipes/pull/12016

I'd be great if you all can look them over!

[jakirkham]

Those all appear to be closed. Is that expected or am I looking in the wrong place?

[beckermr]

I made some improvements and so closed them. Sorry!

The recipe for zip 3 on x86_64 was just merged. You can check that one out.

[isuruf]

Should we be moving everything from defaults or just ones that we actually use in conda-forge? I don't think it makes sense to move everything and increase the maintenance burden.

[jakirkham]

Either way seems fine to me. The maintenance burden here is pretty low. These are typically built once and that's it.

[jakirkham]

So what other CDTs do we still need @beckermr? Do you also have a link to the script somewhere?

[beckermr]

I will push the script later. There are ~250 cdts in defaults. Idk which ones we use

[beckermr]

Here is the CDT list: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/cdt_list.yaml

I made this list by grabbing all CDTs in the aggregate repo from anaconda.

Here is the script: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/make_cdt_pr.py

Pls do not make PRs with it yet or push to my fork.

[jakirkham]

Ok should we find some time to align next week? Either at the core meeting or separately?

[beckermr]

I planned to do this at the core meeting

[beckermr]

I’ve archived the zip feedstock so nobody has to maintain it.

@isuruf can you send a list of cdts we need?

[beckermr]

Here is my best guess for the CDTs we need: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/needed_cdt_list.yaml

The script that computed them is here: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/compute_needed_cdts.py

[beckermr]

draft of the CDT builds is here: https://github.com/conda-forge/cdt-builds/pull/2

[beckermr]

I have made the last PR needed here. it may or may not be merged and is not strictly needed. closing this issue!