Conda-forge.github.io: ENH move to glibc 2.17 (aka cos7)

[xhochy]

To do:

• [x] split out the sysroot in the current ct-ng packages and use CDT sysroot (https://github.com/conda-forge/ctng-compilers-feedstock/pull/23)
• [x] remove cos from the sysroot name for binutils (https://github.com/conda-forge/binutils-feedstock/pull/25)
• [x] make new binutils not like old compilers (https://github.com/conda-forge/binutils-feedstock/pull/27)
• [x] patch the old binutils packages to have them be incompatible with the sysroot packages (https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/49)
• [x] fix issue with track features and current_repodata.json (https://github.com/conda/conda-build/pull/3957, https://github.com/conda-forge/linux-sysroot-feedstock/pull/12, https://github.com/conda-forge/linux-sysroot-feedstock/pull/11)
• [x] build the cos7 glibc 2.17 CDT
• [x] test compiler w/ new cos7 CDT
• [x] make sure the proper virtual package constraints are present in the CDT package and exported to the proper runtimes
• [x] fix extra DSO errors (https://github.com/conda/conda-build/pull/3969)
• [x] merge final PRs for sysroot shims (https://github.com/conda-forge/binutils-feedstock/pull/32, https://github.com/conda-forge/ctng-compilers-feedstock/pull/29)
• [x] ~mark the latest version of the compilers as broken to move to new sysroot (https://github.com/conda-forge/cf-mark-broken/pull/67)~
• [x] ship 9.3 libs for ctng-compilers feedstock based on new sysroot build (https://github.com/conda-forge/ctng-compilers-feedstock/pull/34)
• [ ] fix issue with virtual package constraints (https://github.com/conda/conda/issues/9896)
• [x] Bring all x86_64 cos6 CDTs into conda-forge
• [x] Bring all x86_64 cos7 CDTs into conda-forge
• [x] After the 2 above, move CDTs into conda symlink
• [x] make old compiler activation scripts conflict with new compilers (https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/68)
• [x] add sysroot deps to new compiler activation scripts (https://github.com/conda-forge/ctng-compiler-activation-feedstock/pull/24)
• [x] make sure to add a sysroot run dep to gcc_impl, gxx_impl and gfortran_impl so that the repo data patches for the CDTs don't break the new compilers (conda-forge/ctng-compilers-feedstock#32)
• [x] ~patch conda-build for sysroot distro name (https://github.com/conda/conda-build/pull/3999)~
• [x] fix python sysconfig data

stretch goals

• [ ] maybe build gcc from a new gcc-feedstock instead of ct-ng?

other relevant issues

• uninstalling from symlinks in other packages is broken (https://github.com/conda/conda/issues/9810#issuecomment-607941126)

I (@beckermr) am pretty out of my depth here but ready to learn!

old comment from @xhochy

I have packages that require newer glibc features like setns or process_vm_readv. These are not included in the glibc version currently used for building on Linux. Is there a way to use a newer baseline glibc* for a package or impossible until we up the base CentOS version globally?

*I'm aware that the package would run on less systems but that would do it either way.

[isuruf]

It's not possible at the moment. There are several steps involved,

1. Add a virtual package to conda. This is discussed in https://github.com/conda-forge/conda-forge.github.io/issues/891 for newer OSX SDKs as well.

2. Add a run_exports type for run_constrained. (This is optional) This is so that the virtual package is added as a run_constrained automatically.

3. Build new compilers.

4. Add a new docker image.

[jakirkham]

Raised issue ( https://github.com/conda/conda/issues/9348 ) to discuss 1.

[jakirkham]

We had a discussion similar to 2 for cudatoolkit. ( https://github.com/conda/conda/issues/9115 )

[mingwandroid]

Pinging @kalefrannz and @msarahan for comments too. We had specced out some metadata for the solver, is @isuruf's plan enough here? Do we need virtual packages?

[msarahan]

Virtual packages are probably not strictly necessary, but they will make it much easier to tell people that have inadequate glibc exactly why they can't install a package. It probably needs to be two virtual packages, though: one for libc implementation (glibc, musl) and another for version.

[isuruf]

It probably needs to be two virtual packages, though: one for libc implementation (glibc, musl) and another for version.

Can't it be one each for libc implementation? A virtual package like glibc=2.19 would do right? We don't need mutual exclusivity because the system ensures that.

[jakirkham]

We don't need mutual exclusivity because the system ensures that.

Not exactly. glibc and musl do have some level of compatibility. However this isn't always the case and when there is an issue it can be quite subtle. Would much rather have two packages like Mike suggests.

[scopatz]

@isuruf - what needs to be done for (3) build newer compilers at this point?

[isuruf]

The sysroot for the compiler is built at https://github.com/conda-forge/ctng-compilers-feedstock
We need to change the sysroot from 2.12.2 to 2.17. To ensure that we don't use 2.17 by default, we first have to split the sysroot to another output, call it sysroot_cos6_linux-64 and then pin that in the activation scripts. After that we can bump the sysroot to 2.17.

[scopatz]

@isuruf - how are we suppossed to select between the different sysroots? It seems that both of them have the [linux64] selector. Maybe another question is do we intend to build both the centos6 & centos7 stacks simeltaneously as different ouptuts, or is the intention that we simply flip the sysroot version number whenever we want to build a different version?

[isuruf]

@scopatz, I'm sorry I don't understand. Can you explain more?

[scopatz]

I don't fully understand what to do. I guess I can try putting in a PR and you can give feedback / push to that branch? :wink:

[isuruf]

@conda-forge/core, the compilers unfortunately have conda_cos6 name in them and some packages hardcode this name for later use.

There are two options,

1. Rename to conda_cos and have a symlink from conda_cos6 to conda_cos
2. Have conda_cos7 and have a symlink from conda_cos6 to conda_cos7

1 makes it easier.
2 is also fine and has the added benefit that packages which were built with conda_cos7 will require a new sysroot instead of possibly silently using the conda_cos6 sysroot, but when conda_cos8 comes along we need 2 sets of symlinks.

Which option should we go with? Since this affects defaults, pinging @kalefranz, @mingwandroid

[scopatz]

I think we should go with the most future-proof way of handling this, which seems to be option 2.

[isuruf]

@kalefranz, @mingwandroid, @msarahan, @jjhelmus, @nehaljwani, need your feedback too as we want this to be consistent with defaults.

[msarahan]

Option 2 seems the wisest.

[nehaljwani]

My personal opinion is to drop the distro name from the sysroot completely. I don't see any reason for associating a substring of HOST/BUILD with 'CentOS'. The pseudo cross compiler targets a particular kernel version and glibc version, not a Linux distribution.

One of the reasons why I say that is because CentOS folks maintain a huge number of patches over glibc, which crosstool-ng doesn't. For example, they back-ported IP_MULTICAST_ALL to glibc 2.12, which crosstool-ng didn't. So using _cos6_ was also misleading IMO. Also, if, in the future, CentOS decides to become more bleeding edge, and is no longer the widely used distro with the oldest glibc, then we might want to switch to debian/opensuse's glibc as the baseline version, then it would become even harder to get rid of 'cosX' in the name.

[mingwandroid]

For example, they back-ported IP_MULTICAST_ALL to glibc 2.12, which crosstool-ng didn't.

The first version of these compilers targeted CentOS5. I took care to port all patches at that time. I consider this a bug and we should be aim to port the CentOS6 patches across.

An alternative, which might be faster would be to use CDT packages for glibc, glibc-devel, kernel-headers and what not. That should work.

[scopatz]

It still is not entirely clear what to do here...

[isuruf]

https://github.com/conda-forge/staged-recipes/pull/10479 adds the sysroot from CDT packages. After merging that PR, the compiler package should remove building glibc and depend on that package.

[scopatz]

I see that makes sense

[jjhelmus]

comment moved to staged-recipes PR.

[jack-pappas]

The staged-recipes PR was merged back on 2020-01-20. Does that mean conda-forge/ctng-compilers-feedstock#3 can move forward using the last suggestion from @isuruf ?

[jakirkham]

Just a note, we ran into a bit of surprise recently with the __cuda virtual package when trying to constrain cudatoolkit with it ( https://github.com/conda/conda/issues/9115#issuecomment-620168193 ). Not sure if this extends to __glibc or __osx when constraining compiler runtimes with them. Figured it was worth mentioning just in case.

[beckermr]

I am getting interest from ppl I work with on seeing the move to 2.17 go through. Also, I think rhel6 is sunsetting in November. Our CDTs could not receive updates after that in principle at least. We may not actually update the CDTs OFC.

[isuruf]

WIP stuff at
https://github.com/isuruf/binutils-feedstock/tree/name
https://github.com/isuruf/gcc-feedstock

Feel free to take over

[beckermr]

Thank you!

[isuruf]

gcc-feedstock is a WIP recipe to build gcc without ct-ng. It needs to be put through staged-recipes when it can be built.

[jakirkham]

Just a note, we ran into a bit of surprise recently with the __cuda virtual package when trying to constrain cudatoolkit with it ( conda/conda#9115 (comment) ). Not sure if this extends to __glibc or __osx when constraining compiler runtimes with them. Figured it was worth mentioning just in case.

Now tracked as issue ( https://github.com/conda/conda/issues/9896 ).

[jakirkham]

I'm curious what the current status is here. Are there things people need help with to move this forward? Where would be the best place to chip in?

Also with this item...

fix issue with track features and current_repodata.json (conda/conda-build#3957, conda-forge/linux-sysroot-feedstock#12, conda-forge/linux-sysroot-feedstock#11)

It appears all PRs are merged. Should it be checked or are there still outstanding tasks here?

[kkraus14]

Just a note: CUDA 11.0 dropped CentOS 6 support https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#deprecated-features

[isuruf]

I'm sorry I don't have time to answer all your questions.

[beckermr]

I need to update this issue. We found another conda build issue that I put up a PR for. Once that is done, I think we can release the compilers. If you run conda-build but include my personal channel (-c beckermr) then you can try out the new compilers. They will work, but you will see extra DSO errors that are false positives.

[beckermr]

@jakirkham I updated the list

[beckermr]

If you all want to help, making a PR on conda to fix the virtual package issue would be appreciated.

[jakirkham]

Thanks @beckermr! 😄 Meaning issue ( https://github.com/conda/conda/issues/9896 )? Happy to take a look 🙂

[beckermr]

Yup that issue @jakirkham. Very glad that you are willing to look into it!

[beckermr]

Ok everyone. The compilers with cos7 are live. The docs pr is up to.

If anyone here wants to try them out, that’d be great!

[leofang]

@beckermr Great! Is it possible now to build packages that only support glibc 2.17? If so, how and where do I add the constraint __glibc==2.17?

[beckermr]

See the announcement on the docs page. You'll pick up a slightly older libcxx and libgcc but this should be fixed in a day or so

[xhochy]

@beckermr Using this in the feedstock for which this issue was originally written: https://github.com/conda-forge/py-spy-feedstock/pull/6

conda smithy rerender is failing for me locally with:

% conda smithy rerender
No azure token. Create a token and
put it in ~/.conda-smithy/azure.token
INFO:conda_smithy.configure_feedstock:Downloading conda-forge-pinning-2020.07.26.20.00.32
INFO:conda_smithy.configure_feedstock:Extracting conda-forge-pinning to /var/folders/3r/k0d_q3hs2mb2_jxfh_9hjyqr0000gn/T/tmprnlbfuv0
Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
WARNING:conda_build.config:Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
WARNING:conda_build.config:Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /var/folders/3r/k0d_q3hs2mb2_jxfh_9hjyqr0000gn/T/tmprnlbfuv0/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /var/folders/3r/k0d_q3hs2mb2_jxfh_9hjyqr0000gn/T/tmprnlbfuv0/conda_build_config.yaml
Adding in variants from /Users/uwe/Development/py-spy-feedstock/recipe/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /Users/uwe/Development/py-spy-feedstock/recipe/conda_build_config.yaml
Traceback (most recent call last):
  File "/Users/uwe/miniconda3/bin/conda-smithy", line 10, in <module>
    sys.exit(main())
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/cli.py", line 592, in main
    args.subcommand_func(args)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/cli.py", line 398, in __call__
    self._call(args, tmpdir)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/cli.py", line 409, in _call
    temporary_directory=temporary_directory,
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 1898, in main
    render_azure(env, config, forge_dir, return_metadata=True)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 1190, in render_azure
    return_metadata=return_metadata,
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 570, in _render_ci_provider
    os.path.join(forge_dir, forge_config["recipe_dir"]), config=config
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/variants.py", line 525, in get_package_combined_spec
    combined_spec = combine_specs(specs, log_output=config.verbose)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/variants.py", line 277, in combine_specs
    log_output=log_output)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/variants.py", line 252, in _combine_spec_dictionaries
    spec_source))
ValueError: variant config in /Users/uwe/Development/py-spy-feedstock/recipe/conda_build_config.yaml is ambiguous because it does not fully implement all zipped keys, or specifies a subspace that is not fully implemented

[mingwandroid]

@kkraus14, is there any way to get any insight into why and when the base glibc version for CUDA gets updated? It's something that creates a lot of uncertainty as things stand.

[beckermr]

@xhochy, see the PR. We have zipped docker image with cuda_compiler_version so we need this

cuda_compiler_version:                    # [linux64]
  - None                                  # [linux64]
docker_image:                             # [linux64]
  - condaforge/linux-anvil-cos7-x86_64    # [linux64]

[xhochy]

@xhochy, see the PR. We have zipped docker image with cuda_compiler_version so we need this

cuda_compiler_version:                    # [linux64]
  - None                                  # [linux64]
docker_image:                             # [linux64]
  - condaforge/linux-anvil-cos7-x86_64    # [linux64]

This needs to be added to https://conda-forge.org/docs/maintainer/knowledge_base.html#centos7 then

[beckermr]

Yup! I was about to do that. :)

[beckermr]

https://github.com/conda-forge/conda-forge.github.io/pull/1118

[kkraus14]

@kkraus14, is there any way to get any insight into why and when the base glibc version for CUDA gets updated? It's something that creates a lot of uncertainty as things stand.

It's typically tied to the supported CentOS / RHEL version and I believe that roughly aligns to the EOL for that distribution, but there's nothing formal AFAIK.

[beckermr]

I found a problem where conda is setting the build machine using the old sysroots. Patch submitted here: https://github.com/conda/conda-build/pull/3999

We'll need to patch the CI setup scripts to set this env var as well.

[jakirkham]

Is there a good way to specify the CDT's include and lib directories? Perhaps an environment variable?

[beckermr]

Is there a good way to specify the CDT's include and lib directories? Perhaps an environment variable?

What? They are in the sysroot so they should just work I thought...

[beckermr]

Error logs?

[jakirkham]

The build in particular looks in /usr by default. So we have to override the path to look at the right path specifically. Just asking to figure out how we do that.

[isuruf]

Please open a different issue. This is about supporting glibc 2.17

[jakirkham]

Right I'm trying to do this as part of a CentOS 7 build. There's no issue on CentOS 6.

[isuruf]

Please open another issue with conda list information and details on how to reproduce.

[jakirkham]

Just asking whether there is an environment variable (maybe CONDA_BUILD_SYSROOT?) or similar that points to the location of the CDTs once installed during a build. Am hoping this is an easy answer that someone here knows 🙂

[isuruf]

Yes CONDA_BUILD_SYSROOT is there, but it shouldn't be needed.

[mingwandroid]

CONDA_BUILD_SYSROOT is macOS only. I'm not opposed to setting it on Linux, but as @isuruf says its not needed. The compilers hard-code the relative path (kind-of) to the sysroot, and the sysroots are searched. If you pass e.g. -I/usr/include or -L/usr/lib then you are doing it wrong, those are not needed and prevent cross-compilation from working correctly.

[jakirkham]

Great, thanks! 😄

I'm not passing those flags, but the build's configure scripts search in /usr by default and needs to be overridden. So we are trying to wrangle it into doing the right thing. Agree it would be nice if we didn't have to (would make my life easier 😅)

[mingwandroid]

Build configure scripts should not add system paths, the compilers already know those. Good luck fixing this!

[beckermr]

OK. cos7 is done and fully shipped. I know of no more PRs that need to be made. A few need to be merged, but they are not essential. Closing!