Conda-forge.github.io: Packaging CDTs

Created on 1 May 2019 · 44Comments · Source: conda-forge/conda-forge.github.io

Have recently been running into some use cases where we are going to want to package some additional CDTs outside those already in defaults. One of the questions this raises is where these CDTs live? Should they be added to AnacondaRecipes' aggregate repo? Should they be added as feedstocks here? Or is there some other way we should be handling these? Thoughts?

Discussion

Source

jakirkham

Most helpful comment

Had an offline chat with @jjhelmus and I agree with getting the CDTs in conda-forge. We need just to coordinate more carefully when (if) we move away from CentOS 6.

ocefpaf on 3 May 2019

👍2

All 44 comments

cc-ing @mingwandroid (as he probably has thoughts on this 🙂)

jakirkham on 1 May 2019

I think having them as feedstocks is fine as defaults and conda-forge use docker images based off the same centos image.

isuruf on 1 May 2019

Personally I done mind where they are stored. Best they're not duplicated, so I'm in favor of moving them to conda-forge. Ping @msarahan, @jjhelmus, @nehaljwani, @csoja.

mingwandroid on 2 May 2019

I would be fine with CDT recipe being created and built by conda-forge. If needed in defaults the feedstocks could be forked to the AnacondaRecipes organization in the same manner as other feedstocks are now.

jjhelmus on 2 May 2019

I agree that we should not duplicate, have a canonical place for them, and we must keep them consistent. With that said, what are the advantages of having them in conda-forge? They are pretty "low-level" and something that we don't want to update frequently and risk getting out of sync with defaults.

My opinion is, ever since we got the strict channel option, leave all the slow moving low level stuff in defaults. Let conda-forge be about the fast moving latest releases that users want to publish/have control over. It is not a strong opinion, just a practical one based on the time we would have to maintain them and the chances of getting them out-of-sync with defaults.

ocefpaf on 2 May 2019

FWIW I have no strong feelings about where the existing CDTs live. Am mainly curious about where new CDTs live. That said, I understand the benefits of having them all in one place.

We can certainly ask defaults to package new CDTs for us, but this can be a bit challenging as people working on packages for defaults may already have their hands full; whereas, users here may need particular CDTs today. Packaging them in conda-forge would be one way to address this issue. It also has some nice scaling benefits.

If there is a way to resolve this issue in defaults, I'd be open to discussing that as well. 🙂 It just may be more practical to handle this in conda-forge.

jakirkham on 2 May 2019

@jjhelmus and @mingwandroid would you be opened for us to send a PR to AR with the new CDTs or should we just move everything to CF? I have to confess that the lowest effort path for CF would be the former, with the advantage that we would neve publish something that may go stale and break our stack when using strict.

ocefpaf on 2 May 2019

Sending PRs are only one part of the problem. The other part is reviewing/merging them and building/uploading the packages. How does this get prioritized?

jakirkham on 2 May 2019

I think a better course of action would be for conda-forge to build CDT packages. I have a hard time seeing a benefit for CDT packages being added to defaults if they are only used by conda-forge and I can't commit to timely updates since other priorities can (and do) come up. By building CDTs itself conda-forge can control its own timeline and cater to its own needs.

I do not foresee problem with these packages going stale. The recipes are generated from conda skeleton and the contents of the packages are pulled from CentOS 6 RPM which rarely change.

jjhelmus on 2 May 2019

👍2

Had an offline chat with @jjhelmus and I agree with getting the CDTs in conda-forge. We need just to coordinate more carefully when (if) we move away from CentOS 6.

ocefpaf on 3 May 2019

👍2

Makes sense. FWIW EOL of CentOS 6 is 30 November 2020. So it is something we want to prepare for.

ref: https://wiki.centos.org/About/Product

jakirkham on 3 May 2019

Makes sense. FWIW EOL of CentOS 6 is 30 November 2020. So it is something we want to prepare for.

Good to know. We should probably create the new CDTs and test them before 2020.

ocefpaf on 3 May 2019

Hopefully before we move beyond centos6 (I would love to get peoples' feelings for when that should happen!) we will have begun recording minimum required versions of our system dependencies.

mingwandroid on 3 May 2019

👍1

(I would love to get peoples' feelings for when that should happen!

IMO, now :smile:

ocefpaf on 5 May 2019

So the next question is how we decide when to use a CDT or build a Conda package from source? Will this be case by case or are there general recommendations we can provide?

jakirkham on 6 May 2019

That is a tricky one. We share GL at present and defaults has X11 too.

Mark wants libusb (I do too). @hmaarrfk, did you try to build a from-source package?

edit: Some random nonsense from my clipboard!

mingwandroid on 6 May 2019

The issue with building libusb from source is that it doesn't provide support for hotplugging USB devices. Not sure what that means, I have little time to test this, but it is my understanding that in certain cases, you may loose certain features of low level packages this way.

My solution has been to:
https://github.com/ramonaoptics/libusb-feedstock/

but notice that there is no "run exports" on purpose. I depend on that package when I need to build something, and on Ubuntu 18.04, the only platform I support, it kinda just works with system libraries.

hmaarrfk on 6 May 2019

Maybe this doc provides clarity on what hot plugging means. Do you know where it said hot plugging was disabled?

jakirkham on 6 May 2019

I think this is the only thing I read
https://github.com/libusb/libusb/blob/master/configure.ac#L112

hmaarrfk on 6 May 2019

Looking at the src.rpm from Centos 7, it seems that they don't build with any special flags

%build
%configure --disable-static --enable-examples-build
# Parallel builds seem to be broken
make
pushd doc
make docs
popd

hmaarrfk on 6 May 2019

Maybe we need a libudev CDT then? There are some other use cases for this CDT. FWIW NixOS builds libusb and so does Homebrew. So I agree with Ray there’s value in building this one from source.

jakirkham on 6 May 2019

👍1

right, i compile with --diable-udev, so the libudev is the right CDT in this case.

hmaarrfk on 6 May 2019

To start off, have added PR ( https://github.com/conda-forge/staged-recipes/pull/8560 ) for a CDT of libnl. Perhaps we can shape this to match what we want CDT recipes in conda-forge to look like. 😉

jakirkham on 14 Jun 2019

It would nice if the meta.yaml had some way of declaring itself a CDT (extras: cdt: true?) so that the linter could skip some of the spurios error messages

scopatz on 18 Jun 2019

We put several fixes into conda-build, which should mitigate the linter complaints greatly. Though there still are a few. The main ones are...

Checksums don't exist for source RPMs ( https://github.com/conda/conda-build/issues/3552 )
Tests
1. Need to be added ( https://github.com/conda/conda-build/issues/3550 )
2. Adding tests fails ( https://github.com/conda/conda-build/issues/3553 )
Including license file ( https://github.com/conda/conda-build/issues/3568 )

Though 3 is not raised by the linter, it would be good to solve. It's unclear whether this will be trivial (RPM has the license file somewhere) or more complicated (need to unpack original source and search for it).

Additionally we may need to clean up the skeleton a bit to accommodate these changes and better handle other OSes and architectures. ( https://github.com/conda/conda-build/issues/3556 ) There are also some subtle issues with copying the CDT components over into the right path that would be good to straighten out. ( https://github.com/conda/conda-build/pull/3531#issuecomment-502187457 )

jakirkham on 25 Jun 2019

I think 1. should be a blocker for all the CDT PRs. We don't allow any other package to be merged without a checksum and CDT recipes are not special.

isuruf on 26 Jun 2019

Raising this here as there are multiple PRs. license_file for GPL, BSD and many other licenses is required and we cannot simply ignore it. There was a discussion with @scopatz somewhere, but can't seem to find it.

isuruf on 28 Jun 2019

Yeah, it is basically only not required for public domain and WTFPL (and maybe CC0?)

scopatz on 28 Jun 2019

Maybe we should move the license file discussion into issue ( https://github.com/conda/conda-build/issues/3568 ).

jakirkham on 28 Jun 2019

I'm interested in getting these CDTs merged (listed below). Have another set incoming that are waiting on these. I acknowledge there is room for improvement in conda-build and our generation of these CDTs and am happy to continue pushing on that (as I have been). Though would like to get these CDTs through so we can do some more interesting work for the community. Namely high speed network communication, which will be useful for getting InfiniBand speeds. Any thoughts on how we can move these forward? Thanks in advance. 🙂

xref: https://github.com/conda-forge/staged-recipes/pull/8630
xref: https://github.com/conda-forge/staged-recipes/pull/8631
xref: https://github.com/conda-forge/staged-recipes/pull/8632
xref: https://github.com/conda-forge/staged-recipes/pull/8633

jakirkham on 2 Jul 2019

Manually package the license file?

isuruf on 2 Jul 2019

Is that the only blocker @isuruf? If I fix that, would you be willing to merge?

jakirkham on 2 Jul 2019

That and testing. There's a workaround at https://github.com/conda-forge/staged-recipes/pull/8630/commits/992682c1cff2997d6b07734fa695468ceb37a031
If those 2 issues are fixed, I'll merge.

isuruf on 2 Jul 2019

👍1

Thanks for taking the time to find a workaround for testing. Ok will make those changes and update here.

jakirkham on 2 Jul 2019

I've updated all the recipes. Should be ready for another review. 🙂

In one case the library itself doesn't include a license file. So unfortunately was unable to include it.

jakirkham on 2 Jul 2019

@isuruf, can I ask you to merge the ones that you are ok with?

jakirkham on 2 Jul 2019

Done. I didn't merge numactl. Let me know if there are any more PRs ready.

isuruf on 2 Jul 2019

Thanks. Understood. Will do.

jakirkham on 2 Jul 2019

Ok, @isuruf, these 2 are now also ready.

jakirkham on 3 Jul 2019

Once this is completed, the cdt section in the docs will be outdated: https://conda-forge.org/docs/maintainer/knowledge_base.html#core-dependency-tree-packages-cdt.

I think it would be helpful to list all the dependencies that with these changes now can/should be satisfied with CDT packages.

croth1 on 3 Jul 2019

I agree with @croth1!

scopatz on 4 Jul 2019

XREF: https://github.com/conda-forge/staged-recipes/pull/8739

hmaarrfk on 7 Jul 2019

@isuruf, could you please give PR ( https://github.com/conda-forge/staged-recipes/pull/8632 ) another look? I think I've addressed your concerns there. 🙂

jakirkham on 16 Aug 2019

I think time has answered all of the questions / issues here.

conda-forge now builds all of its CDTs outside of feedstocks.
We have included licenses for all of them.
We have checksums for the binaries and are not packaging source RPMs.
The testing issues have been resolved.
The docs have been updated.
We now use strict channel priority when building and so only use our CDTs and not those on defaults.

I am going to close this issue. Feel free to reopen!