Conda-build: can/should conda-build be smart about detecting C-level linkages?

[msarahan]

Numpy needs a strict version pin whenever a package links against the numpy C API. I think we could scan for #include <numpy.h> statements, and know when we need a numpy version pin.

This is fairly extensible to any other library - we just need to know if there's a characteristic import. Some things will be more forwards-compatible (minor version) than others (bugfix version, or exact match), so the exact pin condition should be flexible.

[frol]

I can imagine people writing Python C extensions in, for example, Rust, so it seems impractical to try guessing whether a package needs a strict version pin by searching any sources. There just must be a way to specify this in meta.yaml.

[msarahan]

Yeah, the sources definitely won't work. I think inspecting libraries after the build might, though.

[frol]

@msarahan Why don't you like the idea of explicit pinning via meta.yaml syntax?

[msarahan]

It's not that I don't like it, it has a very important place. It's also very easy to mess up, and causes lots of pain. Still, I would always give deference to a human-set value. The more information we can capture automatically, and include for conda's usage, the better we'll be able to understand why anything is not working.

This is really more for people who don't necessarily understand when pinning is necessary. I personally would find this very useful - with numpy as an example, one always has to go looking for whether a package actually uses the C API or not. Having this info automatically would be convenient - even if it only guides a human, rather than defining pinning itself.

[groutr]

A few ideas:

• CRAN metadata has a field for whether a package needs compiling or not. If a package needs compilation then we can pin build dependencies. This could be defined in meta.yaml.
• maybe we can separate actual build dependencies from tools needed to build (in the requirements/build list). This would somehow capture the notion that a package doesn't depend on cython or mercurial, but only needs it to build. Maybe this could accomplished by taking a difference between the requirements/run and requirements/build lists. If something like numpy appears in both lists, then it is likely that numpy needs to be pinned.
• inspect linkages of resulting binaries. Haven't looked at this yet, but the linkages of dynamic libraries might give a clue as to what libraries are needed.

[msarahan]

Yes, these are good ideas. Regarding your second point, though, I really want to tell people when they need pinning - not determine that the package needs pinning based on their input. I think we should be able to do this with some reasonably simple rules for each package. There aren't very many packages in our system that need this knowledge, so I think it can be manually captured and codified.

[mingwandroid]

I like these ideas.

Would requirements/build and requirements/buildtools work? Pacman has depends and makedepends.

I suppose that 'needs compiling' could be implied by listing certain tools as build dependencies, you could imagine some that are scraped for (I know our packages are not always split at this level of detail): gcc, g++, gfortran, rustc. Maybe we could have something like gcc:g++ to classify the exact nature of the use of the build tools?

For inspecting the linkages, I wonder if we could at the same time figure out that if a library appears in requirements/build but none of it's shared libraries appear in ldd, then it's been linked statically, and make not of that too. Tools like nm could also help here to weed out libraries that didn't get used at all.

[groutr]

@mingwandroid I imagine the distinction would look a lot like that.

I looked some more at inspecting linkages. I used h5py as my case study.
On windows, we can use dumpbin to inspect the dependents of a library. Unfortunately, numpy won't appear in this list. The only place I was able to find it was via dumping the raw data in the dll sections.
On OSX, nm shows some symbols that are clearly numpy specific.

@msarahan could you give an example of how such information would be codified?

[jankatins]

debian has some great tools to figure out the link time dependencies (even down to individual symbols -> only linking to old symbols will give you a dependency on >= old_version and then adding a newer symbol will result in a dependency on >= newer_version -> The symbol information is semi-automatically build during packaging time of the lib)

-> https://www.debian.org/doc/debian-policy/ch-sharedlibs.html#s-sharedlibs-depends
-> https://www.debian.org/doc/manuals/maint-guide/advanced.en.html#librarysymbols

But for that to work, conda and meta.yml would need to have some kind of BINARY_DEPENDENCY variable support and packages exporting a lib need to add some meta information what their compatibility is (e.g. "when linked to a file in this conda package, include a native dep as this_package >= 3.2" and someone (or a tool) needs to add that to all library packages during build time...

Just for reference: debian has a similar proces (but based on Requirement.txt) for python packages: http://anonscm.debian.org/cgit/dh-python/dh-python.git/plain/dh_python2.rst

[groutr]

Thanks for the info about debian. Their symbol tracking approach is interesting.

[jankatins]

I can only recommend the debian documentation on how to package apps and libs (the packaging guides, debian policy) and the tools which were developed to help that along. It's quite a lot to read, but basically every problem conda has or will get, debian has probably faced ten years ago and build a tool to solve it :-)

[stuarteberg]

My 2 cents: I like the idea of detecting problems in the user's recipe (such as forgetting to pin the numpy version). I think the appropriate response is to give the user warnings or errors, not to "fix" their recipe for them.

The ideal place for this sort of thing would be a recipe linting tool, possibly external to conda-build. I know that conda-forge has some sort of linting tool, but I don't know the details.

[jakirkham]

My 2 cents: I like the idea of detecting problems in the user's recipe (such as forgetting to pin the numpy version). I think the appropriate response is to give the user warnings or errors, not to "fix" their recipe for them.

👍

The ideal place for this sort of thing would be a recipe linting tool, possibly external to conda-build.

Depends. The initial discussion focused on acquiring the source so maybe a linting tool would not work there.

I know that conda-forge has some sort of linting tool, but I don't know the details.

This is mainly for parsing the meta.yaml. It doesn't acquire the source. It also doesn't really help detecting things like you are linking against NumPy or similar.

Basically it exists to make meta.yaml look similar across large swaths of recipes. Also, it tries to take catch of some tedious issues so that human reviewers don't have to.

[jakirkham]

I think @stuarteberg and I talked to @ukoethe about handling problematic packages before and the thought was to have a sanitizer type tool. This would be used to check what happened to the final package and make sure it met some set of expectations. It seems like such a tool is what you would be looking for here.

[jakirkham]

Will conda-verify be used to solve this problem? Whether or not it will, how does it relate to this problem domain?

cc @ilanschnell

[msarahan]

Conda-verify might be used. I think the key to this tool will be @mingwandroid's work on his python implementation of ldd. It behaves much more nicely, especially for cross-compiled packages. I'm punting on this issue until we have that tool. Ray and I have discussed some kind of warning output at the end of builds for links to system libraries outside of the few allowed ones. I imagine there will be other warnings.

[mingwandroid]

The new compiler packages we're working on pass --as-needed to the linker which means there'll be no DT_NEEDED entries for libraries that are not actually used. It would be nice to detect when this has happened and remove them from the dependencies or remove version pinning or just warn that it had happened. Clearly I am not sure which way to go here but dropping unneeded dependencies is a good goal.

[jakirkham]

Maybe of some interest, @isuruf.

[jakirkham]

Should this be closed or re-milestoned?