Conda-build: Requirements Section .a vs .so/.dylib/.dll

Created on 11 Jan 2019 · 5Comments · Source: conda/conda-build

The requirements-section in the user guide could maybe be improved with the following points.

I read it from the perspective of a C/C++ programmer and wonder at each point: does this apply for a) a shared lib b) a static library c) an executable. And what if a package builds all three of them?

Export runtime requirements: When it starts with "Export runtime requirements" please put in that a static library does not need "run_exports". Only dynamic/shared ones do. Also, one should write immediately that these exports are describing oneself for downstream consumers.

Strong exports are a corner-case that is absolutely too early in the docs, since users have not read yet the section below that introduces build:, host: and run:. Maybe change the order alltogether? First "Requirements section", then the additonal info on "Export runtime requirements" for dynamic packages/DSOs, etc.

Requirements section:
It misleadingly only makes comments about shared libraries which have to go into host: and must be fitting the target architecture. Both aspects are also true for static libraries and should be mentioned. Furthermore, one should state that library dependencies (both shared and static) go exclusively in host:. The current phrasing leaves open if shared library dependencies might also need to go into run:, which they don't.

As a recap for the Python-native reader, mention that binary python import modules are always shared libraries at this point.

Maybe also add explicitly a paragraph that run_exports are derived into each dependency of a dependency with the ignore_run_exports counter measure that is mentioned. Can go in the "Export runtime requirements" section that should follow now, after the "requirements section".

Thanks a lot for Q&A in the channel!

Source

ax3l

All 5 comments

This is the discussion that took place leading up to this bug report:

Axel Huebl @ax3l : Hi everyone, I am having a specific question not mentioned in the docs about dependencies
during build, I will link a static library. does it go into build: or host:?
https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#requirements-section
Isuru Fernando
@isuruf 08:47: host
Axel Huebl @ax3l 08:47: good, because Host only reads "For example, in order for a recipe to “cross-capable”, shared libraries requirements"
that should read ,sharedlibraries requirements ... , so that the shared (?!) libraries that get linked are ones for the target platform, rather than the native build platform then
super confusing
also static libraries that get linked must match the target platform
Ray Donnelly @mingwandroid 08:49: removing shared is a good idea.
Axel Huebl @ax3l 08:50: several times in this paragraph, I am just not confident enough to PR it
Ray Donnelly @mingwandroid 08:50: even being explicit?
Axel Huebl @ax3l 08:50 explicit?
Ray Donnelly @mingwandroid 08:50: so that libraries (both static and/or shared)
.. for example
Axel Huebl @ax3l 08:50: also very good
if it's that what it says
ok, now the trickier setup:
now a library let's call it adios, that itself depends on 2 shared and one static library. does it make a difference? Do the shared go in host: and run: and the static one only in host:?
Ray Donnelly @mingwandroid 08:57:all libraries go in host!
transitive, non transitive, shared, static.
the only exception is CDTs and they always go in build
Axel Huebl @ax3l 09:02: alright for host. now the run section docs, which only mention python packages.
would a static and/or a shared dependency also go in run:?
Axel Huebl @ax3l 09:03: because, also in inverse order for the reader, all the export_run and weak/strong exporting is mentioned before the requirements section in the docs. that is also confusing
Axel Huebl @ax3l 09:33:

    all libraries go in host!

exclusively in host:? Additional to that, do any dependent libraries go in run:? Do shared libraries also go in run: but static do not?
Ray Donnelly @mingwandroid 09:33: @ax3l: If you were to file bug reports I'll see if we can assign them to the person looking after docs.
Yes, all except CDTs go in host.
Axel Huebl @ax3l 09:33: sure, just want to make sure I don't miss something.
Ray Donnelly @mingwandroid 09:34: In the perfect scenario run_exports obviate the need for putting things in run entirely.
Axel Huebl @ax3l 09:34: What is the "perfect scenario" in terms of a static or shared lib?
Ray Donnelly @mingwandroid 09:34:
Not sure what you mean ..
Jonathan J. Helmus @jjhelmus 09:35: packages with static libs should not have a run_export, those with shared libs should
Ray Donnelly @mingwandroid 09:35: the only (minor IMHO) gotcha is that if a package carries both shared and static libs and declares run_exports and you explicilty force that the static libs get linked to instead of the shared ones, then you need to use build/ignore_run_exports to tell conda-build about that.
Axel Huebl @ax3l 09:36: you mean if a package carries both a shared and static lib of the same dependency?
Ray Donnelly @mingwandroid 09:36: but conda-build has post checks now (--error-overlinking and --error-overdepending) that detect all of these issues and errors out.
Axel Huebl @ax3l 09:36: yes, I try to fix the warnings currently
Ray Donnelly @mingwandroid 09:36: well, just if a package contains both. maybe carries is a bad choice of words here.
I just mean e.g. both lib/libncurses.a and lib/libncurses.so are part of the package.
nothing to do with deps here
(deps of ncurses anyway!)
Axel Huebl @ax3l 09:37:

    packages with static libs should not have a run_export, those with shared libs should

ok, and how are for shared and static libs that live in host: their necessary exports determined? Automatically detected or shall I add the shared ones manually to run/run_exports?
Ray Donnelly @mingwandroid 09:38: the dependency packages themselves are enquired for their run_exports.
in the most simple case, a run_export is usually declare in package A to package A.
so ncurses.tar.bz2 has a file called info/run_exports and that file lists ncurses as a 'weak' run_export ..
Axel Huebl @ax3l 09:38: aha, that makes sense now. and it can never be downgraded, besides by hacking it with ignore as I saw in the docs
Ray Donnelly @mingwandroid 09:39: then whenever conda-build sees ncurses in host, it automatically copies across the run_exports from ncurses into the run section of this new package it's building.
Axel Huebl @ax3l 09:40 excellent.
Ray Donnelly @mingwandroid 09:41: now there's 2 more complicated things after that, "strong" run_exports (this means they can be carried from build to run => libstdc++ for example), and also you can delare the run_exports in package A to contain package(s) other than package A (again libstdc++ is an example, here the C++ compiler package has a strong run_export of libstdc++)
but those are more expert-level features I guess, only people hacking on cross build tools need to worry about strong run_exports.
BTW, if we are not using --error-overlinking and --error-overdepending at present we really should!
Axel Huebl @ax3l 09:43: that means if I build a static library, I do not need to declare run_exports for itself.
even if the static library depends on shared libraries, since they themselves already declare that they are run exports so I just have them in its host:.
Ray Donnelly @mingwandroid 09:44: A static library has no capability to 'autoload' a shared library (except on Windows)
so it's mostly a moot point, on macOS and Linux it's the responsibility of the package's build system metadata (.pc file mostly) to trasmit that information to the final linker command line.
Axel Huebl @ax3l 09:45: autoload? it's just an archive that might depend on a shared lib when it's used in an app/other lib.
yes, exactly
Ray Donnelly @mingwandroid 09:45: autoload => i.e. uses the operating system's loader's DSO loading mechanism.
Axel Huebl @ax3l 09:45
yes, understood.
Ray Donnelly @mingwandroid 09:46: DT_NEEDED on Linux and LC_LOAD_DYLIB on macOS.
if some package explicitly loads a DSO (dl_load / LoadLibrary) and depends on this to work then it should add the package(s) from which these DSOs come as run exports explicitly.
Axel Huebl
@ax3l 09:47: if I build a shared library now, I do declare myself in run_exports: (weak) and that's it for the default userland
Ray Donnelly @mingwandroid 09:47: that'd be more or less the same as a plugin system where some plugins are necessary.
IMHO any package with DSOs that are meant to be used by other packages should always declare weak run_exports.
Axel Huebl @ax3l 09:48: and static libraries just do nothing
Ray Donnelly @mingwandroid 09:49: Yeah, if you have a project that only has static libs then no run_exports at all, and no need to use build/ignore_run_exports.
Axel Huebl @ax3l 09:49: k
Ray Donnelly @mingwandroid 09:49: we're optimising here for the most common case of preferring shared libs at a small cost to those who do not prefer them