Nixpkgs: How to handle optional dependencies not available on the given (host) platform?

[shlevy]

Example: strace optionally depends on libunwind, which hasn't yet been ported to RISC-V. Do we want the strace build expression to explicitly say optional libunwind.supportsTarget libunwind, or do we want the libunwind expression to simply be null on unsupported systems? Or is there another alternative?

@Ericson2314 @bgamari @dtzWill (we need @nixpkgs-cross...)

[Ericson2314]

assert and tryEval, lolololol.

In seriousness though, this is yet another excellent reason to peak at the meta of invalid derivations. Then we can make some adapter for a "best effort" dependency like pkg: if pkg.meta.available then pkg else null . Let's bring in @vcunat, @oxij and the other meta check people.

[Ericson2314]

C.f. https://github.com/NixOS/nixpkgs/pull/35111 and https://github.com/NixOS/nixpkgs/pull/33057

[7c6f434c]

There is also an option to have requiredBuildInputs and optionalBuildInputs with the latter filtered through a platform-support check.

Of course, this will bring us really close to a point where we need a single buildInputs parameter with a possibility to add attributes native, optional etc.

[shlevy]

@Ericson2314 I think you misunderstood: I'm not asking how we check if something is supported, I'm asking where we check. If we check at libunwind, then strace works out of the box but we silently lose features. If we check at strace, then we explicitly opt-out of the features but have to be more verbose.

[shlevy]

@7c6f434c yeah, that's a good way to do this. @Ericson2314 can you fit that into your mkDerivation model?

[vcunat]

Peeking at meta of invalid derivations works already. EDIT: almost all attributes should work lazily, not just meta.

[Ericson2314]

@shlevy I was trying to answer that, just skipped a lot I guess.

1. We can't just make libunwind null across the board, because some packages might require it.

2. We don't want to make checking at strace too verbose.

Let's have our cake and eat it too:

# in lib
{ 
  optionalDep = dep: if dep.meta.available or false then dep else null;
}

# strace v1, shorter
{ stdenv, libunwind }:

stdenv.mkDerivation {
  buildInputs = [ (stdenv.lib.optionalDep libunwind) ];
}

# strace v2, longer
{ stdenv
, supportUnwinding ? libunwind.meta.availible or false
, libunwind
}:

stdenv.mkDerivation {
  buildInputs = stdenv.lib.optional supportUnwinding libunwind;
  /* somethingElse = ... supportUnwinding ..; */
}

[Ericson2314]

Of course, this will bring us really close to a point where we need a single buildInputs parameter with a possibility to add attributes native, optional etc.

@7c6f434c yeah, that's a good way to do this. @Ericson2314 can you fit that into your mkDerivation model?

That's a fine idea, and it fits in that I don't see affecting the core model at all. As long as we keep doing dependency propagation in bash, We'll need to convert that to basically what we have anyways.

[shlevy]

Cool, sounds good. I'll go with the optionalFoo approach until optionalInputs etc. is available.

[Ericson2314]

optionalFoo is which approach? You mean making my library helper like that?

[Ericson2314]

See also https://github.com/NixOS/nixpkgs/pull/34444 for making the meta checks work for cross.

[7c6f434c]

Is it a good idea to (eventually) handle native-ness in the same way? Or propagation?

[Ericson2314]

@7c6f434c I'm open to that. I mean I think of deps not in terms of the names but the two axes (see unstable nixpkgs manual), so the flattening that top-level attrs do is unfortunate. The limitations of bash mean that effectively we'll convert away from this stuff in the end so it must be just an easy superficial change.

[shlevy]

@Ericson2314 so I'd have foo attribute and an optionalFoo attribute that is null on unsupported systems and thus can be safely added to inputs.

Speaking of limitations of bash... We may want to take advantage of __structuredAttrs = true here.

I'm definitely in favor of inputs being a list (set?) of packages with optional tags like buildTime = true and propagated = false if we can make that work nicely.

[Ericson2314]

@shlevy errrr not to bikeshed, but I must say I like my optionalDep helper better. I don't think optionalFoo scales elegantly. If we end up making an optionalFoo attribute for every Linux-only package a Darwin package could use on Linux, for example, that's a lot of extra bloat in all-packages.nix.

CC @NixOS/darwin-maintainers

[Ericson2314]

(Renamed because I don't see this as a cross-specific issue but general portability one. Reopened waiting on the darwin people's input more than my little bike-shed above.)

[shlevy]

Right, that's better.

[7c6f434c]

@Ericson2314 I wonder how to treat noarch packages in the two-axis model (for example, fonts — and yes, they are needed in the build time).

[oxij]

Oops, I guess, the hidden SLNOS agenda behind the sequence of #33057 and #35111 got uncovered :)

We had exactly the same discussion in SLNOS (without reaching a consensus).

I made the two aforementioned PRs to ease SLNOS (and, eventually, nixpkgs) into "optionalBuildInputs" route (yep, I confess, the use of .available in relatedPackages with which I pushed .available through review was an accidental revelation, not the original purpose, the original purpose of .available was exactly this issue).

As to the essence of issue, I do want them unsupported packages to be explicit because I hate to debug those optional nulls. You should try tracing null inputs in stdenv.mkDerivation and you are going to be surprised by how many overrides with nulls you already use. It's in hundreds (well, maybe less, as those traces do have lots of duplicates, still grep -r '= null' is pretty enormous).

They are especially hard to debug when using a package with a custom callPackage or overriding an already overridden package. Anything is better than nulls. I vote for anything except nulls. Explicitly filtering with filter (p: p.meta.available) is fine for me. To be honest, even "strace v2" from above and simple

buildInputs = [
  ...
]
++ optionals stdenv.isLinux [
  ...
]
++ optionals stdenv.isDarwin [
  ...
]

are fine. But I think most people will steer away from such things as they are too verbose. Hence, I vote for optionalBuildInputs. Anything but the nulls.

To be even more honest, now I'm unsure that even meta.available is good enough. It's too permissive with all its options of config.checkMeta, config.checkMetaRecursively and config.handleEvalIssue.

Now I'm experimenting with meta.supported that only checks the platforms (I also call optionalBuildInputs platformBuiltInputs, the name is subject to change, though).

As a side note, meta.platforms should be checked (or just autogenerated, or maybe both, maybe with config.checkMetaPlatforms or something) against platforms produced by intersecting platforms of derivations from buildInputs. But I have not implemented that properly yet.

As another side note, we could have meta.hostPlatforms or something, that describes platforms you can build the derivation on by doing the same for nativeBuildInputs.

[shlevy]

Action item tracked in #36226

[7c6f434c]

Unfortunately, null provides a way to quickly remove some dependencies without adding extra support into packages. USE flags require too many changes across the tree and will be hard to negotiate; also, a global override of null ensures that success does mean «no such packages is used», USE flags with Nixpkgs callPackage require active maintenance and vigilance.