Nixpkgs: Documentation for mkDerivation

[chris-martin]

After a year of using NixOS, stdenv.mkDerivation is still something of a mystery to me. I only know how to use it by copying examples from other packages.

Is there documentation somewhere - most importantly, a list of _all_ of the arguments that mkDerivation accepts, and what each of them does?

(And if it does already exist, how can we make it more discoverable?)

I can see some of them in pkgs/stdenv/generic/default.nix:

  # Add a utility function to produce derivations that use this
  # stdenv and its shell.
  mkDerivation =
    { buildInputs ? []
    , nativeBuildInputs ? []
    , propagatedBuildInputs ? []
    , propagatedNativeBuildInputs ? []
    , crossConfig ? null
    , meta ? {}
    , passthru ? {}
    , pos ? null # position used in error messages and for meta.position
    , separateDebugInfo ? false
    , outputs ? [ "out" ]
    , __impureHostDeps ? []
    , __propagatedImpureHostDeps ? []
    , sandboxProfile ? ""
    , propagatedSandboxProfile ? ""
    , ... } @ attrs:

But clearly that arg list isn't complete - for example, it doesn't contain src, or any of the build phases.

[dezgeg]

Most of it is documented in the nixpkgs manual (one of the three manuals we have): https://nixos.org/nixpkgs/manual/#chap-stdenv

[domenkozar]

And partially in: http://nixos.org/nix/manual/#ssec-derivation

I think we should do two things to improve this:

• review existing documentation and add/extend parts that are not clear enough
• link between the two sections explaining derivation and mkDerivation for better UX
• check if all arguments are documented and add those missing

[cleverca22]

some of the arguments arent handled at the nix level, but at a bash level at build-time
src for example gets handled by the unpackPhase function in https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh#L548

all arguments passed to mkDerivation also become environment variables that exist during the build

[CMCDragonkai]

So I'm confused why the manual itself says that buildInputs is confusing "A less confusing solution to this would be nice.".

Is it safe to say that statically and dynamically linked code would be used at the runtime of the package. What about deps that just need to provide executables on the PATH on the runtime? Should this then be in the propagatedBuildInputs?

[LorenzoFernando]

Echoing the concerns introduced by chris-martin, it's still pretty difficult to get a sense of a full capabilities of mkDerivation even after reading the documentation linked at the top of [1].

I'm willing to help write documentation for this, but so far I've been unable follow the code to the complete list of arguments. Can someone point me in the right direction? Thank you!

https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix

[veprbl]

At this moment we have at least three chapters of the nixpkgs manual devoted to this:

• https://nixos.org/nixpkgs/manual/#chap-stdenv
• https://nixos.org/nixpkgs/manual/#chap-meta
• https://nixos.org/nixpkgs/manual/#chap-multiple-output

They describes the arguments that stdenv.mkDerivation processes during evaluation and also those handled in bash by the setup.sh[1]. Then, there are infinite possible extensions that can be dynamically loaded. If some derivation sets, for example, a dontUseCmakeConfigure attribute, it's not the stdenv that knows how to handle it, it's the setup hook of cmake that is loaded by stdenv that interprets the value of the $dontUseCmakeConfigure environment variable. And pretty much the rest of the manual is devoted to documenting these extensions, and many of them are covered reasonably well, I think.

There are also user-provided extensions. And, indeed, there is no way to discover all loaded or available extensions and their options by any means other than just reading the code. I don't think many people are bothered too much by this. Once one gets a hang of the stdenv basics, then the code becomes the documentation. We do, for example, now render a (complete?) list of functions in lib sourced from the code. However, unlike functions from the lib, the hooks are defined in bash instead of nix, so they would be a bit harder to render directly into the documentation.

What I'm saying is that the documentation is now much better than it was in 2016, so I propose we close this issue as successfully resolved.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
[2] https://github.com/NixOS/nixpkgs/blob/124ec9c18e557e16109d77b8e93f75e3fe4a12a0/pkgs/development/tools/build-managers/cmake/setup-hook.sh#L118-L121

[LightAndLight]

I disagree that this issue should be resolved, having stumbled upon it today when trying to write a derivation. I'm familiar with nix but don't use mkDerivation often, so I needed a refresh.

I think that the documentation as it stands, while extensive, is not designed to efficiently convey the necessary information.

Here is an example I hacked together that's equivalent[^1] to https://nixos.org/nixpkgs/manual/#sec-using-stdenv, but I find much easier to consume. What do you think of this style of presentation?

[chris-martin]

@LightAndLight, I like where you're going with that revision. I think it's important for "mkDerivation" to appear in the table of contents to make it more discoverable.

I agree that there's a lot more information there than there used to be - I also agree that the documentation has room for improvement - but I feel like my original complaint has been addressed and remaining problems might be better refocused in a new issue if someone would like to take charge of leading this effort to better perfection.

[domenkozar]

I don't think this has been addressed at all.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/use-buildinputs-or-nativebuildinputs-for-nix-shell/8464/1