Nixpkgs: Derivation inputs determined by another derivation? (metaprogramming question)

[lukego]

Question about Nix programming style which I _hope_ is on topic for this repo! (Or maybe I should have posted to nixos/nix?)

Is there a straightforward idiom for defining derivations whose inputs are calculated dynamically e.g. where one derivation determines the inputs to another?

For example suppose that I have this file at url http://foo.com/foo.yaml:

sources:
  - http://foo.com/a.tar.gz
  - http://foo.com/b.tar.gz
  - http://foo.com/c.tar.gz

and now I want to write a nix expression that first "manually" fetches http://foo.com/foo.yaml and then also "automatically" fetches each of the sources that are listed in that file?

The use case is processing log files with nix. The simple case is to process the log from one process (e.g. a.tar.gz). The hard case is making it possible to define a set of processes whose logs should be processed in combination (that's what foo.yaml is for.)

[abbradar]

There is a feature that does just that, named "import from derivation": you can do import "${foo}/bar.nix".

EDIT: But there is a problem if the remote file changes dynamically. IFD works if you change the checksum, but in case you essentially need to check if the file changed each time you can't do this in Nix currently. Intensional strore model would help I imagine, but it's a long way to go.

EDIT2: Actually you can! builtins.fetchurl "http://foo.bar" will do just what you want (download the file each time).

[copumpkin]

Except builtins.fetchurl caches and is hard to force-expire the cache. See https://github.com/NixOS/nix/issues/1223

[lukego]

Thanks for the info guys. Let me just check if I understand correctly before closing this issue.

There is a feature that does just that, named "import from derivation": you can do import "${foo}/bar.nix"

Neat. So in my example I could have a first derivation (d1) that downloads the foo.yaml file and transforms it into a nix expression listing some sources, then a second derivation (d2) that loads this from the output of the first?

let d1 = mkDerivation { src = fetchurl "http://foo.com/foo.yaml"; ... };
    d2 = mkDerivation { src = import "${d1}/inputs.nix"; ... }; in

and this would be perfectly valid nix code and nothing to be ashamed about? :-)

[abbradar]

@lukego We try to avoid it in nixpkgs because this makes determining and printing build plan impossible (Those will be built...., those will be fetched....), but that's completely okay in general!

EDIT: I meant complete build plan: when IFD, readFile-from-derivation and other functions like this are used Nix first builds all those "imported" derivations and then makes the remaining plan.

[lukego]

Thank you for the very helpful explanations!