Nixpkgs: modified `findInputs` breaks backward compatibility

[sjmackenzie]

Issue description

findInputs no longer returns the propagated inputs of a derivation

Steps to reproduce

• please execute the below nix expression

with import <nixpkgs> {};
let
  deriv = name: props: stdenv.mkDerivation {
    name = name;
    phases = [ "installPhase" ];
    propagatedBuildInputs = [ props ];
    installPhase = ''
      mkdir -p $out/nix-support
      for i in $propagatedBuildInputs; do
        echo PROP-IN-FOR-$name
        echo $i
        echo $i >> $out/nix-support/propagated-build-inputs
      done
    '';
  };
  A = deriv "A" B;
  B = deriv "B" C;
  C = deriv "C" nix;
  A1 = deriv "A1" B1;
  B1 = deriv "B1" C1;
  C1 = deriv "C1" nix;
  derivs = [ A A1 ];
in
  stdenv.mkDerivation {
    name = "test";
    phases = [ "installPhase" ];
    propagatedBuildInputs = derivs;
    installPhase = ''
      mkdir $out
      propagated=""
      for i in $propagatedBuildInputs; do
        echo PROP-OUT-FOR-$name
        echo $i
        local p
        findInputs $i p propagated-build-inputs
        propagated="$propagated $p"
      done
      echo PROPAGATED
      echo $propagated
    '';
  }

Expected behaviour

echo $propagated (the last line of the above code) lists all the progagated inputs

Actual behaviour

• The second last line of the output contains two derivations pointing to A

$ nix-build test.nix --show-trace
these derivations will be built:
  /nix/store/18d3zfgxgwnmzggk9dsb7vgb2hvwg9di-C1.drv
  /nix/store/x8ygbxngihwyr3xl0jc71yvba3d23y8c-C.drv
  /nix/store/2pc9fs958psarpnfksrzi7vnjd0m90pl-B.drv
  /nix/store/3dhdw7y5qf7j71piw48insdh15x7c77i-B1.drv
  /nix/store/f94bzzh54d0dg5zlpyi9q0j3gfw3c7l0-A1.drv
  /nix/store/icwqmrjf95hv9g8xfsrijp05qr8906ls-A.drv
  /nix/store/vvzbrccjmh43vxnj6b5fsm4nip4fwghj-test.drv
building path(s) ‘/nix/store/rf6fvwhbqjkdr9ki7x1yl89qvj06ixyz-C’
building path(s) ‘/nix/store/p19i3r07lap9hqz386givgzpbb1da6j2-C1’
installing
PROP-IN-FOR-C
/nix/store/mkmm32n05qg9nhssdb4ldpam17fj0yky-nix-1.11.13-dev
installing
building path(s) ‘/nix/store/1dh3h878fny24czzpwf7nbsv0chnn7q2-B’
PROP-IN-FOR-C1
/nix/store/mkmm32n05qg9nhssdb4ldpam17fj0yky-nix-1.11.13-dev
building path(s) ‘/nix/store/clyi5jd9j8v3kgysjh5cx009945k8jma-B1’
installing
PROP-IN-FOR-B
/nix/store/rf6fvwhbqjkdr9ki7x1yl89qvj06ixyz-C
building path(s) ‘/nix/store/f52xkncbdvfc03qg3y88ydlvp5426ks5-A’
installing
PROP-IN-FOR-B1
/nix/store/p19i3r07lap9hqz386givgzpbb1da6j2-C1
building path(s) ‘/nix/store/mx0cf07xwjhmmqv1iw4qdy9g6dlz8qn9-A1’
installing
PROP-IN-FOR-A
/nix/store/1dh3h878fny24czzpwf7nbsv0chnn7q2-B
installing
PROP-IN-FOR-A1
/nix/store/clyi5jd9j8v3kgysjh5cx009945k8jma-B1
building path(s) ‘/nix/store/3si4ciki7j1yaklvrwmqfibrwhflb5pv-test’
installing
PROP-OUT-FOR-test
/nix/store/f52xkncbdvfc03qg3y88ydlvp5426ks5-A
PROP-OUT-FOR-test
/nix/store/mx0cf07xwjhmmqv1iw4qdy9g6dlz8qn9-A1
PROPAGATED
/nix/store/f52xkncbdvfc03qg3y88ydlvp5426ks5-A /nix/store/f52xkncbdvfc03qg3y88ydlvp5426ks5-A
/nix/store/3si4ciki7j1yaklvrwmqfibrwhflb5pv-test

Technical details

• System: (NixOS: 17.09pre113138.96457d26dd (Hummingbird))
• Nix version: nix-env (Nix) 1.11.13
• Nixpkgs version: 17.09pre113138.96457d26dd
• Sandboxing enabled: build-use-sandbox = false

[matthewbauer]

I think this is what was causing #27873.

/cc @Ericson2314

[sjmackenzie]

yeah it's completely broken my project Fractalide :-(

[Ericson2314]

It accumulates in an array now. Let me hop on computer to fix example.

[Ericson2314]

let
    installPhase = ''
      mkdir $out
      propagated=""
      for i in $propagatedBuildInputs; do
        echo PROP-OUT-FOR-$name
        echo $i
        findInputs $i propagated propagated-build-inputs
      done
      echo PROPAGATED
      echo ''${propagated[@]}
    '';

[Ericson2314]

But note that findInputs really isn't a good function to reuse---it has a bunch of side effects, and if you use it with different vars than before, setup hooks will be run twice which they generally aren't made to be.

Furthermore in https://github.com/NixOS/nixpkgs/pull/26805 I make even more changes to it. Maybe I should rename it, and add an old-style pure findInputs as a utility function?

[sjmackenzie]

Could backward compatibility be respected please? User space is now broken, with this mentality it means building on nixos is a complete nonstarter.

[globin]

Hmm, I'd personally call this an implementation detail and internal API, I'd prefer adding a utility function

[Ericson2314]

@sjmackenzie I don't understand

Could backward compatibility be respected please?

No one is trying to break things, but it's hard with nixpkgs because there is no agreement as to which interfaces are public/stable.

User space is now broken, with this mentality it means building on nixos is a complete nonstarter.

What userspace? Unless I'm missing something, all uses of findInputs in nixpkgs at least have been upgraded. You mean of Fractalide?

[sjmackenzie]

findInputs is being used by the public therefore it automatically becomes user space, unless ofcourse it's documented as a draft function subject to change, then it's my prerogative to keep tracking the bleeding edge. One wants to encourage more projects to build on nixos. Just because it's a small function documented in the public facing manual doesn't mean one can hand wave it away as being internal and therefore may be broken at will. To see how critical this is imagine linux just changed some api and broke nix code. This is a severe violation and I hope it doesn't become an insular-shoot-from-the-hip culture in nix just because everything "we use" is in nixpkgs so be damned with anything else!

[sjmackenzie]

@Ericson2314 anyway I'm pleased to say your amended example works. So I'm going to close the issue and thanks for your help.

[vcunat]

Just because it's a small function documented in the public facing manual doesn't mean one can hand wave it away as being internal and therefore may be broken at will.

It's a function inside stdenv that is not documented in the manuals, and it clearly feels an internal function to me. (Well, point me to the point where it is documented, but I did search really hard.)

BTW, if you want nixpkgs with minimal changes, we have the stable branch exactly for that purpose.

[sjmackenzie]

@vcunat Ah indeed you're right, I conflated findInputs with the documented propagatedBuildInputs in https://nixos.org/nixpkgs/manual/index.html

Anyway, it seems I'm using internal stuff, is there an appropriate means to find dependent derivations that is "public facing and stable"?

[Ericson2314]

There isn't, which is why I want to add one / change findInputs itself to not have side effects.

[sjmackenzie]

I see, okay perfect, thanks for this @Ericson2314

[Ericson2314]

If you have any ideas how to change https://github.com/NixOS/nixpkgs/pull/26805 to do that, would be glad to hear it!

[sjmackenzie]

I do have an idea, keen to hear your suggestion.

See these three lines https://github.com/sjmackenzie/fractalide/blob/master/support/unifySchema.nix#L13-L15

The unifySchema.nix file composes user defined capnproto schema snippets into one file to be compiled by the capnproto compiler. If there are duplicate snippets used in arbitrarily deep composition trees the capnproto compiler will complain and not build the composed schema.

The $edges array essentially points to a list of top level capnproto schema snippet compositions. So instead of iterating over $edges I'd like to simply pass $edges into findInputs i.e. findInputs $edges propagated propagated-build-inputs and I'll get a unique list of derivations despite some capnproto schemas using duplicate capnproto schema snippets.

Using the Original Post as an example: say derivation A depends on B and B1 and derivation A1 depends and B1 by iterating over $derivs (which consists of A and A1) I'll get a duplicate B1. Hence it would be super if I could just pass in $derivs to findInputs and findInputs takes care of uniqueness. Hence removing side effects.

Currently if I use findInputs $edges propagated propagated-build-inputs in fractalide I get this error:

$ nix-build -A nodes.fvm_rs_find_node
these derivations will be built:
  /nix/store/47jayd8fh4zhv5w1m2bni6z1z2nzi2fi-fvm_rs_find_node_rs.drv
  /nix/store/vyng5m0gg06hhaxcz9gx851axw5s3a9q-rustfbp.drv
  /nix/store/n50np4c3ffjqvsnx5ry665yn7bq34mym-fvm_rs_find_node.drv
building path(s) ‘/nix/store/s4wnd56nzya77h4l3y0y5vlaiqq5dlqw-fvm_rs_find_node_rs’
building path(s) ‘/nix/store/fizshvhqpj98db1sz0mi7axanqlzpw9p-rustfbp’
building
/nix/store/4ga70wn23hh5k2bsi7d56wlkfz4s3dji-stdenv/setup: line 296: /nix/store/1q1w1qnsd745yzyj6l9x48a16n34amaa-FsPathOption[*]: bad substitution
builder for ‘/nix/store/47jayd8fh4zhv5w1m2bni6z1z2nzi2fi-fvm_rs_find_node_rs.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/n50np4c3ffjqvsnx5ry665yn7bq34mym-fvm_rs_find_node.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/n50np4c3ffjqvsnx5ry665yn7bq34mym-fvm_rs_find_node.drv’ failed