Nixpkgs: builtins.toPath is misbehaving

[adnelson]

I want to have a function which can produce a path, in order to reduce a lot of boilerplate. But when I build a string dynamically at runtime, instead of hard-coding in a path, it fails to "notice" when the directory changes. It doesn't look at the directory to see if it's changed; it only looks at the AST and if the directory contents changes, it will not pick up that change.

Let's say I have a folder in /home/anelson/project. Now let's say I have a default.nix that looks like:

# /home/anelson/project/default.nix
with import <nixpkgs> {};
buildPythonPackage {
  name = "test";
  src = ./.;
}

Now if I build it twice, it will correctly cache, and if I modify the folder, it will correctly rebuild:

/home/anelson> nix-build project # build first time
these derivations will be built:
  /nix/store/blahblah
unpacking sources
[...]
/nix/store/somehash

/home/anelson> nix-build project # Nothing changed, should cache
/nix/store/somehash

/home/anelson> touch project/newfile # modify the folder

/home/anelson> nix-build project # should rebuild
these derivations will be built:
  /nix/store/blahblah
unpacking sources
[...]
/nix/store/somenewhash

Yay! But, if I do the same thing with a string, this doesn't work:

# /home/anelson/project/default.nix
with import <nixpkgs> {};
buildPythonPackage {
  name = "test";
  src = builtins.toPath "/home/anelson/project";
}

It will build once, but no more after that, unless I garbage collect.

/home/anelson> nix-build project # build first time
these derivations will be built:
  /nix/store/blahblah
unpacking sources
[...]
/nix/store/somehash

/home/anelson> nix-build project # Nothing changed, should cache
/nix/store/somehash

/home/anelson> touch project/newfile # modify the folder

/home/anelson> nix-build project # should rebuild
/nix/store/somehash

How can I get around this? It's really tedious to hard-code in all of the paths. I want to be able to generate the paths declaratively. For example, I have a function which roughly looks like this:

{repo_dir}:
{
  buildPkg = {repo, pkg_name, ...}@attrs: 
    buildPythonPackage (attrs // {
      name = pkg_name;
      src = builtins.toPath "${repo_dir}/${repo}/${pkg_name}";
    });
}

But as it is, I can't rely on changes being pulled in when developing on my local machine. How can I get around this?

[lethalman]

cc @edolstra

[edolstra]

This might work:

src = /. + "${repo_dir}/${repo}/${pkg_name}";

[lethalman]

@edolstra but what's the reason it doesn't with toPath ?

[adnelson]

@edolstra That totally worked! But I'm also confused as to why toPath doesn't work that way... what does it even do? :|

[edolstra]

Nobody knows what toPath was supposed to do (there is just a comment in the source about it being possibly obsolete). And it doesn't convert to a path but to a string...

[vcunat]

@edolstra: so let's mark toPath as deprecated in nix manual, agreed?

[jagajaga]

@vcunat maybe let's change toPath to a working one?

[vcunat]

Or at least document clearly what it's doing. Changing semantics might break existing code.

[adnelson]

Hadn't looked at this for a while, but it's worth noting that toPath is barely used in nixpkgs:

grep -r toPath nixpkgs
nixpkgs/nixos/lib/eval-config.nix:                   in if e == "" then [] else [(import (builtins.toPath e))];
nixpkgs/pkgs/development/r-modules/cran-packages.nix:MC2toPath = derive { name="MC2toPath"; version="0.0.16"; sha256="0jdn9wpxavn2wrml907v23mfxr62wwjdh7487ihjj59g434ry7wh"; depends=[RNetCDF]; };
nixpkgs/pkgs/os-specific/darwin/apple-source-releases/default.nix:    in callPackage (./. + builtins.toPath "/${namePath}");
nixpkgs/pkgs/top-level/all-packages.nix:      toPath = builtins.toPath;
nixpkgs/pkgs/top-level/all-packages.nix:        builtins ? pathExists && builtins.pathExists (toPath name);
nixpkgs/pkgs/top-level/all-packages.nix:        else if configFile != "" && pathExists configFile then import (toPath configFile)
nixpkgs/pkgs/top-level/all-packages.nix:        else if homeDir != "" && pathExists configFile2 then import (toPath configFile2)
nixpkgs/pkgs/applications/editors/vim/ft-nix-support.patch:+syn keyword     nixBuiltin         __currentSystem __currentTime __isFunction __getEnv __trace __toPath __pathExists

So simply changing toPath to a more sensible definition might be a reasonable course.

[Profpatsch]

(triage) @adnelson was your problem solved?

[adnelson]

To the extent that the workaround that @edolstra suggested worked, yes. As far as the other issue (what to do about builtins.toPath) I don't know what if anything has happened on that.

[Profpatsch]

The nix path logic still behaves in a lot of strange ways, but I suggest to suggest changes about that the nix repository is a better place.