Nix: Deprecate builtins.filterSource in favor of something that takes a a few regexes (perhaps different regexes for files/symlinks/directories)

[copumpkin]

It'll likely be faster (no evaluation of nix expressions), easier to use, and we can serialize its invocations for #709.

Perhaps we could keep the current one, but I know that my uses of it would be covered by the regex version, and be easier to use.

[cstrahan]

As an alternative to regexes, glob patterns would also be nice. Would be cool if both could be supported.

[dezgeg]

Or even .gitignore-compatible support.

[adnelson]

Is this still a thing?

I was going to make an issue that the filterSource function is hard to write against because the path argument in the filter function is an absolute path, not relative to the path being filtered, so it's almost impossible to filter on anything except the baseName.

That said, I would LOVE .gitignore-style support. .nixignore?

[adnelson]

Also, from examining trace calls, it seems that filterSource only examines files at the top level of the directory, which obviously limits its use.

[teh]

I'd also be interested in this - some low level function that's easier to use than filterSource (which, depending on evaluation sometimes sees /nix/store/... and sometimes /home/...).

Willing to put in the work if there is consensus around the details. I.e. what name to pick, exact API.

Longer term I'd like to see standard filterPythonSource, filterHaskellSource, .. library functions in nixpkgs.

[michalrus]

Please… :broken_heart: :sob:

[teh]

@michalrus can you try lib.sourceByRegex (https://github.com/NixOS/nixpkgs/blob/ec75a30b66748c1a411151b3c5ee477364ee4d14/lib/sources.nix#L31) ?

[michalrus]

Oh, that’s cool! I wonder if it works with negative lookahead.

OK, so how hard would it be to implement honoring .gitignore? Could we just:

1. Read its lines.
2. Change glob to regex.
3. Use regex, as lib.sourceByRegex does?

I remember once trying to run ${git}/bin/git clean -nxd in a “source derivation” to get which files would need to be ignored, but there were some cryptic errors.

[michalrus]

Hey,

so I finally looked again at this issue and a basic version doesn’t look very hard (a fragment of default.nix for a Haskell project):

let

  # TODO: How to fully re-use recursive .gitignore’s? https://git.io/vSo80
  gitignoreToRegexes = gitignorePath:
    builtins.map (line: if lib.hasPrefix "/" line then line else ".*/" + line + "($|/.*)")
      (builtins.map (builtins.replaceStrings ["." "**" "*"] ["\\." ".*" "[^/]*"] )
         (builtins.map (lib.removeSuffix "/")
            (lib.filter (line: line != "" && !(lib.hasPrefix "#" line))
               (lib.splitString "\n" (builtins.readFile gitignorePath)))));

  sourceByNegativeRegex = regexes: src:
    builtins.filterSource (path: type:
       let relPath = lib.removePrefix (toString src) (toString path);
       in lib.all (re: builtins.match re relPath == null) regexes) src;

  build = let
    src = sourceByNegativeRegex (gitignoreToRegexes ./.gitignore ++ ["/config" "/default.nix"]) ./.;
  in haskell.lib.overrideCabal (haskellPackagesWithOverrides.callCabal2nix "backend" src {}) (drv: {
    …
  });

in …

It seems to be working correctly, at least for this .gitignore:

/result
/dist
/dist-newstyle

# We’re absolutely not using Stack, but Intero uses these:
.stack-work/
flycheck_*.hs

Now we’d have to have some clever way of running gitignoreToRegexes for all .gitignore files in the directory tree. I don’t need that yet, so if anyone wants to try, please, do!

[arximboldi]

@michalrus that is a great step in the right direction but I think that doesn't handle a few .gitignore like exclusions, etc...

[shlevy]

IMO we should close this, as the core Nix bit of things can be done with builtins.match

[copumpkin]

Well the goal here wasn't so much to enable .gitignore, but to make a serializable version of the builtin 😄

[shlevy]

@copumpkin Ah. If you do end up doing that, I'd prefer it be on top of builtins.path

[antislava]

After struggling with nix-build errors, which turned out to be caused by .ghc.environemnt.* files autogenerated by cabal new-build within nix-shell, and trying to find a solution with builtins.filterSource without success (namely, .ghc.environment.* files still picked up by nix-build even if filtered out by filterSource), I ended up with the following hack to emulate .gitignore

let
  pkgs = import <nixpkgs> { };
  this-src = builtins.fetchGit { url = ../.; }; # url corresponds to git root, './.' in most cases 
  this-nix = builtins.toPath (this-src + "/project1/default.nix");
in  { project1 = pkgs.haskellPackages.callPackage this-nix { }; }

which behaved as expected/desired so far (i.e. no nix-build errors following cabal new-build in nix-shell + no rebuilds if any git-ignored or untracked files are created or modified + uncommitted changes in tracked files ARE picked up). An additional benefit is that the local build above is more consistent with the one where the source is fetched from a remote url.

[veprbl]

By the way, here is another implementation for gitignore support https://github.com/numtide/nix-gitignore (haven't tried it myself)

[siers]

Here's another: https://github.com/siers/nix-gitignore (for nix 2.0)

It has tests against the real git, though it doesn't require it or builtins.exec, is importable easily and has nice wrapper utils.

(By the way, I've just realized this might not've been the best place for such a comment, so sorry.)

[FRidh]

Unless builtins.path is used, the build is going to depend on the basename of the src. Using builtins.path however blocks one from using lib.sourceByRegex because filterSource returns absolute paths and sourceByRegex needs to remove this.

[chloekek]

Something that I find works very well is simply write a derivation that outputs the source files. This way you can do any simple or complicated logic that you want using the magic of Bash. For example:

buildRakuPackage {
    name = "infobox";
    src = stdenvNoCC.mkDerivation {
        name = "infobox-src";
        phases = [ "installPhase" ];
        installPhase = ''
            mkdir "$out"
            ln --symbolic ${./bin} "$out/bin"
            ln --symbolic ${./lib} "$out/lib"
            ln --symbolic ${./META6.json} "$out/META6.json"
        '';
    };
    # ...
}

Substitute cp, find, rsync, or anything else for ln as needed.

[roberth]

@chloekek Filtering sources in a derivation causes unnecessary rebuilds when "ignored" files change, because the hash of your filtered source depends on the hash of the unfiltered input.

Content-addressable outputs solve this, but will still require putting a potentially large unfilter file tree in the store for the derivation to work. It also requires that all the unfiltered files are accessible by the user, which is not the case with builtins.filterSource/builtins.path.

You can definitely use a derivation to post-process sources, but it's recommended to use one of the non-derivation-based methods first, to avoid unnecessary rebuilds.