Nixpkgs: Overriding (Haskell) packages repeatedly is surprising and non-modular

Created on 18 May 2017 · 6Comments · Source: NixOS/nixpkgs

Issue description

The .override attribute doesn't allow you to repeatedly override the set of Haskell packages. This means it's impossible to compose multiple overlays that change the Haskell package set.

Steps to reproduce

The following Nix expression fails to evaluate:

let hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-1 = hs-0.override { overrides = self: super: { x = 42; }; };
    hs-2 = hs-1.override { overrides = self: super: { y = super.x; }; };
in hs-2.y

error: attribute ‘x’ missing, at (string):3:59

But this is surpring, because:

let hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-1 = hs-0.override { overrides = self: super: { x = 42; }; };
in hs-1.x

evaluates to 42 as expected. The problem is overrides only looks at the initial result. Further overrides does not update what overrides begins with.

This issue is probably not limited to Haskell packaging.

Importance

The new overlays system means that one would expect it to be possible to combine multiple Haskell overlays. I wanted to have an overlay that contained some build fixes & unreleased (but public) Haskell libraries, and another overlay that contained company internal projects. This isn't possible, because as shown above - these overlays don't compose.

Source

ocharles

👍3

Most helpful comment

Is extend mentioned anywhere in the nixpkgs documentation? It seems its existence really needs to be better documented and advertised.

bgamari on 3 Jul 2018

👍6

All 6 comments

You can do that with extend.

let hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-1 = hs-0.extend (self: super: { x = 42; });
    hs-2 = hs-1.extend (self: super: { y = super.x; });
in hs-2.y

It is not surprising if it is clear what override does and where overrides comes from. override overrides function arguments. But it doesn't give you the previous set of arguments as as an argument like e.g. overrideAttrs does (not that you could use that instead). So you can only keep an argument in place or replace it completely. If override would provide you with the previous set of arguments, this would work:

hs-0.override (oldArgs: { overrides =
  composeExtensions (self: super: { y = super.x })
    (oldArgs.overrides or (self: super: {});
})

Theoretically override could work behind the scenes to consume and accumulate the overrides arguments. But despite the similar names those are two independent mechanisms. The overrides argument is haskell specific, it is declared in haskell-modules/default.nix. The override attribute is the result of wrapping a function call into callPackage which in turn wraps it into makeOverrideable. That happens in all-packages.nix.
To complete the saga, the extend attribute I suggested exists because all haskell package sets are wrapped with makeExtensible also in haskell-modules/default.nix.

EDIT:
override works with functions as well as attribute sets, so what is said above, that it couldn't give the old arguments, is false. The example in question could be written like this:

let lib = (import <nixpkgs> {}).lib;
    hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-2 =
      (hs-0.override (oldArgs: { overrides =
        lib.composeExtensions
          (lib.composeExtensions
            (oldArgs.overrides or (self: super: {}))
            (self: super: { x = 42; }))
          (self: super: { y = super.x; });}));
in hs-2.y

This even seems to be the recommended way to do it, please see the discussion linked further down the thread.

ljli on 25 May 2017

👍5

Thanks, I wasn't even aware of extend!

ocharles on 30 May 2017

Is every example in the documentation rewritable with extend? Since the introduction of overlays, this has only become more confusing, since an overlay in the overrides style will entirely nuke nixpkgs packageOverrides, as far as I can tell. If this is not the case with extend, it seems a better choice. Thoughts?