Nixpkgs: Python's packageOverrides isn't composible

[twhitehead]

Issue description

@FRidh the packageOverrides function argument approach to overriding package used in the base python function (here is the 2.7 one) doesn't compose in overlays. Specifically, if you follow the directions in the nixpkgs manual only the last declared override winds up being applied.

I haven't actually thought of a solution to this yet, other than determining that it is pretty much impossible to work around as you can only get a hold of a closed version of the previous package set via python.pkgs. Perhaps looking into the haskell infrastructure might give some directions (haven't looked at it in a while, but the fixpoint stuff looks like it came from there)?

Steps to reproduce

Put the following into ~/.config/nixpkgs/overlays/1.nix

self: super:
rec {
  python27 = super.python27.override {
    packageOverrides = python-self: python-super: {
      one = "definition from 1.nix";
    };
  };
}

and this into ~/.config/nixpkgs/overlays/2.nix

self: super:
rec {
  python27 = super.python27.override {
    packageOverrides = python-self: python-super: {
      two = "definition from 2.nix";
    };
  };
}

Ideally both of these would get applied. This is not the case though as you can see from the following

nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.one'

error: attribute 'one' missing, at (string):1:28



md5-9a9ddd4cd3d92af44a09d5730f69341d



nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.two'



md5-9a9ddd4cd3d92af44a09d5730f69341d



"definition from 2.nix"



md5-ca350cb04a285c1dcd68d783465e7fce



mv ~/.config/nixpkgs/overlays/2.nix{,-inactive}
nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.one'



md5-9a9ddd4cd3d92af44a09d5730f69341d



"definition from 1.nix"



md5-9a9ddd4cd3d92af44a09d5730f69341d



nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.two'



md5-9a9ddd4cd3d92af44a09d5730f69341d



error: attribute 'two' missing, at (string):1:28

[FRidh]

Yep, this is a known issue, one which bothers me as well. I think we should pull the creation of the package set out of the interpreters.

As a workaround, keep your overrides as an attribute, something like pythonOverrides. The next overlay can then use composeExtensions.

[FRidh]

Related https://github.com/NixOS/nixpkgs/pull/44196

[twhitehead]

Thanks for the suggestions and links.

If I'm understanding where you are going with your comment, you are saying if the interpreter package pulled in the package set instead of producing it, you could then override the package set in the standard way with overlays.

The interpreter would automatically pick up changes because it would be closed over the final package set. You could suck the final interpreter in the package set if you just re-exported it for compatibility without creating any loops.

Cheers! -Tyson

[FRidh]

Exactly that!

[FRidh]

if the interpreter package pulled in the package set instead of producing it

But, having the overrides is enough:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
  python36 = python36.pkgs.override { packageOverrides = pythonOverrides; };
}

overlay_b.nix:

self: super: {
  pythonOverrides = lib.composeExtensions (selfPython: superPython: {
    pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  }) super.pythonOverrides;
}

(not tested)

[twhitehead]

Thanks. I implemented that and it works well. :+1:

Unless you would like to leave it open, feel free to close this issue.

[Shados]

For a more standard-nixos version of said workaround, here's more or less what I'm using:

python-override-setup.nix

```python-override-setup.nix
{ config, lib, pkgs, ... }:
{
# Setup for https://github.com/NixOS/nixpkgs/issues/44426 python
# overrides not being composable...
nixpkgs.overlays = lib.mkBefore [
(self: super: let
pyNames = [
"python27" "python34" "python35" "python36" "python37"
"pypy"
];
overriddenPython = name: [
{ inherit name; value = super.${name}.override { packageOverrides = self.pythonOverrides; }; }
{ name = "${name}Packages"; value = super.recurseIntoAttrs self.${name}.pkgs; }
];
overriddenPythons = builtins.concatLists (map overriddenPython pyNames);
in {
pythonOverrides = pyself: pysuper: {};
# The below is just a wrapper for clarity of intent, use like:
# pythonOverrides = buildPythonOverrides (pyself: pysuper: { ... # overrides }) super.pythonOverrides;
buildPythonOverrides = newOverrides: currentOverrides: super.lib.composeExtensions newOverrides currentOverrides;
} // listToAttrs overriddenPythons)
];
}

#### `some-module.nix`
```some-module.nix
{ config, lib, pkgs, ...}:
{
  nixpkgs.overlays = [
    (self: super: {
      pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
        some-package = "definition from some-package.nix";
      }) super.pythonOverrides;
    })
  ];
}

another-module.nix

another-module.nix { config, lib, pkgs, ...}: { nixpkgs.overlays = [ (self: super: { pythonOverrides = super.buildPythonOverrides (pyself: pysuper: { another-package = "definition from another-package.nix"; }) super.pythonOverrides; }) ]; }

Some notes

• This does work, for at least some values of 'work'...
• The use of mkBefore is to ensure that this is independent of the default order of expression evaluation.
• I've a sneaking suspicion I've done something terrible somewhere in there, so please do tell if so.

[danbst]

This should be solved with https://github.com/NixOS/nixpkgs/pull/54266

$ code='with import ./. { overlays = [
    (self: super: {
        pythonPackages.one = "definition from 1.nix";
    })
    (self: super: {
        pythonPackages.two = "definition from 2.nix";
    })
]; }'

$ nix-instantiate --eval -E "$code; pythonPackages.one"
"definition from 1.nix"

$ nix-instantiate --eval -E "$code; pythonPackages.two"
"definition from 2.nix"

[deliciouslytyped]

I'm not quite sure about this, but .override can be used with a function which takes the previous argument set: .override (a: { stuff = doSomething a.oldStuff}) so I think in theory one could write a function that overrides the previous arguments with a recursive merge? It's been a while but that sounds close to what I did here: https://github.com/deliciouslytyped/nix-ghidra-wip/blob/43b2207729db83f84c0812ff9a054907d5ddc222/packages.nix#L6

[Infinisil]

Actually this PR of mine solves this issue: https://github.com/NixOS/nixpkgs/pull/67422

It allows overrides of the form

python3.pkgs.overrideScope' (self: super: {
  # ...
})

Without discarding previous changes.

And this with very little code changed.

[DavHau]

if the interpreter package pulled in the package set instead of producing it

But, having the overrides is enough:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
  python36 = python36.pkgs.override { packageOverrides = pythonOverrides; };
}

overlay_b.nix:

self: super: {
  pythonOverrides = lib.composeExtensions (selfPython: superPython: {
    pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  }) super.pythonOverrides;
}

(not tested)

Thanks, with this i got it working. Though, it needed a few modifications to work. Here the corrected version in case anyone needs it:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py = superPython.py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
}

overlay_b.nix:

self: super: rec {
  pythonOverrides = self.lib.composeExtensions super.pythonOverrides (selfPython: superPython: {
    pytest = superPython.pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  });
  python36 = super.python36.override { packageOverrides = pythonOverrides; };
}

[deliciouslytyped]

I forgot about this thread and I haven't read carefully enough to tell if this helps here, but maybe it's relevant; https://discourse.nixos.org/t/makeextensibleasoverlay/7116

[DavHau]

I'm now using the following function for mach-nix to merge an arbitrary list of pythonOverrides:

  mergeOverrides = with pkgs.lib; overrides:
    if length overrides == 0
    then a: b: {}  # return dummy overrides
    else
      if length overrides == 1
      then elemAt overrides 0
      else
        let
          last = head ( reverseList overrides );
          rest = reverseList (tail ( reverseList overrides ));
        in
          composeExtensions (mergeOverrides rest) last;

[andersk]

@DavHau mergeOverrides = lib.foldr lib.composeExtensions (self: super: { })

[deliciouslytyped]

Maybe we should put that in lib (or not) :p

https://github.com/deliciouslytyped/nix-rootedoverlay/blob/84f14bae677d4cb9a311bdedc066e0c91d4f0703/rooted.nix#L35

[FRidh]

@deliciouslytyped that was proposed in https://github.com/NixOS/nixpkgs/issues/33258