Nixpkgs: Using python.withPackages with nixpkgs-overlays does not work

[knedlsepp]

Issue description

I'm currently trying to make python.withPackages work with some custom overlays, but can't find to make it work.

Steps to reproduce

~/.config/nixpkgs/overlays/test.nix

self: super: with self;
{
  pythonPackages = super.pythonPackages.override {
    overrides = self: super: {
      my_stuff = pythonPackages.buildPythonPackage rec {
        pname = "pyaes";
        version = "1.6.0";
        name = "${pname}-${version}";
        src = pythonPackages.fetchPypi {
          inherit pname version;
          sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
        };
      };
    };
  };
  my_stuff = pythonPackages.buildPythonPackage rec {
    pname = "pyaes";
    version = "1.6.0";
    name = "${pname}-${version}";
    src = pythonPackages.fetchPypi {
      inherit pname version;
      sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
    };
  };
}

nix-shell -p 'python.withPackages(ps: [ps.my_stuff])'

gives

error: attribute ‘my_stuff’ missing, at (string):1:91
(use ‘--show-trace’ to show detailed location information)

whereas nix-shell -p my_stuff works.

[FRidh]

self: super:
# Within the overlay we use a recursive set, though I think we can use `self` as well.
rec {
  # nix-shell -p python.pkgs.my_stuff
  python = super.python.override {
    # Careful, we're using a different self and super here!
    packageOverrides = self: super: {
      my_stuff = super.buildPythonPackage rec {
        pname = "pyaes";
        version = "1.6.0";
        name = "${pname}-${version}";
        src = super.fetchPypi {
          inherit pname version;
          sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
        };
      };
    };
  };
  # nix-shell -p pythonPackages.my_stuff
  pythonPackages = python.pkgs;

  # nix-shell -p my_stuff
  my_stuff = pythonPackages.buildPythonPackage rec {
    pname = "pyaes";
    version = "1.6.0";
    name = "${pname}-${version}";
    src = pythonPackages.fetchPypi {
      inherit pname version;
      sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
    };
  };
}

[knedlsepp]

So it works if you do it correctly. Thanks for the example!

[knedlsepp]

@FRidh: After using this for a while, I'm finally convinced that this is somehow not picked up by all tools.
Using nix-env -qaP or nix search, none of the python Packages show up. However all other derivations show up in the search just fine.
Would you know how this can be fixed?

[knedlsepp]

For anybody wondering why their python-overlays don't show up in the search nix-env -qaP or nix search
You should use:

pythonPackages = super.recurseIntoAttrs (python.pkgs);

[danbst]

@knedlsepp add a like to this PR https://github.com/NixOS/nixpkgs/pull/54266 ! With it your problem would look like

self: super:
with self.python27.pkgs; {

  _merge_python27 = true;
  python27._merge_pkgs = true;

  # nix-shell -p python.pkgs.my_stuff
  # nix-shell -p 'python.withPackages(ps: [ps.my_stuff])'
  python27.pkgs.my_stuff = buildPythonPackage rec {
    pname = "pyaes";
    version = "1.6.0";
    name = "${pname}-${version}";
    src = fetchPypi {
      inherit pname version;
      sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
    };
  };
}

Even more, nix-env works out of box:

$ nix-env -qaP '*' | grep my_stuff
python27Packages.my_stuff                                        python2.7-pyaes-1.6.0