Nixpkgs: "haskellPackages.extend" and "haskellPackages.override" are incompatible

[ElvishJerricco]

If you use extend to change stuff like the package set, it will remove override, preventing you from doing things like overriding the GHC used, or the all-cabal-hashes. Even if it didn't remove override, I'm guessing the override function installed by nixpkgs.callPackage would operate on the pre-extend object, thus eliminating any extensions added that way.

Fundamentally, I think nixpkgs.callPackage just needs to install an extend function itself, rather than having haskellPackages make its own. It needs to use the self: super: style so that you can extend the package with overrides arbitrarily many times. If it had this, the overrides argument to haskell-modules/default.nix wouldn't be necessary, since it would essentially be a strictly worse version of extend.

$ nix-repl "<nixpkgs>"
nix-repl> ((haskellPackages.extend (self: super: {})).override {}).ghc
error: attribute ‘override’ missing, at (string):1:2

[FRidh]

I'm not really familiar with haskellPackages, but why do you need to use override when you can use extend? I think the whole idea is to only override the set through the fixed-point combinator. Therefore, I think the override should actually be removed (which was likely added by callPackage).

[ElvishJerricco]

callPackage's override does a different thing than extend. override allows you to change the arguments that callPackage provides to the package, while extend allows you to coherently change fields of the resulting package. In haskellPackages, the most common argument overridden was the similarly named overrides field, which is now just worse than extend (not really; see below). But as an example: you can use override to change the all-cabal-hashes that callHackage will use, but you cannot do this with extend.

[ElvishJerricco]

@shlevy This needs to be reopened now

[jwiegley]

I just ran into this today too.

[tekul]

It looks like this was changed here but it's not clear why the changes for this issue were reverted. Is there a reason why extend can't be used for defining haskellPackages?

I'd been using extend after reading about it in this issue as a solution for composing overlays. I ran into this problem after updating nix-unstable at the end of last year, though I don't understand why it didn't break earlier, since I update quite regularly.

[ElvishJerricco]

@tekul Those changes were reverted because they were much more complicated than the issue really required, and it created a performance regression in the evaluator. I think we just need a tailored solution; i.e. use a custom thing instead of callPackage to define haskellPackages.

[tekul]

Thanks for the info, @ElvishJerricco :-). Could you recommend a workaround?

[ElvishJerricco]

@tekul Don't use haskellPackages.extend :P You can use override, it's just annoying

haskellPackages.override (old: {
  overrides = lib.composeExtensions (old.overrides or (_: _: {})) (self: super: {
    # ...
  });
})

[pjones]

I ran into this while overriding enableLibraryProfiling via mkDerivation. It sounds like all uses of haskellPackages.override need to use the function version of override with composeExtensions.

Are there any other options? I'm trying to write some generic Nix code and don't want to force my users to rewrite their code to use them more verbose version of override. But if I don't, their use of override shadows mine.

An overlay-like feature would be much nicer.

[ElvishJerricco]

@pjones I think we should just use a custom callPackage to define haskellPackages. Something adds extend in a simple and compatible way. I'll put up a PR later

[nh2]

Don't use haskellPackages.extend :P You can use override, it's just annoying

It's getting ever more confusing.

https://github.com/NixOS/nixpkgs/issues/25887#issuecomment-304077409 recommends to use extend, this thread recommends not to use extend, what are the real guidelines now?

[ElvishJerricco]

@nh2 The problem is that people perceive override as being unable to add overrides without nuking previous overrides. If you use only overrides, you can use the old attrset to maintain old overrides:

haskellPackages.override (old: {
  overrides = lib.composeExtensions (old.overrides or (_: _: {})) (self: super: {
    # ...
  });
})

If you use extend, then you immediately lose the ability to use override for inputs like ghc or all-cabal-hashes. extend is fundamentally worse, but ergonomically much better. Thus I propose that we fix extend. Until then, you unfortunately have to know what kind of package set you're receiving to know which you should use; if you're receiving one that has been extend'd, then you must use extend. Otherwise you're free to use either, though you should use override, as it will continue to allow future overrides on other inputs and still allow extend to come after.

TL;DR: overrides is technically better, but only if you know the quirk about preserving old overrides, and it's not always actually possible.

[mpickering]

This is still an issue. I just ran into it now.

[nh2]

haskellPackages.override (old: {

@ElvishJerricco Still confused by this. It gets me

error: value is a function while a set was expected

And indeed so far all uses of override I've seen look like haskellPackages.override { ... }.

Am I missing something?

[nh2]

Am I missing something?

My bad, I had a typo, writing

  self: super:
    self: super:

when it should be only 1 pair of self: super:.

[DanBurton]

Bump. Still confused about which to use, though the gist of it seems to currently be "use override, but you have to manually preserve old overrides".

I made an attempt at abstracting the "correct" override pattern, though I'm fairly sure it's wrong in some way.

https://gist.github.com/DanBurton/bf79410a9ed639bcd3d66dac58b645d3

[dhess]

@DanBurton Here is what I'm using, based on @ElvishJerricco's suggestion earlier in the thread:

https://github.com/dhess/dhess-lib-nix/blob/f3f9660c4d86eabee40fd587b15c3535936857a3/overlays/haskell/lib.nix#L11

The syntax of properExtend is exactly the same as extend; e.g.:

https://github.com/dhess/dhess-nix/blob/8d2b3a1e840cc906224284f463ed990d8c13f07d/overlays/haskell-packages.nix#L18

Using this function has fixed a few problems I was having with extend.

[ElvishJerricco]

I still use extend quite frequently because I'm lazy and it's convenient. But yea, any time the package set could conceivably require further modifications, I raise my diligence and always use override

[Infinisil]

I see no reason why we couldn't define .extend as the properExtend version

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Fresheyeball]

Ran into this today.

[chreekat]

So what would the patch look like?

This issue starts off suggesting that haskellPackages shouldn't make its own extend, but if I'm following correctly, it's the other way around: haskellPackages doesn't provide a custom extend, but it should.

I flailed feebly to attempt this, but never managed to succeed. As far as I can tell, whatever I was doing just got overridden, and I ended up back at error: attribute 'override' missing,

(Pun not intended!)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/patching-ghc-for-your-project/9614/1