Nixpkgs: How do you work around broken haskell packages?

[codygman]

Apologies if this doesn't belong here, I can post it somewhere else if needed.

My problem

In #83380 I ran into an issue trying to use beam-core with nixos. I know from experience that it's probably not building because nixpkgs has a newer hashable and/or vector-sized than the constraints of beam-core allow which looked to be resolved in the next beam release.

So the fix here that would have let me continue using nix instead of using stack would be to use the version of beam-core from github.

I understand plain nix can be used for this, but it's hard to recall. It's hard to recall because I have to keep reverting back to stack. This issue is a large part of the reason I can't just use nix by default.

My Attempt

Where I'm at

I started by just trying to use nix-shell -p '(haskellPackages.ghcWithPackages (p: [p.beam-core p.beam-sqlite p.text]))' mostly because I couldn't remember how to make a shell.nix. Stack for instance lets you just use either stack init or stack new for this case.

Saying that though, I remember I can generate a default.nix and shell.nix with cabal2nix.

Generating default.nix and shell.nix

$ cabal2nix . > default.nix  # remembering this is easy enough
$ cabal2nix --shell . > default.nix # I only remember I can use --shell here because of the amount of times i've tried to do this, I think many others would have given up

Maybe generating both default.nix and shell.nix when calling cabal2nix init would be a good idea?

The observant might notice I overwrote default.nix with the shell version. I make this mistake often, so one command to generate these common files to get started would be nice.

Quick pause

Now I've got the equivalent of my original nix-shell -p '(haskellPackages.ghcWithPackages (p: [p.beam-core p.beam-sqlite p.text]))' command (almost, I didn't have text in my cabal file).

I can just run nix-shell now, it will use shell.nix by default, and I'll get the same error.

Using latest beam-core from git

So by luck, I noticed when I accidentally executed cabal2nix with no parameters that it can do things like:

  Recognized URI schemes:

    cabal://pkgname-pkgversion     download the specified package from Hackage
    cabal://pkgname                download latest version of this package from Hackage
    file:///local/path             load the Cabal file from the local disk
    /local/path                    abbreviated version of file URI
    <git/svn/bzr/hg URL>           download the source from the specified repository

That means I can just call cabal2nix on the git clone url right?

[cody@nixos:~/code/beam-minimal-newtype-problem-repro]$ cabal2nix [email protected]:tathougies/beam.git > beam.nix
error: unable to download '[email protected]:tathougies/beam.git': URL using bad/illegal format or missing URL (3)
Initialized empty Git repository in /tmp/git-checkout-tmp-ycaxCEHw/beam/.git/
remote: Enumerating objects: 274, done.
remote: Counting objects: 100% (274/274), done.
remote: Compressing objects: 100% (238/238), done.
remote: Total 274 (delta 20), reused 89 (delta 10), pack-reused 0
Receiving objects: 100% (274/274), 1.18 MiB | 1.21 MiB/s, done.
Resolving deltas: 100% (20/20), done.
From github.com:tathougies/beam
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...

git revision is ff6d16daa189355db4cf24e90d8173768c1418bb
path is /nix/store/grxnkp115a3i4cln9j08gpwf6ks677fl-beam
git human-readable version is -- none --
Commit date is 2020-03-05 21:34:59 -0500
hash is 11f1nrw3s7ihf3lyskjx1mdfi4s5d3rfn0fwwmcc8xl2dgjdlnk8
abort: repository [email protected]:tathougies/beam.git not found!
svn: E155007: '/home/cody/code/beam-minimal-newtype-problem-repro/[email protected]:tathougies/beam.git' is not a working copy
failed to open trace file: [Errno 2] No such file or directory: '/homeless-shelter/.bzr.log'
bzr: ERROR: Unsupported protocol for url "[email protected]:tathougies/beam.git"
cabal2nix: user error (Failed to fetch source. Does this source exist? Source {sourceUrl = "[email protected]:tathougies/beam.git", sourceRevision = "", sourceHash = UnknownHash, sourceCabalDir = ""})

Hmm, okay let's try the https url then (note though that by default most developers will probably have git:// urls selected by default):

[cody@nixos:~/code/beam-minimal-newtype-problem-repro]$ cabal2nix https://github.com/tathougies/beam.git > beam.nix
[0.1 MiB DL]
*** parsing cabal file: https://github.com/tathougies/beam.git: openBinaryFile: does not exist (No such file or directory)
Initialized empty Git repository in /tmp/git-checkout-tmp-OLvN5dsF/beam/.git/
remote: Enumerating objects: 274, done.
remote: Counting objects: 100% (274/274), done.
remote: Compressing objects: 100% (238/238), done.
remote: Total 274 (delta 20), reused 89 (delta 10), pack-reused 0
Receiving objects: 100% (274/274), 1.18 MiB | 1.51 MiB/s, done.
Resolving deltas: 100% (20/20), done.
From https://github.com/tathougies/beam
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...

git revision is ff6d16daa189355db4cf24e90d8173768c1418bb
path is /nix/store/grxnkp115a3i4cln9j08gpwf6ks677fl-beam
git human-readable version is -- none --
Commit date is 2020-03-05 21:34:59 -0500
hash is 11f1nrw3s7ihf3lyskjx1mdfi4s5d3rfn0fwwmcc8xl2dgjdlnk8
abort: 'https://github.com/tathougies/beam.git' does not appear to be an hg repository:
---%<--- (text/html; charset=utf-8)






<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://github.githubassets.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">



  <link crossorigin="anonymous" media="all" integrity="sha512-5Bs4ERl99/u2AUfpOZF2F0cdfNby7+Vd9teUshXUBPo5CjwECR7IAEf+weI/eCk5tF7K1h3O8hd8k0+P/HePeg==" rel="stylesheet" href="https://github.githubassets.com/assets/frameworks-e41b3811197df7fbb60147e939917617.css" />
  <link crossorigin="anonymous" media="all" integrity="sha512-iXrV/b4ypc1nr10b3Giikqff+qAx5osQ0yJRxHRDd8mKFefdMtEZ0Sxs1QysJxuJBayOKThjsuMjynwBJQq0aw==" rel="styleshee
---%<---
!

No luck there either it seems. Luckily since i've ran into this before, I know that beam is a repo with multiple cabal projects and cabal2nix might not handle those.

Let's go checkout the cabal2nix issues.

It looks like someone shares my issues in NixOS/cabal2nix#440

Support cabal.project files #286 is likely the one biting me now I think. I just verified beam does in fact have a cabal.project file.

"Until this ticket gets an implementation the solution is to use stack2nix?" - said one commenter

I suppose I could try using stack2nix, nix makes it easy to try those things out with nix-shell.

Trying stack.nix workaround since cabal2nix doesn't work with cabal.project

So now I'm at a new tool and learning how it works. I'll note that stack2nix has taken 3-4 minutes by now, I guess the reason may be that it uses a different GHC version than 8.8.3 I use?

Oh no :(

Setup: Encountered missing or private dependencies:
Cabal >=2.0.0.2 && <2.5,
base >=4.9 && <4.13,
distribution-nixpkgs >=1.1 && <1.3,
optparse-applicative >=0.13.2 && <0.15,
regex-pcre >=0.94.4 && <0.95

builder for '/nix/store/hrz327wypldjrkbj2c3j3nsdhi7rfyd7-stack2nix-0.2.3.drv' failed with exit code 1
error: build of '/nix/store/hrz327wypldjrkbj2c3j3nsdhi7rfyd7-stack2nix-0.2.3.drv' failed

I guess I can't do that one.

I know about this niv dependency thing... maybe it can help

I learned about niv some time back and it seemed very nice. I had a hang-up I don't remember and couldn't put more time in at the moment.

Let's install it:

```
$ nix-env -iA niv -f https://github.com/nmattia/niv/tarball/master \
--substituters https://niv.cachix.org \
--trusted-public-keys niv.cachix.org-1:X32PCg2e/zAm3/uD1ScqW2z/K0LtDyNV7RdaxIuLgQM=
warning: ignoring untrusted substituter 'https://niv.cachix.org'
warning: ignoring the user-specified setting 'trusted-public-keys', because it is a restricted setting and you are not a trusted user
unpacking 'https://github.com/nmattia/niv/tarball/master'...
installing 'niv-0.2.13'
building '/nix/store/46caknkivrcpi3x2yj3dwymhq7kb8hjj-user-environment.drv'...
error: packages '/nix/store/na2q08zpv9l2kxm5xs7ajvykhpws6656-home-manager-path/bin/niv' and '/nix/store/1adlf6i681hki2mmn4glmdi3is796qhy-niv-0.2.13/bin/niv' have the same priority 5; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)
builder for '/nix/store/46caknkivrcpi3x2yj3dwymhq7kb8hjj-user-environment.drv' failed with exit code 1
error: build of '/nix/store/46caknkivrcpi3x2yj3dwymhq7kb8hjj-user-environment.drv' failed
````

Oh no! Wait... That's just an error because I've already installed it with home-manager. I had many battles with this priority issue before understanding I should just uninstall packages everywhere and try again. A better error message would go a long way here I think.

Okay, since I have niv... (Actually I'm out of time for now, will continue this at a later time)

[cdepillabout]

@codygman

Sorry you're having problems with the Haskell infrastructure.

Would you be able to summarize your questions in one or two sentences? It is slightly hard to follow exactly what you want help with.

Also, if you haven't seen it, you may find some help in the following two resources:

1. The Haskell section of the nixpkgs manual
2. https://discourse.nixos.org/t/nix-haskell-development-2020/6170

If you want more real-time help, maybe you could jump on to one of the chat channels listed at the end here: https://discourse.nixos.org/t/call-to-action-for-updating-haskell-packages-after-bump-to-lts-15/6071

[sorki]

You can use nixpkgs provided tools like haskell.lib.doJailbreak to make package ignore its bounds - for example:

nix-shell -p '(haskellPackages.ghcWithPackages (p: [(haskell.lib.doJailbreak p.beam-core)] ))'

although it's probably better to start a small overlay as in your case you might need to fix more than just a top-level package. In my case doJailbreak won't help with building beam-core with ghc883 due to MonadFail (easily fixable):

Database/Beam/Backend/SQL/Row.hs:70:3: error:
    ‘fail’ is not a (visible) method of class ‘Monad’
   |
70 |   fail = FromBackendRowM . liftF . FailParseWith .

What I do in such case is I fork the package from upstream, fix the bug, create a PR and for time being I use my fork instead via overlay.

Make sure to read https://nixos.org/nixpkgs/manual/#haskell and https://github.com/Gabriel439/haskell-nix

Also don't waste time with stack, in Nix/OS context it's completely redundant concept.

[cdepillabout]

Also don't waste time with stack, in Nix/OS context it's completely redundant concept.

This is incorrect.

stack makes it easy to use past LTS releases (along with old compilers). It also makes it easy to override single packages. There is no super-easy way to do this with nix/nixpkgs.

It does this in a way that is not specific to Nix/NixOS, so you can easily collaborate with people who are not using Nix.

There is also nix-specific support built into stack, which can be convenient.

That said, it is often more "nixy" to get everything building without relying on stack. In my experience, it makes it much easier to get builds running on CI, etc.

[sorki]

Also don't waste time with stack, in Nix/OS context it's completely redundant concept.

This is incorrect.

Well, opinionated at least :) I'm talking about Nix/OS context and my experience.

stack makes it easy to use past LTS releases (along with old compilers). It also makes it easy to override single packages. There is no super-easy way to do this with nix/nixpkgs.

You can do the same with Nix and pinned nixpkgs.

It does this in a way that is not specific to Nix/NixOS, so you can easily collaborate with people who are not using Nix.

That's a valid point but you can easily install Nix in most environments and it can be as easily removed as everything is stored in /nix/store. It also depends on the complexity of the project - like if you manage to keep up with the ecosystem then there's no problem to build your package with either tool or even with plain cabal. Sure the situation is completely different when you have tons of dependencies which needs overrides / specific versions.

There is also nix-specific support built into stack, which can be convenient.

The problem I have with this is it actually makes it more complex than just using cabal2nix or callCabal2nix.

That said, it is often more "nixy" to get everything building without relying on stack. In my experience, it makes it much easier to get builds running on CI, etc.

Yes, also easier to maintain, deploy, reproduce.

@cdepillabout we shouldn't derail this issue into ideological discussion, better use #nixos-chat or Discourse for that.

[codygman]

Using everything I've learned about nix up to this point including looking at all resources everyone posted (thanks!) I ran into another wall detailed in discourse at Failing to override network dependency for bson.

I want to transition from using stack for most Haskell projects to nix and help others do the same if I gain the benefits I believe I will, but walls like this and lack of resources make me wonder if that's possible.

That's the reason for me posting this issue. Hopefully this concrete example will let us figure out where the gaps are and improve documentation for others trying to migrate to using Nix instead of Stack as well.

As a last note I want to say that I've looked at the manual and I even went into the nixpkgs source code to
try and figure things out. I suppose if I knew how to print out the derivations after a function like developPackage modifies them I could figure it all out even in the absence of better documentation for these specific cases.

[rpearce]

(hopefully this is helpful...)

I'm currently writing a blog post on using hakyll with nix (update: https://robertwpearce.com/hakyll-pt-6-pure-builds-with-nix.html), and an issue with unavailable or broken dependencies came up. I found this blog post, https://turbomack.github.io/posts/2020-02-17-cabal-flags-and-nix.html, and it helped me reach this point for a release.nix file:

{ compiler ? "ghc883"
, sources ? import ./nix/sources.nix
}:

let
  overlay = _: pkgz: {
    niv = import sources.niv { };
  };

  pkgs = import sources.nixpkgs {
    config = {
      packageOverrides = pkgz: rec {
        haskell = pkgz.haskell // {
          packages = pkgz.haskell.packages // {
            ${compiler} = pkgz.haskell.packages.${compiler}.override {
              overrides = hpNew: hpOld: rec {
                hakyll = hpOld.hakyll.overrideAttrs(oldAttrs: {
                  configureFlags = "-f watchServer -f previewServer";
                  patches = [ ./hakyll.patch ];
                });

                project = hpNew.callCabal2nix "hakyll-nix-example" ./. { };
              };
            };
          };
        };
      };
    };
    overlays = [ overlay ];
  };

  haskellPackages = pkgs.haskell.packages.${compiler};
in
  {
    project = haskellPackages.project;

    shell = haskellPackages.shellFor {
      packages = p: with p; [
        haskellPackages.project
      ];
      buildInputs = with haskellPackages; [
        ghcid
        hlint
        niv
        #ormolu
        pkgs.cacert
        pkgs.nix
      ];
      withHoogle = true;
    };
  }

(If you don't use niv already, I'd consider giving it a shot, for it makes it super easy to maintain pinned versions of any package (as opposed to manually doing nix-prefetch-url or nix-prefetch-git).)

The hakyll.patch file is the result of

1. forking the original hakyll project
2. making the changes that fix the build in nix
3. $ git diff > hakyll.patch
4. including that file in my project
   diff diff --git a/hakyll.cabal b/hakyll.cabal index fcded8d..9746f20 100644 --- a/hakyll.cabal +++ b/hakyll.cabal @@ -199,7 +199,7 @@ Library If flag(previewServer) Build-depends: wai >= 3.2 && < 3.3, <ul> <li>warp &gt;= 3.2 &amp;&amp; &lt; 3.3,</li> <li>warp,<br /> wai-app-static &gt;= 3.1 &amp;&amp; &lt; 3.2,<br /> http-types &gt;= 0.9 &amp;&amp; &lt; 0.13,<br /> fsnotify &gt;= 0.2 &amp;&amp; &lt; 0.4<br />

The key part is the packageOverrides bit:

packageOverrides = pkgz: rec {
  haskell = pkgz.haskell // {
    packages = pkgz.haskell.packages // {
      ${compiler} = pkgz.haskell.packages.${compiler}.override {
        overrides = hpNew: hpOld: rec {
          hakyll = hpOld.hakyll.overrideAttrs(oldAttrs: {
            configureFlags = "-f watchServer -f previewServer";
            patches = [ ./hakyll.patch ];
          });

          project = hpNew.callCabal2nix "hakyll-nix-example" ./. { };
        };
      };
    };
  };
};

Hope that helps!

[rpearce]

Update: here's something I use that is a little more dense, but the point is that you can have a hpkgs.nix file where you define your updated haskellPackages, then you use that whenever you need to use haskellPackages:

# hpkgs.nix

{ compiler ? "ghc884"
, pkgs
}:

let
  inherit (pkgs.lib.trivial) flip pipe;
  inherit (pkgs.haskell.lib) appendPatch appendConfigureFlags;

  hakyllFlags = [ "-f" "watchServer" "-f" "previewServer" ];

  haskellPackages = pkgs.haskell.packages.${compiler}.override {
    overrides = hpNew: hpOld: {
      hakyll =
        pipe
           hpOld.hakyll
           [ (flip appendPatch ./hakyll.patch)
             (flip appendConfigureFlags hakyllFlags)
           ];

      # ...
    };
  };
in haskellPackages