Nixpkgs: ghcWithPackages wrapper does not work with ghc-8.10.1

[peti]

Using the current master branch @ 57ed5db343f7883c90c8a5a6b771216690842723, the following simple development environment cannot be created:

~
$ nix-shell --run zsh -p "haskell.packages.ghc8101.ghcWithPackages (pset: with pset; [zlib])"
these derivations will be built:
/nix/store/rq2yzy1kc1vlvld8lsv1dwycq6l30b8k-ghc-8.10.0.20200123-with-packages.drv
these paths will be fetched (0.38 MiB download, 5.85 MiB unpacked):
/nix/store/fig25wwp1gwv4q4ygxxa75xscc5z6cqv-zlib-0.6.2.1-doc
/nix/store/knhfvkm690z4qv8wpnq8dza9dvrfidnx-zlib-0.6.2.1
copying path '/nix/store/fig25wwp1gwv4q4ygxxa75xscc5z6cqv-zlib-0.6.2.1-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/knhfvkm690z4qv8wpnq8dza9dvrfidnx-zlib-0.6.2.1' from 'https://cache.nixos.org'...
building '/nix/store/rq2yzy1kc1vlvld8lsv1dwycq6l30b8k-ghc-8.10.0.20200123-with-packages.drv'...
/nix/store/xkm18nmijvgs3822q10lkdbf9sssb11a-ghc-8.10.0.20200123/nix-support:
propagated-build-inputs: /nix/store/knhfvkm690z4qv8wpnq8dza9dvrfidnx-zlib-0.6.2.1/nix-support/propagated-build-inputs
ghc-pkg: Couldn't open database /nix/store/grg4b1j9zbi4f0a5ijhp0zlzgii723sy-ghc-8.10.0.20200123-with-packages/lib/ghc-8.10.0.20200123/package.conf.d for modification: {handle: /nix/store/grg4b1j9zbi4f0a5ijhp0zlzgii723sy-ghc-8.10.0.20200123-with-packages/lib/ghc-8.10.0.20200123/package.conf.d/package.cache.lock}: hLock: invalid argument (Bad file descriptor)
builder for '/nix/store/rq2yzy1kc1vlvld8lsv1dwycq6l30b8k-ghc-8.10.0.20200123-with-packages.drv' failed with exit code 1
error: build of '/nix/store/rq2yzy1kc1vlvld8lsv1dwycq6l30b8k-ghc-8.10.0.20200123-with-packages.drv' failed
~~

The relevant bit is:

ghc-pkg: Couldn't open database /nix/store/grg4b1j9zbi4f0a5ijhp0zlzgii723sy-ghc-8.10.0.20200123-with-packages/lib/ghc-8.10.0.20200123/package.conf.d for modification: {handle: /nix/store/grg4b1j9zbi4f0a5ijhp0zlzgii723sy-ghc-8.10.0.20200123-with-packages/lib/ghc-8.10.0.20200123/package.conf.d/package.cache.lock}: hLock: invalid argument (Bad file descriptor)

[matthewbauer]

That looks like ghc-pkg recache is failing.

Is it possible the version number being 8.10.0.20200123 is causing problems? It may somehow think it is 8.10.1.

[peti]

https://gitlab.haskell.org/ghc/ghc/issues/13945 may or may not be relevant. The bug is supposed to be fixed since ghc-8.2.x, but the error messages look exactly like the one we are getting.

[vaibhavsagar]

Still doesn't work with the most recent haskell-updates branch:

$ nix-shell -I nixpkgs=. -p 'haskell.packages.ghc8101.ghcWithPackages (p: [ p.vector ])'
these derivations will be built:
  /nix/store/g42dzyfixjsdic6v941pwr640my07712-ghc-8.10.1-with-packages.drv
building '/nix/store/g42dzyfixjsdic6v941pwr640my07712-ghc-8.10.1-with-packages.drv'...
/nix/store/y5mlq6k40lmglk83g2d4wwrqbnczlfcc-vector-0.12.1.2/nix-support:
propagated-build-inputs: /nix/store/mjgcc312nynrbf4157g6yyll362swykh-primitive-0.7.0.1/nix-support/propagated-build-inputs
/nix/store/si82shf1fvp87zkv0dx6bsr0hw4h0v0q-ghc-8.10.1/nix-support:
propagated-build-inputs: /nix/store/mjgcc312nynrbf4157g6yyll362swykh-primitive-0.7.0.1/nix-support/propagated-build-inputs
ghc-pkg: Couldn't open database /nix/store/8k4pha8h3ah1dar9985g9p1bs64k04a3-ghc-8.10.1-with-packages/lib/ghc-8.10.1/package.conf.d for modification: {handle: /nix/store/8k4pha8h3ah1dar9985g9p1bs64k04a3-ghc-8.10.1-with-packages/lib/ghc-8.10.1/package.conf.d/package.cache.lock}: hLock: invalid argument (Bad file descriptor)
builder for '/nix/store/g42dzyfixjsdic6v941pwr640my07712-ghc-8.10.1-with-packages.drv' failed with exit code 1
error: build of '/nix/store/g42dzyfixjsdic6v941pwr640my07712-ghc-8.10.1-with-packages.drv' failed

I asked on the #ghc channel and I was pointed to this possibly related issue: https://github.com/haskell/cabal/issues/6222.

[vaibhavsagar]

@mpickering suggests that this error is related to https://gitlab.haskell.org/ghc/ghc/-/merge_requests/1996, and reverting that patch might make this error go away.

[phadej]

The strace ghc-pkg-8.10.1 recache does contain lines like (on Ubuntu, but it's irrelevant).

openat(AT_FDCWD, "/opt/ghc/8.10.1/lib/ghc-8.10.1/package.conf.d/package.cache.lock", O_RDWR|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/opt/ghc/8.10.1/lib/ghc-8.10.1/package.conf.d/package.cache.lock", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
ioctl(3, TCGETS, 0x7fff08716e70)        = -1 ENOTTY (Inappropriate ioctl for device)
fcntl(3, F_OFD_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = -1 EBADF (Bad file descriptor)

So there is some code in ghc-pkg which tries to open lock file in read-write mode, then looks like to fallback to read-only one, but then acquiring lock fails.

This makes sense to me. You shouldn't be able to ghc-pkg recache something which is readonly!

I don't know what's the best way to tackle that. Maybe adding a flag to ghc-pkg to make it not take locks would make sense for nixpkgs usage, as these directories are "unique". (I guess you have to recache in the destination to make paths right).

Alternatively you can hack, and make that particular lock file (which is in known place) writable.

[phadej]

And quick tests suggest that whether you have OFD locking or flock ones, if the lock wile is opened with O_RDONLY, both fail with Exception: fdLock: invalid argument (Bad file descriptor).

---

There is a regression in GHC:

% ghc-pkg-8.8.3 recache
ghc-pkg: /opt/ghc/8.8.3/lib/ghc-8.8.3/package.conf.d/package.cache: you don't have permission to modify this file

% ghc-pkg-8.10.1 recache
ghc-pkg: Couldn't open database /opt/ghc/8.10.1/lib/ghc-8.10.1/package.conf.d for modification: {handle: /opt/ghc/8.10.1/lib/ghc-8.10.1/package.conf.d/package.cache.lock}: hLock: invalid argument (Bad file descriptor)

I suggest opening an issue at least for that.

[vaibhavsagar]

Thanks for looking into this! I tried the most obvious thing that came to mind and added chmod +w $(readlink "${packageCfgDir}/package.cache.lock") before the line that calls ghc-pkg recache but it didn't work because i can't change the permissions of files in the Nix store. Maybe there's another way to work around the read-only lock file?

[phadej]

recache is write operation, therefore I think there's something wrong in the setup if you need to recache stuff which is (already) in /nix/store.

Unfortunately I cannot afford digging further in what the derivation tries to do here, and why. I just don't think this is ghc issue.

[vaibhavsagar]

I'm a bit confused: you're saying that there is a regression in GHC, but this isn't a GHC issue?

[phadej]

@vaibhavsagar the regression is that 8.8.3 says

 you don't have permission to modify this file

but 8.10.1 fails with weird error

hLock: invalid argument (Bad file descriptor)

---

EDIT: "neither work", but 8.8.3 at least tells you why.

[vaibhavsagar]

Thanks for clarifying!

[guibou]

I've opened PR #85446 which does the simplest thing: rm the old package symlink database and let ghc-pkg recache rebuild it.

[guibou]

Thanks for looking into this! I tried the most obvious thing that came to mind and added chmod +w $(readlink "${packageCfgDir}/package.cache.lock") before the line that calls ghc-pkg recache but it didn't work because i can't change the permissions of files in the Nix store. Maybe there's another way to work around the read-only lock file?

I started from this and realized that it was not writable because it was a symlink to another derivation. However rming it was fine.

[ocharles]

Could the fix be added to the 20.03 release?

[chessai]

Could the fix be added to the 20.03 release?

Seconded, that would be great.

[ocharles]

I'm actively working on cherry picking this into 20.03. I'm just building everything on our work Hydra - if it fixes the build I'll open a PR.

[chessai]

Thanks @ocharles!