nix-store --repair-path isn't repairing, manually repairing works

Created on 18 Mar 2017 · 21Comments · Source: NixOS/nix

[root@nixos:~]# nix-store --verify-path /nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3
path ‘/nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3’ was modified! expected hash ‘db80ad6193a44f9b000c3d5aa5eafef54f9ed98e02b7649edb3239f9bdfa9195’, got ‘c5506d9735b079b1144bf3898eed3460e0fe3804863a7c9206661e898d18337c’

[root@nixos:~]# nix-store --repair-path /nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3
fetching path ‘/nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3’...

*** Downloading ‘http://cache.nixos.org/nar/00vk5ys9yyqqvifxrb5v883y0hmhxdh4awkxsj40d4y9y6nvczw4.nar.xz’ to ‘/nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3674k  100 3674k    0     0  1307k      0  0:00:02  0:00:02 --:--:-- 1325k


[root@nixos:~]# nix-store --verify-path /nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3
path ‘/nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3’ was modified! expected hash ‘db80ad6193a44f9b000c3d5aa5eafef54f9ed98e02b7649edb3239f9bdfa9195’, got ‘c5506d9735b079b1144bf3898eed3460e0fe3804863a7c9206661e898d18337c’

[root@nixos:~]# wget http://cache.nixos.org/nar/00vk5ys9yyqqvifxrb5v883y0hmhxdh4awkxsj40d4y9y6nvczw4.nar.xz
--2017-03-18 10:31:24--  http://cache.nixos.org/nar/00vk5ys9yyqqvifxrb5v883y0hmhxdh4awkxsj40d4y9y6nvczw4.nar.xz
Resolving cache.nixos.org... 54.192.9.115, 54.192.9.81, 54.192.9.226, ...
Connecting to cache.nixos.org|54.192.9.115|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3762668 (3.6M) [binary/octet-stream]
Saving to: ‘00vk5ys9yyqqvifxrb5v883y0hmhxdh4awkxsj40d4y9y6nvczw4.nar.xz’

100%[====================================================================================================================================>] 3,762,668   2.03MB/s   in 1.8s   

2017-03-18 10:31:26 (2.03 MB/s) - ‘00vk5ys9yyqqvifxrb5v883y0hmhxdh4awkxsj40d4y9y6nvczw4.nar.xz’ saved [3762668/3762668]


[root@nixos:~]# xz -d 00vk5ys9yyqqvifxrb5v883y0hmhxdh4awkxsj40d4y9y6nvczw4.nar.xz

[root@nixos:~]# rm -fr /nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3/*

[root@nixos:~]# cp -r 4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3/* /nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3/

[root@nixos:~]# nix-store --verify-path /nix/store/4pi2bmg1jnr8795b70lyxkq2v4slchny-font-bh-100dpi-1.0.3

[root@nixos:~]#

Source

ocharles

👍1

Most helpful comment

I've just hit this problem in a NixOS on a USB flash drive. I could not repair the store until i disabled the nix.autoOptimiseStore option. Now it seems repaired.

alexeymuranov on 31 Jan 2019

🚀1 👍1

All 21 comments

[root@nixos:~]# nix-store --repair-path /nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1
fetching path ‘/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1’...

*** Downloading ‘http://cache.nixos.org/nar/00ai1rkh1vr91zl4szk2lxzgl3wzhhiwb50vybq3pa5hgg3yn847.nar.xz’ to ‘/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11492  100 11492    0     0   162k      0 --:--:-- --:--:-- --:--:--  162k


[root@nixos:~]# nix-store --verify-path /nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1
path ‘/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1’ was modified! expected hash ‘c9dd6d0766c6630f6f8010f6a1498d061dcd1f8280b7812491b65d6fe8f4eea9’, got ‘5b010f969fc7e1971053d1d2930384e40fd3afc8dd45ab383ecef73c276d5181’

[root@nixos:~]# rm -fr /nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1/*

[root@nixos:~]# nix-store --repair-path /nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1
fetching path ‘/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1’...

*** Downloading ‘http://cache.nixos.org/nar/00ai1rkh1vr91zl4szk2lxzgl3wzhhiwb50vybq3pa5hgg3yn847.nar.xz’ to ‘/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11492  100 11492    0     0   159k      0 --:--:-- --:--:-- --:--:--  160k


[root@nixos:~]# nix-store --verify-path /nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1
path ‘/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1’ was modified! expected hash ‘c9dd6d0766c6630f6f8010f6a1498d061dcd1f8280b7812491b65d6fe8f4eea9’, got ‘5b010f969fc7e1971053d1d2930384e40fd3afc8dd45ab383ecef73c276d5181’

[root@nixos:~]# ls /nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1/*
/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1/etc:
fonts

/nix/store/5nq7sp6liwn9gxwv8d52vkgq0gwvcjw3-fontconfig-2.12.1/share:
fontconfig  xml

ocharles on 18 Mar 2017

Are you using build-sandbox-dirs to set some strange paths?

domenkozar on 20 Mar 2017

No, never heard of that option :)

ocharles on 21 Mar 2017

I have run into a similar problem. Many files in the nix store have to correct length, however every byte in them is zero.

To fix it I made a list of broken paths with nix-store --verify --check-contents. Found a machine with these paths on and verified those. On the broken machine mounted the store with rw, removed the broken paths with rm and used rsync --files-from=my-list to copy the files from the working machine into the store.

expipiplus1 on 11 Apr 2017

This issue reoccurred, although most paths were fixed manually, some are still being quite stubborn. This makes me think it's less likely to be a Nix issue.

expipiplus1 on 11 Apr 2017

@expipiplus1 not sure if this is related, but in https://github.com/NixOS/nix/issues/942 I've noticed that paths are zeroed during nix-build.

domenkozar on 11 Apr 2017

@domenkozar I don't think so, I have build-use-sandbox = false in nix.conf

expipiplus1 on 11 Apr 2017

@expipiplus1 I don't necessary have exact evidence that build-use-sandbox was the issue there, just a suspect.

domenkozar on 11 Apr 2017

Bumped into this again. Interestingly the issue persists after converting the disk from vdi to qcow2

expipiplus1 on 4 Jun 2017

Here is my best fix for this:

Find the broken path, write something to the broken files, call nix-store --repair-path.

unshare -m bash
mount -o remount,rw /nix/store
cd /nix/store/something-broken
for f in $(find . -type f); do < $f tr -d '\0' | read -n 1 || (echo something > $f); done
nix-store --repair-path .
exit

expipiplus1 on 4 Jun 2017

This is on an ext4 filesystem. @ocharles do you know what you were using?

expipiplus1 on 4 Jun 2017

Perhaps it has to do with the hardlink optimization? Do you use auto-optimize-store or have you optimized the store manually?

bennofs on 4 Jun 2017

I'm using auto-optimize-store

expipiplus1 on 4 Jun 2017

@expipiplus1 Also ext4, along with auto-optimise-store = true.

ocharles on 4 Jun 2017

👍1

I just got a new hard drive and dded the old partition over to the new drive. I didn't --verify on the old drive, but the new drive shows a few corrupt paths. --repair-path is once again not repairing them, so I highly doubt this is a drive problem.

ocharles on 28 Sep 2017

I also note that my previous steps aren't even working, but @expipiplus1's tr technique does the job.

ocharles on 29 Sep 2017

I _think_ this is because auto-optimization assumes that the hash of the link matches the content, but I believe that's not a good assumption when running --repair or --repair-path https://github.com/NixOS/nixpkgs/issues/35020#issuecomment-368188684

vcunat on 25 Feb 2018

I've just hit this problem in a NixOS on a USB flash drive. I could not repair the store until i disabled the nix.autoOptimiseStore option. Now it seems repaired.

alexeymuranov on 31 Jan 2019

🚀1 👍1

This is still a thing on Nix 2.3.4. --verify --check-contents-repair and --repair-path did not fix modified hashes, I had to delete the paths.

hyperfekt on 4 Jul 2020

I had a problem with libreoffice and some other applications not starting, so I ran sudo nix-store --verify --check-contents to see if something's wrong. It found various mismatching hashes. Next I ran sudo nix-store --verify --check-contents --repair which apparently didn't work. I ran it multiple times and every time it still found the same mismatching hashes.
Once I disabled nix.autoOptimiseStore as suggested and ran the repair command again it worked and my broken apps started working again.

felschr on 8 Aug 2020

@FelschR definitely sounds like a hardlink in /nix/store/.links/ doesnt match its contents
you likely need to nuke that dir, then redo the whole verify and repair, then you can safely turn optimize back on