Nix: Make garbage collector work if there is no free space

[edolstra]

There are several things that can prevent GC if the disk is full:

• Creating /nix/store/.trash.
• Moving stuff to /nix/store/.trash (since this may require additional allocations).
• Updating the SQLite database.

The first two should be easy to fix: if creating/moving to .trash fails, just delete store paths directly.

The third is trickier. We should at least ensure that we delete invalid paths if the database cannot be updated.

[shlevy]

For 3, can we truncate(2) the database to something a bit bigger than its native size upon close? I suppose that's racy though...

[edolstra]

That doesn't work because SQLite needs to create a WAL journal and some other stuff. Also, we already create a file /nix/var/nix/db/reserved which we delete prior to garbage collection, so this is about the case where even such hacks are not enough. (We should probably make it bigger though.)

[vcunat]

Well, with btrfs I had even problems deleting most files when the disk was full... I thought similar problems are why a non-root shouldn't be able to allocate the last bits of space (I don't know if it's still so).

[edolstra]

I think btrfs is kind of special when it comes to deleting files under disk-full conditions :-)

[vcunat]

Theoretically, for a standard FS, journalling deletions also takes space (also temporarily). Only it's so little that they'll likely have enough pre-allocated.

[CMCDragonkai]

I'd also like to bring attention to the boot partitions especially ESP. This mailing list discussion discusses this: http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021837.html

Basically it should be more obvious as to a way to force clean the boot partition.

[cleverca22]

and for any copy-on-write based FS, deleting /nix/var/nix/db/reserved wont regain any space

[vcunat]

I've had experience with btrfs where even deletion would fail for most files, due to out-of-space :-) (even though I had no snapshots or such)

[wmertens]

Idea: for btrfs make add a /nix/var/nix/db/reserved.btrfs subvolume

[aaronchall]

I ran into this problem when doing sudo nix-collect-garbage

Deleting a project and some files (maybe 10MB?) didn't give me enough space. To work around this problem I did the following:

sudo rm -rf /nix/var/log/nix/drvs/* 
sudo nix-collect-garbage -d

I now understand I could have

sudo nix-collect-garbage --max-freed 1

after freeing up a little space and it would have slowly freed more and I could have incremented the argument, then removing it.

If we can't solve this universally with code, perhaps a helpful message and link to directions would suffice?

[wmertens]

Isn't this issue solved by the 8MB reserved file that Nix automatically creates and deletes when it will need space?

@aaronchall are you using btrfs? If there is no more metadata space, deletes won't work even if you have gigabytes free. I know, it sucks.

[aaronchall]

I'm not using btrfs. I'm using a "4GB RAM 80GB Storage" Linode formatted with ext4. To be clear, I did resolve my problem, it just stems from this still unresolved bug, and I had to get help on IRC (#nixos) to do it. So I wanted my solution to be somewhere near the right place, hence my post.

I was not aware of the reserved file but that sort of thing was in my head as a solution. Perhaps it wasn't big enough? Anyways, maybe I ran into a rare corner case I don't rightly know how to describe. I think I was starting to build something late on a lark, went to bed, woke up with it failed, and rebooted my laptop losing my place in the whole thing... and just got around to revisiting using my play server.