Nixpkgs: Building images doesn't work with Nix 1.12 on NixOS

[copumpkin]

Steps to reproduce:

1. Get a NixOS box
2. Install nixUnstable on it
3. Run nix-build -j8 --cores 8 /path/to/nixpkgs/clone/nixos -A config.system.build.image --argstr configuration /path/to/configuration.nix

Use this simple configuration.nix:

{ pkgs, lib, modulesPath, config, ... }:

{
  imports = [ "${modulesPath}/profiles/minimal.nix" ];
  environment.systemPackages = lib.mkForce [];

  fileSystems."/" = {
    device     = "/dev/disk/by-label/nixos";
    autoResize = true;
  };

  boot.loader.grub.device = "/dev/sda";

  system.build.image = import "${pkgs.path}/nixos/lib/make-disk-image.nix" {
    name = "test";
    inherit pkgs lib config;
    partitioned = true;
    diskSize = 1024;   
  };
}

The build will spit out something like:

error: changing modification time of ‘/nix/store/1hkp2n6hz3ybf2rvkjkwrzgbjkrrakzl-update-users-groups.pl’: Operation not permitted

Oddly enough, this doesn't seem to affect regular Linux running nixUnstable on the same nixpkgs version.

cc @edolstra who might have some ideas about what's different in 1.12. I'm guessing something in the builder is trying to do something unwise with the read-only /nix/store mount but that doesn't explain why it works on 1.11, when the builds are pure (the image builder runs regularly in Hydra with build-use-sandbox turned on)

[copumpkin]

Hmm, I think the issue is a bit more complicated than I paint it in the repro above. Still looking into it.

[dezgeg]

Are you sure you don't have non-canonical timestamps in the store on your host machine, or during some part of the image generation?

Something like find /nix/store/ -printf '%T@ %p\n' | grep -v '^1\.0' tells.

[mbbx6spp]

@dezgeg what is the fix to make sure Nix store timestamps are all epoch and never get set to something slightly different? :) Is there a Nix/NixOS option i should know about? I am seeing this issue on EC2 instances I built from master recently. (I am not modifying the timestamps via my own actions, I am merely attempting to build images on NixOS Ec2 images.)

[dezgeg]

Well, a similar bug in other place was fixed by 1d72474df7fa361d72a04d2a3ec4589b286ab6f2. Maybe something like this:

````
diff --git a/nixos/lib/make-disk-image.nix b/nixos/lib/make-disk-image.nix
index d4b2e338c3..8105a7cede 100644
--- a/nixos/lib/make-disk-image.nix
+++ b/nixos/lib/make-disk-image.nix
@@ -64,7 +64,7 @@ let
${channelSources}
'';

• prepareImageInputs = with pkgs; [ rsync utillinux parted e2fsprogs lkl fakeroot config.system.build.nixos-prepare-root ] ++ stdenv.initialPath;

• prepareImageInputs = with pkgs; [ rsync utillinux parted e2fsprogs lkl fakeroot faketime config.system.build.nixos-prepare-root ] ++ stdenv.initialPath;

  I'm preserving the line below because I'm going to search for it across nixpkgs to consolidate

  image building logic. The comment right below this now appears in 4 different places in nixpkgs :)

  @@ -124,7 +124,7 @@ let
  fakeroot nixos-prepare-root $root ${channelSources} ${config.system.build.toplevel} closure

  echo "copying staging root to image..."

• cptofs ${pkgs.lib.optionalString partitioned "-P 1"} -t ${fsType} -i $diskImage $root/* /
• faketime -f "1970-01-01 00:00:01" cptofs ${pkgs.lib.optionalString partitioned "-P 1"} -t ${fsType} -i $diskImage $root/* /
  '';
  in pkgs.vmTools.runInLinuxVM (
  pkgs.runCommand name
  ``` Or maybe one of thosersync`s above need some extra flags to preserve the timestamp.

[copumpkin]

Yeah, sorry I haven't tracked this down yet. It does seem like building images on images built by NixOS is currently screwed up but I'll be fixing it before 17.09 release if someone doesn't figure it out first.

[copumpkin]

Okay so adding faketime will likely fix this (I verified by manually changing timestamps on an image I had running), you're right @dezgeg. I'll try to get a patch into master and 17.09 later today and @edolstra has agreed to generate new 17.09 AMIs with the patch once it's in.

[polynomial]

Would love to buy someone a 6 pack of tasty beer to fix this (:

[copumpkin]

For now, if you have a dud instance that isn't building images properly, try this after remounting your /nix/store read-write:

sudo find /nix/store -exec touch –a -m -d @1 {} \;

Still need to find time to make the proper patch to the image builder but the above seemed to fix my instance.

[copumpkin]

Almost there, but somehow the timestamps are all coming out as 2 instead of 1 if I do the obvious thing with faketime: https://github.com/lkl/linux/issues/393

[copumpkin]

Got it, I think, at the cost of some weird noise

[polynomial]

YAY!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!