Nixpkgs: 17.09.2253.559ebb7ed02 image, both minimal and graphical, fail installation with: `changing mode of '/nix/store/[...]-update-users-groups.pl' to 100444: Read-only file system`

Created on 1 Dec 2017  路  22Comments  路  Source: NixOS/nixpkgs

Issue description

The current iso cannot complete a nixos-install.

The issue has been reported by a couple users on IRC, and I was able to reproduce easily.

Known problematic isos:

nixos-install
[installation noise]
error: changing mode of '/nix/store/[...]-update-users-groups.pl' to 100444: Read-only file system
# echo $?
1

Steps to reproduce

  • Boot the image (VirtualBox is fine).
  • Create a partition (I did dos/mbr, full disk one parttion)
  • mkfs.ext4 -L nixos /dev/sda1 as in the manual
  • Mount it at /mnt as in the manual
  • nixos-generate-config --root /mnt as in the manual
  • edited the config for grub mbr boot, but this might not be necessary
  • nixos-install

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

This was manually transcribed, sorry if there are typos

 - system: x86_64-linux
 - host os: Linux 4.9.65, NixOS, 17.09.2253.559ebb7ed02 (Hummingbird)
 - multi-user?: no
 - sandbox: yes
 - version: nix-env (Nix) 1.11.15
 - channels(root): nixos-17.09.2253.559ebb7ed02
 - nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

Workaround

Use these (known working isos):

bug
Source
picture samueldr
👍3

Most helpful comment

So, after dozens of job-restart commands over several hours, I think all the four primary channels have advanced past your hack-fix commit. :tada:

picture vcunat on 4 Dec 2017
🎉3

All 22 comments

Using nixos-17.09.git.6b8833d-x86_64-linux.iso, built from the previously released channel commit worked. The issue should be from a change between 6b8833d and 559ebb7ed02 (see later comments)

samueldr picture samueldr on 2 Dec 2017 
~/.../nixpkgs/nixos $ git rev-parse HEAD
559ebb7ed02f88ce8ed5ab38e849ad01cdbebcc8

Using this command:

nix-build -A config.system.build.isoImage -I nixos-config=modules/installer/cd-dvd/installation-cd-minimal.nix default.nix

The installation can complete.

Here are the nix-info for the system from which the working image was built. Don't know what factors could make it a different image, other than plain old corruption.

~/.../nixpkgs/nixos $ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.9.61, NixOS, 17.09.git.78eed74 (Hummingbird)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 1.11.15`
 - channels(samuel): `"unstable-18.03pre120398.aeff3080d0"`
 - nixpkgs: `/etc/nixos/nixpkgs`
samueldr picture samueldr on 2 Dec 2017

I got this today when installing a clean system using the official image.

alexanderkjeldaas picture alexanderkjeldaas on 2 Dec 2017
👍1

I feel like this issue should be merged or kept open until a test is available.

alexanderkjeldaas picture alexanderkjeldaas on 2 Dec 2017

Another report on IRC, for unstable though, with unknown specific image.

samueldr picture samueldr on 2 Dec 2017

Uh oh. It seems that hydra-built nixos-minimal-17.09.2249.6b8833df272-x86_64-linux.iso is broken too.

That's a bother.

Downloaded it from there: https://hydra.nixos.org/build/65188560

Hydra-built f9390d652f9 works.

samueldr picture samueldr on 2 Dec 2017

Hydra unstable built 18.03pre121253 2f1a818d00 affected too.

https://d3g5gsiof5omrk.cloudfront.net/nixos/unstable/nixos-18.03pre121253.2f1a818d00/nixos-graphical-18.03pre121253.2f1a818d00-x86_64-linux.iso

I reproduced it the same way.

http://i.imgur.com/ylRIf2g.png

yellowgh0st picture yellowgh0st on 2 Dec 2017

It's a wonder that nixos.tests.installer.simple passes when on my hydra (with update-users-groups.pl being writable among others) it fails just like the installer on the cdrom. Are the published images built on different hosts from those that run the tests?

orivej picture orivej on 2 Dec 2017

I tried to compare machines that build nixos.iso_minimal and machines that run nixos.tests.installer.simple. These sets are almost disjoint (with wendy and 52.91.90.209 building and lucifer and packet-t2-4 testing), however occasionally the testing machines do build an iso, and the installer on the iso works, e.g. https://hydra.nixos.org/build/65178276 from two days ago.

orivej picture orivej on 2 Dec 2017

On my system (Macbook 2,1 with coreboot) nixos-17.09.2253.559ebb7ed02 even don't boot.

img_20171202_192030

17.09.2075.ac35504065 works.

davidak picture davidak on 2 Dec 2017

This is a different issue, and looks like the problem with the media (incompletely written image, or bad USB connection).

According to the hypothesis that the only issue is rw files on some build machines, one of the latest good release-17.09 images is nixos-minimal-17.09.2241.f9390d652f9-x86_64-linux.iso made by lucifer in https://hydra.nixos.org/build/65100017, and indeed it works.

orivej picture orivej on 2 Dec 2017

I suppose 52.91.90.209 is an EC2 machine built from the images before the commit https://github.com/NixOS/nixpkgs/commit/f5b3f2c5a7f2b51e80ac32fb47fd1d7d3e475ad1.

dezgeg picture dezgeg on 3 Dec 2017

I pushed a hack (574526d510bbfabd66fc251ef6054604c8221ca3) that should prevent building ISOs with such bad paths.

dezgeg picture dezgeg on 3 Dec 2017
❤1

Good idea to break invalid builders! It works. I have also confirmed that my hydra cluster that exhibits the same issue was deployed from the image with wrong permissions and times in /nix/store, and all the wrong paths existed from the moment of deployment.

orivej picture orivej on 3 Dec 2017

I think we should fix the underlying problem on 17.09 as well. Do you think it's better to have b8cc69b3 or f5b3f2c or something else?

vcunat picture vcunat on 3 Dec 2017

I think they are already all cherry-picked, but the EC2 base images are still very old or something.

dezgeg picture dezgeg on 3 Dec 2017
👍1

Right, the first way is picked. I don't know why I didn't check that before posting :-/

vcunat picture vcunat on 3 Dec 2017

@edolstra: apparently we would better regenerate EC2 images of Hydra build machines. (Or something like that; I doubt our permissions can do it.)

vcunat picture vcunat on 3 Dec 2017

In any case, with the hack in nixpkgs now, the issue shouldn't be experienced in newly published images for unstable or 17.09.

vcunat picture vcunat on 3 Dec 2017

Well we still don't have an actual channel with working ISOs since the build job very frequently ends up on the EC2 nodes. So reopening until the problem is really gone.

dezgeg picture dezgeg on 3 Dec 2017

So, after dozens of job-restart commands over several hours, I think all the four primary channels have advanced past your hack-fix commit. :tada:

vcunat picture vcunat on 4 Dec 2017
🎉3

I believe this has been fixed for a while.

dezgeg picture dezgeg on 24 Jan 2018
🎉1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

matthiasbeyer picture matthiasbeyer  路  3Comments
tomberek picture tomberek  路  3Comments
teto picture teto  路  3Comments
vaibhavsagar picture vaibhavsagar  路  3Comments
ayyess picture ayyess  路  3Comments