Nixpkgs: Upgrading EC2 from 18.09 to 19.03 fails

[jluttine]

Issue description

I installed my 18.09 EC2 instance with the links that were in https://nixos.org/nixos/download.html in February. I added a bunch of stuff to configuration.nix and switched to nixos-unstable channel. Now, after a few months, I tried to upgrade from some February nixos-unstable commit to some June nixos-unstable commit. But I get the following error:

$ nixos-rebuild switch
.....
..... many many lines here .....
.....
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
/nix/store/3a977zd2c6da1qkxfr7arpxfisvh0vrd-grub-2.04-rc1/sbin/grub-install: error: cannot find a GRUB drive for /dev/xvda.  Check your device.map.
/nix/store/vb1gr0qmwayfdfmyy9wfd661z188v5mn-install-grub.pl: installation of GRUB on /dev/xvda failed
warning: error(s) occurred while switching to the new configuration

The machine doesn't have /dev/xvda. I couldn't figure out why this is now expected if it hasn't been expected in February. How should I upgrade my NixOS on this EC2 instance, any ideas?

Steps to reproduce

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

[jluttine]

For what it's worth, the main disk is /dev/nvme0n1.

[jluttine]

As a workaround(?), I created a symlink /dev/xvda that points to /dev/nvme0n1. I have no idea whether this really works or makes sense, but at least I got rid of the error/warning. Still, I don't understand why this had suddenly broken.

[damianbaar]

it happens to me only on c5 instances of ec2, but for t2.micro or t2.xlarge seems to work well. Your workaround works for me as well 👍 but not sure how to do that declaratively within configuration.nix

[takeda]

It's basically happening to new instances that contain NVMe, for example t3.nano or t3a.nano etc:

The output of nix-info:

 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.35, NixOS, 19.03.172286.8ea36d73256 (Koi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2.2`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[damianbaar]

I guess here is the problem. I've solved it by adding boot.loader.grub.device = lib.mkForce "/dev/nvme0n1";

[domenkozar]

See https://github.com/NixOS/nixpkgs/pull/67349

[robin-engineml]

@damianbaar provided a solution that worked for me, above. Thank you.

[domenkozar]

Same in 20.03, probably needs nixops release.

[nh2]

Same in 20.03, probably needs nixops release.

What is the situation here? #67349 is closed since half a year ago, but with my nixops I still cannot boot many new AWS instance types.

What needs to be done?

[steve-chavez]

Thank you so much @damianbaar, your solution worked perfectly,

I also got the issue on nixops. In case it helps anyone, here's how I'm appliyng the fix: https://github.com/steve-chavez/postgrest-benchmark/blob/7939c3a8e9d0fe8a011b3a6e3a18f191c97579a9/deploy.nix#L95.