Nixpkgs: AWS ARM A1 instances

[memberbetty]

Issue description

Amazon supports ARM now (https://aws.amazon.com/ec2/instance-types/a1/). Is it possible already to use nixops to deploy to arm today with just modifying a tiny part of the declaration?

[rehno-lindeque]

I happen to have just done my first nixops deploy to an A1 instance.

It works, but requires some bootstrapping steps:

1. You need to have access to a aarch64 build machine (I have a raspberry pi 3B+) and turn on distributed builds in your own nix config. Be prepared to set aside some days for compiling. (Alternatively you can attempt to cross-compile, but that's a lot of work and I didn't succeed at it for this use case.) For me, the remote build setup looks like this:

{
nix = {
    distributedBuilds = true;
    buildMachines = [ 
        # Automatically build arm expressions on my raspberry pi
        # First ensure that non-interactive ssh to 192.168.0.xxx works
        # See https://nixos.wiki/wiki/Distributed_build
        { hostName = "192.168.0.xxx";
          sshUser = "root";
          sshKey = "/home/me/.ssh/id_xxxxx"; # ssh-copy-id to your machine
          system = "aarch64-linux";
          maxJobs = 1;
          supportedFeatures = [ "big-parallel" "kvm" ];
        }
    ];
};
}

1. You must build your own nixos AMI since there isn't an official one. I described roughly how I managed this here: https://discourse.nixos.org/t/prototype-aarch64-ami-builder-for-amazon-ec2/2431. (Once you have an AMI you can also create a spot request manually and use it as a faster remote builder for the next steps).

In your nixops deployment you have to set:

{
  nixpkgs.localSystem = lib.systems.examples.aarch64-multiplatform;
  deployment.ec2.ami = "ami-xxxxxxxxxxxxxxxxx";
}

1. Alter the efi/grub stuff needed to boot successfully:

{
  fileSystems."/boot" = {
    device = "/dev/disk/by-label/ESP";
    autoResize = true;
  };

  boot.loader.grub.efiSupport = true;
  boot.loader.grub.efiInstallAsRemovable = true;
  boot.loader.grub.device = lib.mkForce "nodev"; # forcibly override the amazon image in vanilla nixpkgs
  boot.loader.timeout = lib.mkForce 5; # forcibly override the amazon image in vanilla nixpkgs
  boot.loader.grub.copyKernels = true;
}

At least, this has worked for me. I blindly cribbed the grub setting from elsewhere.

I have my personal nixops network config here in case that helps anyone: https://github.com/rehno-lindeque/remotebuilders-network

[memberbetty]

Thanks for reporting on your progress.

I am mostly interested in a fully automated cross-compilation setup to eliminate the dependency on Intel.

[rehno-lindeque]

I suspect there's a decent amount of work left to do on that front.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[thefloweringash]

IIUC: We have images built natively by hydra as part of the regular nixos jobsets. What we don't yet have is a set of images registered by the NixOS entity on Amazon. To use a hydra built image, the end user must upload and register their own AMI.

[stephank]

I see two jobsets on Hydra. Is this still correct, or is one of them simply old?

• https://hydra.nixos.org/job/nixos/release-20.09/nixos.amazonImage.aarch64-linux
• https://hydra.nixos.org/job/nixos/release-20.09-aarch64/nixos.amazonImage.aarch64-linux

What's the next step to get these uploaded to Amazon on the NixOS account? 🙂

[stephank]

For what it's worth, I verified these images are working nicely, specifically this build: https://hydra.nixos.org/build/129496702

Maybe I'm missing something, but it appears it's not possible to use aws ec2 import-image --architecture arm64 directly. It complains with Parameter architecture=arm64 has an invalid format, despite the manpage saying it should work. Instead, import-snapshot + register-image is what you want.

I'm not sure if the code that pushes the existing x86 images is public somewhere. I'd be happy to take a look, at least.

If anyone else wants to try this:

• You'll need to create an IAM role vmimport, documented here: https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role

• Place the VHD from the above Hydra link somewhere on S3.

• aws ec2 import-snapshot --disk-container Format=VHD,UserBucket="{S3Bucket=YOUR-BUCKET,S3Key=nixos-amazon-image-20.09.1599.d105075a1fd-aarch64-linux.vhd}"

  Replace YOUR-BUCKET and maybe the filename.

• Monitor progress with aws ec2 describe-import-snapshot-tasks until it says completed

• aws ec2 register-image --cli-input-json "$(<register-image.json)"

  You can use CLI arguments for this, but I preferred providing this input as JSON.

  {
  "Name": "NixOS-20.09-aarch64",
  "Description": "NixOS-20.09-aarch64",
  "VirtualizationType": "hvm",
  "Architecture": "arm64",
  "EnaSupport": true,
  "SriovNetSupport": "simple",
  "RootDeviceName": "/dev/xvda",
  "BlockDeviceMappings": [
    {
      "DeviceName": "/dev/xvda",
      "Ebs": {
        "DeleteOnTermination": true,
        "SnapshotId": "snap-00000000000000000",
        "VolumeSize": 3,
        "VolumeType": "gp2"
      }
    }
  ]
  }
  

  Replace snap-00000000000000000 with your actual snapshot ID.

  You can get a full template with all options using aws ec2 register-image --generate-cli-skeleton

[thefloweringash]

Maybe I'm missing something, but it appears it's not possible to use aws ec2 import-image --architecture arm64 directly. It complains with Parameter architecture=arm64 has an invalid format, despite the manpage saying it should work. Instead, import-snapshot + register-image is what you want.

@kirelagin has also run into this, see https://github.com/NixOS/nixpkgs/pull/62042#issuecomment-546388152. I only just noticed that he was talking about import-image. The script in nixpkgs uses import-snapshot + register-image.

I'm not sure if the code that pushes the existing x86 images is public somewhere. I'd be happy to take a look, at least.

The script lives in nixpkgs at nixos/maintainers/scripts/ec2/create-amis.sh.

[stephank]

Ah, thanks for the pointer! I should've just grepped for those commands. 🙂

But looks like a dead end for me. The script is fine, it just needs to be run against the aarch64 build output, looks like. But I can't find any reference to the script in the NixOS org here on GitHub, so no clue how it is connected.

Looking at the way nixos/modules/virtualisation/ec2-amis.nix is updated, I'm thinking this is possibly a manual process?