Packer: Proxmox builder - Error getting SSH address 500 QEMU guest agent is not running

[goffinf]

9115

This issue relates to much of the discussion in the one linked above, however, at present I have a very specific issue which I think is worth separating out.

Packer version:

goffinf@DESKTOP-LH5LG1V:~$ packer version
Packer v1.6.0

Builder: proxmox

Proxmox version:

pveversion --verbose
proxmox-ve: 6.2-1 (running kernel: 5.4.44-1-pve)
pve-manager: 6.2-6 (running version: 6.2-6/ee1d7754)
pve-kernel-5.4: 6.2-3
pve-kernel-helper: 6.2-3
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
...

I am trying to run a Packer build using the ubuntu-20.04-live-server-amd64.iso, but when the boot_command runs the build just hangs waiting to get an SSH connection to the launched instance. When the ssh_timeout is reached the build fails and the VM is destroyed.

The Packer log reports that the SSH address cannot be obtained because the QEMU guest agent isn't running. Clearly that is not installed on a fresh ISO. I also set the communicator to ssh, disabled the qemu_agent, and configured an alternate ssh_port, so I'm not sure why this is happening ?

...
  "builders": [
    {
      ...
      "communicator": "ssh",
...
      "qemu_agent": false,
      "ssh_handshake_attempts": "50",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_port": 2222,
      "ssh_pty": true,
      "ssh_timeout": "{{user `ssh_timeout`}}",

packer log:

build times-out:

From the related thread:

The new Ubuntu Server installer starts an SSH server.
The credentials are installer:
Packer wrongfully tries to connect to this SSH, thinking the VM is ready for further provisioning steps - which it is NOT.

Thanks to @JulyIghor we found a workaround.
We simply change the port packer expects the ssh server to run at to something else AND during cloud-init late_commands we override the servers port accordingly. That way once the cloud-init finishes and reboots the VM the ssh server will run at the new port - now packer picks up on that and continues provisiong as we are used to.

As a last step durng provision, we remove the conf file, essentially resettign the ssh server port back to default 22.

I have tried using a different ssh_port (2222 in the example below), but that had no effect.

For completeness, the boot_command, does result in the newly launched VM entering the correct autoinstall process (rather than the standard install dialogue) as evidenced by the following screen shots taken during a Packer debug build:

I have copied the complete Packer build file and the user-data below.

Any guidance on how to ensure that the Packer build uses ssh rather than the qemu agent would be much appreciated (or anything else you think might be the culprit).

Kind Regards

Fraser.

host.json:

{
  "builders": [
    {
      "boot_command": [
        "<enter><enter><f6><esc><wait>",
        "autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
        "<enter>"
      ],
      "boot_wait": "{{user `boot_wait`}}",
      "communicator": "ssh",
      "disks": [
        {
          "disk_size": "{{user `home_volume_size`}}",
          "storage_pool": "local-lvm",
          "storage_pool_type": "lvm-thin",
          "type": "scsi",
          "format": "raw"
        }
      ],
      "http_directory": "{{user `http_directory`}}",
      "insecure_skip_tls_verify": true,
      "iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
      "iso_file": "{{user `iso_file`}}",
      "memory": 2048,
      "name": "ubuntu-20-04-base",
      "network_adapters": [
        {
          "bridge": "vmbr0",
          "model": "virtio"
        }
      ],
      "node": "{{user `proxmox_target_node`}}",
      "password": "{{user `proxmox_server_pwd`}}",
      "proxmox_url": "https://{{user `proxmox_server_hostname`}}:{{user `proxmox_server_port`}}/api2/json",
      "qemu_agent": false,
      "ssh_handshake_attempts": "50",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_port": 2222,
      "ssh_pty": true,
      "ssh_timeout": "{{user `ssh_timeout`}}",
      "type": "proxmox",
      "unmount_iso": true,
      "username": "{{user `proxmox_server_user`}}"
    }
  ],
  "provisioners": [
    {
      "execute_command": "{{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "inline": [
        "ls /"
      ],
      "type": "shell"
    }
  ],
  "variables": {
    "boot_wait": "2s",
    "http_directory": "http",
    "iso_checksum": "caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f",
    "iso_checksum_type": "sha256",
    "iso_file": "local:iso/ubuntu-20.04-live-server-amd64.iso",
    "proxmox_server_hostname": "proxmox-002",
    "proxmox_server_port": "8006",
    "proxmox_server_pwd": "xxxxxxx",
    "proxmox_server_user": "xxxxxxxx",
    "proxmox_target_node": "home",
    "ssh_handshake_attempts": "20",
    "ssh_password": "ubuntu",
    "ssh_username": "ubuntu",
    "ssh_timeout": "10m"
  }
}

user-data:

#cloud-config
autoinstall:
  identity:
    hostname: ubuntu-20-04-base
    password: '$6$wdAcoXrU039hKYPd$508Qvbe7ObUnxoj15DRCkzC3qO7edjH0VV7BPNRDYK4QR8ofJaEEF2heacn0QgD.f8pO8SNp83XNdWG6tocBM1'
    username: ubuntu
  keyboard:
    layout: gb
    variant: uk
  late-commands:
    - sed -i 's/^#*\(send dhcp-client-identifier\).*$/\1 = hardware;/' /target/etc/dhcp/dhclient.conf
    - 'sed -i "s/dhcp4: true/&\n      dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
  locale: en_GB.UTF-8
  network:
    network:
      version: 2
      ethernets:
        ens33:
          dhcp4: true
          dhcp-identifier: mac
  ssh:
    allow-pw: true
    authorized-keys:
    - "ssh-rsa AAAAB3NzaC1yc2...."
    install-server: true
  version: 1

[paginabianca]

I don't know if you found a solution to the problem already if not, but packer is not able to determine the VMs IP address.
Normally Proxmox gets the IP addresses of VMs though the qemu-guest-agent installed on the guest.
Since you do not have the guest agent installed Proxmox does not know the VMs IP address because it cannot communicate with the VM.
There are two ways to solve this problem:

1. Set the ssh_host parameter
2. Use a preseed.cfg and install the qemu-guest-agent with d-i

For the 1st solution, you tell Packer the VMs IP manually by setting the ssh_host parameter inside the template.
As stated in the docs, when no qemu-agent is installed on the system, the ssh_host should be set.
Now you have to make sure that the VM gets assigned that exact IP address. You can do that by setting a static IP inside the VM or if you have a DHCP server, set mac_address in network_adapters and tell the DHCP server to assign the same IP to that MAC address every time.

As for the 2nd solution I recommend reading about how to use preseed.cfg files to automate the installation process on Ubuntu/Debian here.

[rgevaert]

I agree with @paginabianca . To be honest I find the second option the easiest. There are several repo's here on github that should get you a working setup.

[goffinf]

@paginabianca (Andreas) thanks for your comments. There are a couple of issues that I have been wrestling with that impact your suggested options.

When Proxmox launches the VMs, they obtain their IP via DHCP. I don't want to have a fixed IP because that has implications for other automated provisioning. So setting ssh_host isn't really an option I don't think. I did try exactly that at one point but the IP I set didn't actually get assigned so I will try your suggestion for setting the mac_address in network_adapters. I might come back to this one if option 2 continues to be problematic.

The VMs I am launching are Ubuntu 20.04 which has deprecated preseed in favour of autoinstall (which IMO is a cleaner approach). I have tried configuring cloud-config to install the qemu-guest-agent following some other examples but this has been unsuccessful to-date (see below).

TBH, I have tried so many variants its hard to remember them all, but if you can see anything else that is missing or shouldn't be there I can give it another try ?

Right now I'm using Terraform to do all the provisioning at launch time using a very basic base image. That works fine however there are a few things that I would prefer to bake at build time (the un-ending debate around build vs. launch-time provisioning aka: bake vs. fry)

Kind Regards

Fraser

#cloud-config
autoinstall:
  early-commands:
    - systemctl stop ssh # otherwise packer tries to connect and exceed max attempts
  keyboard:
    layout: gb
    variant: uk
  late-commands:
    - sed -i 's/^#*\(send dhcp-client-identifier\).*$/\1 = hardware;/' /target/etc/dhcp/dhclient.conf
    - 'sed -i "s/dhcp4: true/&\n      dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
  locale: en_GB
  network:
    network:
      version: 2
      ethernets:
        eth0:
          dhcp4: true
          dhcp-identifier: mac
  packages:
    - openssh-server
    - qemu-guest-agent
  ssh:
    allow-pw: true
    authorized-keys:
    - "ssh-rsa AAAAB3NzaC1..."
    install-server: true
  storage:
    layout:
      name: direct
   user-data:
    disable_root: false 
    package_upgrade: true
    timezone: Europe/London
    users:
      - name: goffinf
        passwd: $xxxx.
        groups: [adm, cdrom, dip, plugdev, lxd, sudo]
        lock-passwd: false
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        ssh_authorized-keys:
          - "ssh-rsa AAAAB3NzaC1....."
    write_files:
      - path: /usr/local/bin/hello-world.sh
        permissions: "0755"
        content: |
          #!/bin/bash

          FORENAME=${1:-goffinf};
          echo "Hello $FORENAME" >> /usr/local/bin/greeting;
    runcmd:
      - /usr/local/bin/hello-world.sh 'Fraser'
  version: 1

[BarisGece]

Hi @goffinf
Can you try

• to update "qemu_agent": true in _host.json_.
• adding the following commands to user-data(_cloud-config_) so that they will execute after the _qemu-guest-agent_ package is installed
  
  
  
  • sudo systemctl start qemu-guest-agent
  
  
  • sudo systemctl enable qemu-guest-agent
  
  

You may continue to get the same error at first run time. If you are sure that Qemu_agent is installed in the VM and the service is started, you should see that the IP part changes after a while when you check it through proxmox.