Packer: QEMU builder does not over-ride arguments correctly in 1.5.0 (REGRESSION)

[howels]

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

In Packer 1.4.3, when providing our own arguments for a network adapter and disk (to use custom IP ranges and to use IDE disks instead of virtio) the over-rides are recognised and work. In Packer 1.5.0 nightly these overrides no longer work.

Reproduction Steps

Packer version

1.5.0 (nightly due to a bug in long log handling)

Simplified Packer Buildfile

{  "builders":
  [
    {
      "name": "test",
      "type": "qemu",
      "accelerator": "kvm",
      "format":"qcow2",
      "headless": "True",
      "disk_image": true,
      "disk_size": "81920M",
      "use_backing_file": false,
      "disk_discard": "unmap",
      "disk_detect_zeroes": "unmap",
      "qemu_binary": "/usr/libexec/qemu-kvm",
      "cpus": 2,
      "memory": 8192,
      "iso_url": "/home/test/test/input-image.qcow2",
      "iso_checksum_type": "none",
      "output_directory": "/home/test/test",
      "vm_name": "test.qcow2",
      "qemuargs": [
        [ "-netdev", "user,id=user.0,net=192.168.0.0/24,host=192.168.0.1,dhcpstart=192.168.0.10,hostfwd=tcp::2222-:22"],
        [ "-device", "ahci,id=ahci" ],
        [ "-drive", "file=/home/test/test/test.qcow2,driver=qcow2,node-name=hda,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk0" ],
        [ "-device", "ide-drive,bus=ahci.0,drive=drive-virtio-disk0,id=virtio-disk0" ],
        [ "-drive", "driver=qcow2,file=/home/test/test/test.qcow2-1,node-name=hdb,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk1" ],
        [ "-device", "ide-drive,bus=ahci.1,drive=drive-virtio-disk1,id=virtio-disk1" ],
        [ "-drive", "driver=qcow2,file=/home/test/test/test.qcow2-2,node-name=hdc,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk2" ],
        [ "-device", "ide-drive,bus=ahci.2,drive=drive-virtio-disk2,id=virtio-disk2" ]
      ],      "shutdown_command": "shutdown -P now",
      "ssh_username": "root",
        "ssh_password": "password",      "ssh_port": 22,
      "ssh_wait_timeout": "2400s",
      "ssh_host_port_min": 2222,
      "ssh_host_port_max": 2222
    }
  ]
}

Operating system and Environment details

CentOS 7 VM on CentOS 7 host. Latest updates installed. Using qemu 2.12

Log Fragments and crash.log files

Notice that the first disk /home/test/test/test.qcow2 appears in two -drive statements instead of one. Also, we have two netdevs instead of one.

Packer 1.5.0 nightly:

Note duplicate netdevs and that the file /home/test/test/test.qcow2 is referenced twice.

==> test: Overriding defaults Qemu arguments with QemuArgs...
2019/12/02 12:12:27 packer: 2019/12/02 12:12:27 Executing /usr/libexec/qemu-kvm: []string{"-drive", "file=/home/test/test/test.qcow2,if=virtio,cache=writeback,discard=unmap,format=qcow2,detect-zeroes=unmap", "-drive", "file=/home/test/test/test.qcow2,driver=qcow2,node-name=hda,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk0", "-drive", "driver=qcow2,file=/home/test/test/test-1,node-name=hdb,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk1", "-drive", "driver=qcow2,file=/home/test/test/test-2,node-name=hdc,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk2", "-boot", "c", "-m", "8192M", "-name", "fam-0.1.0_733-g4273c2f-dirty-snat.qcow2", "-netdev", "user,id=user.0,net=192.168.0.0/24,host=192.168.0.1,dhcpstart=192.168.0.10,hostfwd=tcp::2222-:22", "-device", "virtio-net,netdev=user.0", "-device", "virtio-net-pci,netdev=user.0,id=user.0", "-device", "ahci,id=ahci", "-device", "ide-drive,bus=ahci.0,drive=drive-virtio-disk0,id=virtio-disk0", "-device", "ide-drive,bus=ahci.1,drive=drive-virtio-disk1,id=virtio-disk1", "-device", "ide-drive,bus=ahci.2,drive=drive-virtio-disk2,id=virtio-disk2", "-vnc", "127.0.0.1:60", "-machine", "type=pc,accel=kvm", "-smp", "cpus=2,sockets=2"}
2019/12/02 12:12:27 packer: 2019/12/02 12:12:27 Started Qemu. Pid: 30844
2019/12/02 12:12:27 packer: 2019/12/02 12:12:27 Qemu stderr: qemu-kvm: -drive file=/home/test/test/test.qcow2,driver=qcow2,node-name=hda,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk0: Failed to get "write" lock
2019/12/02 12:12:27 packer: 2019/12/02 12:12:27 Qemu stderr: Is another process using the image [/home/test/test/test.qcow2]?

Packer 1.4.3: - note that we have a single netdev and the file /home/test/test/test.qcow2 is referenced only once, due to the default drive being over-ridden.

==> test: Overriding defaults Qemu arguments with QemuArgs...
2019/12/02 12:07:53 packer: 2019/12/02 12:07:53 Executing /usr/libexec/qemu-kvm: []string{"-device", "virtio-net-pci,netdev=user.0,id=user.0", "-device", "ahci,id=ahci", "-device", "ide-drive,bus=ahci.0,drive=drive-virtio-disk0,id=virtio-disk0", "-device", "ide-drive,bus=ahci.1,drive=drive-virtio-disk1,id=virtio-disk1", "-device", "ide-drive,bus=ahci.2,drive=drive-virtio-disk2,id=virtio-disk2", "-boot", "c", "-machine", "type=pc,accel=kvm", "-netdev", "user,id=user.0,net=192.168.0.0/24,host=192.168.0.1,dhcpstart=192.168.0.10,hostfwd=tcp::2222-:22", "-m", "8192M", "-smp", "cpus=2,sockets=2", "-vnc", "127.0.0.1:24", "-name", "test.qcow2", "-drive", "file=/home/test/test/test.qcow2,driver=qcow2,node-name=hda,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk0", "-drive", "driver=qcow2,file=/home/test/test/test.qcow2-1,node-name=hdb,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk1", "-drive", "driver=qcow2,file=/home/test/test/test.qcow2-2,node-name=hdc,discard=unmap,detect-zeroes=unmap,if=none,cache=writeback,id=drive-virtio-disk2"}
2019/12/02 12:07:53 packer: 2019/12/02 12:07:53 Started Qemu. Pid: 25805

[SwampDragons]

Thanks for reporting; we will take a look before the official 1.5 release.

[SwampDragons]

I think this was introduced by https://github.com/hashicorp/packer/pull/8380 in an attempt to solve https://github.com/hashicorp/packer/issues/8379. CC @bugbuilder; care to take a look and see if you can find a solution that solves both use cases so that we don't have to revert your PR?

[bugbuilder]

@SwampDragons sure, I'll take a look.

[bugbuilder]

@SwampDragons I was thinking about the flexibility that qemuargs is giving to the builder and probably adding my PR will break a lot of uses cases. For example, @howels is taking care of all devices including the default drive, in my use case I only care about the extra devices, 1 and 2, and leaving the builder to take care of the first device. Having said that, the best approach is leaving to final user the responsibility of the settings by qemuargs and revert my PR.

Thank you @howels for toy feedback! I really appreciate it.

[SwampDragons]

Okay, I appreciate that. I'll revert the PR, and if we can find a better way to make life easier on qemu users in the future without breaking a bunch of mature configs, we can do that.

[howels]

Thanks for the fast responses on this, please let me know when your next build is due and I will do my best to test this code ASAP.

[SwampDragons]

@howels here you go: https://github.com/hashicorp/packer/releases/tag/nightly

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.