Packer: vmware-iso machines remained orphaned in vsphere

[orcutt989]

Overview of the Issue

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted. This seems to have happened in https://github.com/hashicorp/packer/issues/3628.

I set `"keep_registered" : false" even though according to the docs this is done by default. That did not change the behavior.

Reproduction Steps

1. packer build
2. Stop build with Ctrl+C or build could fail because of other error.
3. Datastore files are deleted, and VM is deleted.
4. See vim-cmd vmsvc/unregister ### in logs.
5. VM remains orphaned in Vpshere
   
6. On a successful build the datastore files are gone, but the VM is still in inventory.
   

Packer version

1.5.5

Simplified Packer Buildfile

{
  "builders": [{
    "boot_wait": "5s",
    "boot_command": [
      "<esc>",
      "<wait>linux inst.ks=hd:/dev/fd0:ks.cfg<enter>"
    ],
    "floppy_files": [
      "ks.cfg"
    ],
    "type": "vmware-iso",
    "remote_username": "REDACTED",
    "remote_password": "{{user `vcenter_password`}}",
    "iso_checksum_type": "file",
    "iso_checksum": "http://ftp.ussg.iu.edu/linux/centos/7.7.1908/isos/x86_64/sha256sum.txt.asc",
    "guest_os_type": "centos7_64Guest",
    "iso_urls": "http://ftp.ussg.iu.edu/linux/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-Minimal-1908.iso",
    "vm_name": "devops-packer-base-centos-{{user `unix_timestamp`}}",
    "ssh_username": "REDACTED",
    "ssh_password": "REDACTED",
    "disk_size": "40000",
    "remote_host": "REDACTED,
    "remote_datastore": "ISOs",
    "remote_cache_datastore": "ISOs",
    "remote_cache_directory": "ISOs",
    "network": "VM Network",
    "network_adapter_type": "vmxnet3",
    "memory": "8000",
    "remote_type": "esx5",
    "skip_export": true,
    "format": "ova",
    "vnc_disable_password": true,
    "ssh_timeout": "40m",
    "keep_registered": false
  }
],
  "provisioners" :[{
    "type": "inspec",
    "profile": "inspec/base-os"
  }]
}

Operating system and Environment details

macOS Catalina 10.15.4

Log Fragments and crash.log files

...
==> vmware-iso: Unregistering virtual machine...
2020/05/01 15:37:12 packer-builder-vmware-iso plugin: [DEBUG] Opening new ssh session
2020/05/01 15:37:12 packer-builder-vmware-iso plugin: [DEBUG] starting remote command: vim-cmd vmsvc/unregister 140
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: Cleaning up remote path: /vmfs/volumes/ISOs/ISOs/packer046744254
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: Removing remote cache path /vmfs/volumes/ISOs/ISOs/packer046744254 (local /vmfs/volumes/ISOs/ISOs/packer046744254)
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: [DEBUG] Opening new ssh session
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: [DEBUG] starting remote command: rm -f "/vmfs/volumes/ISOs/ISOs/packer046744254"
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: Deleting floppy disk: /var/folders/pt/kk7xx4jd7wlb1y0d9gk9lws40000gp/T/packer046744254
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: [DEBUG] Opening new ssh session
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: [DEBUG] starting remote command: test -e "/vmfs/volumes/ISOs/devops-packer-base-centos-1588361364"
==> vmware-iso: Deleting output directory...
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: [DEBUG] Opening new ssh session
2020/05/01 15:37:13 packer-builder-vmware-iso plugin: [DEBUG] starting remote command: rm -rf "/vmfs/volumes/ISOs/devops-packer-base-centos-1588361364"
2020/05/01 15:37:13 [INFO] (telemetry) ending vmware-iso
...

[SwampDragons]

Thanks for reaching out. This definitely sounds like a bug.

If you're using vsphere, it's probably better to use the vsphere-iso builder instead of the vmware-iso one, since that one uses the vsphere bindings and is probably less likely to cause random orphanings. I'm not sure if it's possible to deregister vms from VSphere using the esx cli, though I'll try to investigate a solution to this when I have a chance.

[orcutt989]

Thanks for the reply @SwampDragons !

I had to abandon using the vsphere-iso builder because it does not have built-in ovftool functionality and there seems to be A LOT more functionality from the vmware-iso builder anyways.

I was using the vsphere-iso builder and had a different CI action using a container I built with ovftool in it for conversion. However our CI artifact limit is 1GB in GitLab (hard limit as I understand) and the 1.04GB image cannot be passed between jobs because of this. :-( Among other better things happening with the vmware-iso builder.

[SwampDragons]

Thanks for the feedback.

We definitely have some work ahead of us getting those builders equal to each other so having details about workflows that currently don't work is really useful.

[SwampDragons]

I just spent the better part of a day on this, and as far as I can tell there isn't a way to force vCenter to refresh this information. All the forums and docs and blog posts I could find suggest doing what Packer already does, which is use the vim-cmd vmsvc/unregister command. If I look at the vmware web client launched from the specific esx host, it does indeed unregister the vm. But the vCenter client can't seem to figure it out.

I don't think this is solvable on Packer's end, without Packer specifically using vCenter bindings. Packer can't assume those exist in the remote vmware-iso and -clone builds.

[SwampDragons]

We may be able to create a post-processor to follow up and unregister the vm from the vcenter gui...

[SwampDragons]

Are there current issues open for all of the things standing in the way of you switching to the vsphere-iso builder? I think that's the only realistic solution to this.

[orcutt989]

There were many things keeping us from using the vsphere-iso builder but the main one was -

• No built-in ovftool functionality like in vmware-iso and vmware-vmx builders

We ended up putting all the vCenter API commands in a Python script and call that on our CI pipeline. We're able to handle every possible scenario to unregister the machine unless someone manually cancels the pipeline. Its too bad we couldn't bake those API commands into Packer so it could take care of it. Oh well thanks for all the work!

[SwampDragons]

For what it's worth, I'm hoping to add ovftool functionality for exporting this release cycle.

You're welcome. I think for now I'm going to close this as an upstream bug since solving this would break some of the expectations of the builder. But if you'd be willing to share a scrubbed version of your python script, we could recommend users use it in the shell-local postprocessor to clean up if they have this issue.

[SwampDragons]

oh wait!! I think I have a partial solution that'll at least make your cleanup scripts work for the cancelled builds too. I'll push up a fix in a minute.

[orcutt989]

That's awesome! Unfortunately I am no longer with that organization so I don't have access to the script, but it was done all done via the vCenter API docs. Sorry ahh! Wish I would have grabbed it.

[SwampDragons]

Ah well that works.

Of course as soon as I said "one second I'll push this" my packet vsphere test cluster went down so it'll have to wait for tomorrow anyway.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.