Packer: vmware-iso bios.hddOrder and bios.bootOrder need deep dive

[SwampDragons]

This issue combines #6197 and #6742 into one place.

Paraphrased from #6197:

VM is created from a template created by Packer using the builders "vmware-iso" on a Remote vSphere Hypervisor and the post-processors "vsphere-template".

Even though Packer generates a template with a boot order set to "hdd,CDROM", a VM created from that template has the boot order reset to "cdrom". The reason seems to be that the template .vmtx file does not have an entry "bios.hddOrder" such as "bios.hddOrder": "scsi0:0".

A workaround is to set "bios.hddOrder": "scsi0:0" in the vmx_data_post section.

We tried to solve this with PR #6204 by adding "bios.hddOrder": "scsi0:0" in the vmx file. this fixed #6197 but it caused #6742, which found that ovftool couldn't properly upload a vm when bios.hddOrder was set--or at least when it was set the way we did it. We've reached out to VMWare for help with this, because we think it may be a bug with OVFtool; but so far we have no response.

We need to find a solution that works for both situations.

[arizvisa]

Ftr, jetbrains-infra's vsphere builder plugin, packer-builder-vsphere, might solve this specific problem (iff they ever make a PR and work with us towards merging their plugin upstream that is :-/). This is because it modifies the boot-order/hdd-order after being deployed, and totally removes the need for ovftool.

(edited to add reference to other repo)

[SwampDragons]

note to self: we may be able to revert this revert if it turns out that the ovftool issue was resolved in ovftool 4.3

[arizvisa]

When reverting this, I think we should do a version check too. Lmk here before you revert and I'll add the version check to ovftool.

[arizvisa]

I think this should be closed.

I'm pretty convinced the problem is actually in versions of ESX and not ovftool. Although the symptom manifests itself with ovftool, (from reverse-engineering it) it's literally only making a soap request. VMware suggesting that it was fixed in a particular version of ovftool is hence completely misleading.

Users then testing ovftool and still encountering, and then VMware claiming that it was fixed in ESX 6.7 is pretty misleading. But in my day job, lots of vendors are like this so it's not news or anything. So, to me this proves that the issue is strictly in the server and not the client itself.

Unless they backport their patch to 6.5, this is completely unfixable on Packer's end (since we depend on ovftool working properly) and thus there's no reason for us to track it.

[arizvisa]

Actually thinking back on everything I typed, it'd probably be best to fix it with documentation.

[SwampDragons]

We definitely need to at least document it; I'd also like to put some minimal effort to see if there are some conditionals we could add that work around the bug with ofvtool and the one with the vsphere template post-processor so that we're doing our best to make things work out of the box for our users.

[aaronk1]

@SwampDragons I'm a VMware employee and pushing to get this resolved. No guarantees as I'm not directly involved on the vSphere side of the business, but I'll try.

[SwampDragons]

@aaronk1 did you make any progress on getting this addressed inside VMware?

[aaronk1]

@SwampDragons I tried. Sorry I did not get back to you. Short answer, no.

I found a few bugs that may be related, but that's about it. Is this issue still ongoing? And is it still ovftool as the suspected cause?

[SwampDragons]

I haven't looked back into it. I was just grooming old issues and trying to figure out status before trying to figure it out. I'll probably add some documentation and close, since that's where we landed a year ago.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.