Packer: Support VNC-over-websocket in vmware-iso provider for ESXi 7.0

[lamw]

Feature Description

The vmware-iso and vsphere-iso Packer provider relies on ESXi's built-in VNC server to send keystrokes into the Virtual Machine for OS commands/installations. In vSphere 7 (which officially GA'ed today), ESXi 7.0 no longer contains the built-in VNC server and this would impact all users who has/will upgrade to latest ESXi release.

Here's a snippet from official ESXi release notes:

Removal of VNC Server from ESXi
In vSphere 7.0, the ESXi built-in VNC server has been removed. Users will no longer be able to connect to a virtual machine using a VNC client by setting the RemoteDisplay.vnc.enable configure to be true. Instead, users should use the VM Console via the vSphere Client, the ESXi Host Client, or the VMware Remote Console, to connect virtual machines. Customers desiring VNC access to a VM should use the VirtualMachine.AcquireTicket("webmks") API, which offers a VNC-over-websocket connection. The webmks ticket offers authenticated access to the virtual machine console. For more information, please refer to the VMware HTML Console SDK Documentation.

Reference: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-vcenter-server-70-release-notes.html

VMware does provide an HTML5 Console SDK for 3rd party integration with the VM Console which uses websockets. As mentioned above, VNC-over-websockets is supported and should be the main method to interact with the guest when needing VNC access. It would be great to have Packer consume and support this new and more secure method.

Use Case(s)

• Building VMware-based Images using either vmware-iso and/or vsphere-iso provier

[RAGNOARAKNOS]

Would it not be better to update packer to support the VMware USB Scan Code insertion (PutScanCodes()) from the API?

While I understand the want to minimise change, it may be a more pragmatic or supportable approach long term.

[adarobin]

I think only vmware-iso is impacted.

It looks like vsphere-iso already implements the suggestion by @RAGNOARAKNOS
https://github.com/hashicorp/packer/blob/master/builder/vsphere/driver/vm_keyboard.go

[SwampDragons]

Thanks for opening. I've marked this issue as part of our next release milestone, but it's unlikely that we'll be fixing it for the vmware-iso builder; likely our solution is going to be to say "use vsphere-iso", and testing/fixing up the vsphere-iso builder if necessary to make this work. I think we're going to move towards deprecating the vmwrae-iso and vmware-vmx builders in favor of the vsphere-* ones because it's become so difficult to work around the various way newer releases of ESX are locking down their systems.

[lamw]

@SwampDragons I actually work for VMware and I just found out about the change :) and figure I'd report it as I know many folks are still using the vmware-iso (including myself). I'd love to switch over to vsphere-iso but it was missing a few building an OVA, which I think the latest version now supports. If I can port my builds over, then I'd say others _should_ be able to as well but there were some gaps when I had evaluated my builds against vsphere-iso. Assuming there's parity between the two and folks can simply update the provider and with minor changes, then this would be great for everyone to use single provider which relies on vSphere API rather than SSH :)

[jhawk28]

@SwampDragons what about supporting the VMWare Fusion and Workstation using vmware-iso?

[jhawk28]

A benefit of keeping the vmware-iso would be using the ESXi free server. The vsphere-iso only works with a vcenter installation.

[SwampDragons]

I agree but with every subsequent esx release it's getting harder to support that. We're not going to delete the builders, I'm just not sure it makes sense to maintain them for releases like esx 7.0 which are going to remove VNC altogether.

[SwampDragons]

and yeah, we're not removing workstation and fusion support -- I was thinking purely in terms of the remote/esx build functionality

[jhawk28]

@lamw any idea what the specifics of the protocol are? I'm trying to test connecting with the gorilla/websocket and passing it to the go-vnc library, but it doesn't look like it ever receives a response after it sends the protocol version.

[lamw]

@jhawk28 I personally don't know the specifics, but this is all part of our vSphere HTML5 VM Console SDK which provides websockets access and should have what you need for the VNC bits. In addition to the download, there's programming guide (here's online version)

[jhawk28]

@lamw the SDK has a js impl, trying to do a Go based test.

[lamw]

@jhawk28 Ah, yea I'm not too sure. Let me loop in some of the VMwar Engr folks who've been helping out with Terraform provider, they might be able to help out here as well.

[jhawk28]

@lamw this is where it dies: https://github.com/mitchellh/go-vnc/blob/master/client.go#L328

// get webmks ticket from server
acquireTicket, err := vm.AcquireTicket(ctx, "webmks")
if err != nil {
    return err
}

// example on how to build the url
link = fmt.Sprintf("wss://%s:%d/ticket/%s", acquireTicket.Host, acquireTicket.Port, acquireTicket.Ticket)

dialer := &websocket.Dialer{
    Proxy:           http.ProxyFromEnvironment,
    Subprotocols:    []string{"binary", "vmware-vvc"},
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}

nc, _, err := dialer.Dial(link, nil)

if err != nil {
    return err
}
defer nc.Close()

// example of connecting using a VNC client
ccconfig := &govnc.ClientConfig{
    Auth:      []govnc.ClientAuth{new(govnc.ClientAuthNone)},
    Exclusive: false,
}
// TODO this is breaking with an EOF after it gets the protocol version from the server and then requests security codes
c, err := govnc.Client(&ConnWrapper{nc: nc}, ccconfig)
if err != nil {
    return err
}
defer c.Close()

[ahmedcooll]

Hi ,Am trying the same using java code . Right now am using the following webmks which is provided by vmware but with proxy server , not able to reach the Host.
Is that possible to done it via java code . Please guide me .
wmks.connect(url);

Thanks in advance !

[sylviamoss]

Hello there! Is the current vsphere builder version a solution for this?
I've been working to make it possible to run the vsphere-iso builder for single ESXI server and, in case you want to try it out, the solution is within these binaries: https://app.circleci.com/pipelines/github/hashicorp/packer/6931/workflows/3ce57e21-d1d3-4afa-b7f2-1cd1d9687452/jobs/77935/artifacts

You can follow up on some of the discussion of https://github.com/hashicorp/packer/issues/6482 and https://github.com/hashicorp/packer/issues/9791 to understand better what I did and what it was trying to solve.

[sylviamoss]

On v1.6.3 we'll release an option usb_keyboard that will use USB scan codes to send the boot command to the VM, this will be the option to use for everyone that is using ESXi 6.7+ with a paid license.
For free licensed ESXi host we are still figuring out how to implement the VNC over websocket and we ask a little more time to have that implemented.

[markpeek]

@sylviamoss @jhawk28 I wrote up a quick test (elaborating on the above code snippet) that illustrates how to connect to a VM with VNC over websockets. The error above was remedied via switching to a binary frame.

[sylviamoss]

@markpeek Perfect! I'll work on adding that to Packer. Thank you!

[sylviamoss]

A follow up to this. Next release we'll be releasing a vnc_over_websocket option and you can use it instead of the usb_keyboard I said before. The usb_keyboard will be replaced by vnc_over_websocket.
With vnc_over_websocket = true Packer will do what it says and will connect to the VNC over a websocket proxy for hosts running free or paid licenses.

[jdub]

Nice! 👏🏻

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.