Packer: ESXi 6.7 does not support VNC

[mkarg]

Apparently ESXi 6.7 does not support VNC anymore! How to execute _boot_command_ on ESXi 6.7?

[rickard-von-essen]

Are you sure? Do you have any official references?

There have previously been some confusion around this, for example.

[mkarg]

In fact we can build with ESXi 6.0 without any problem, but not with ESXi 6.7, and ALL settings (firewall, GuestIPHack, etc. is 100% _identical_)! On 6.7 packer puts the VMDK file on the ESXi datastore, but not the VMX file (it does exist locally in %TEMP%). Then packer tries to connect VNC, which certainly fails. So it looks like the VMX upload is the problem. Strange, because we run as 'root'.

[marema31]

Hi, I have the same feeling than mkarg and a quick research has spotten me to thread on reddit providing the same information : https://www.reddit.com/r/esxi/comments/6klauz/esxi_65_vnc/.

No luck in finding official reference, but the option was available only on the old clients that are now superseeded by an HTML5 version without this option but providing an official SDK for console access: https://www.vmware.com/support/developer/html-console/html-console-21-releasenotes.html

It seems that the JetBrains packer plugin using direct API calls is also using this SDK, you should give a try. This SDK is not avaible in 6.0 and therefore this packer plugin is not working with version under 6.5 and the maintainer doesn't seems to be willing to do anything for this.

Here the issue corresponding to this limitation: https://github.com/jetbrains-infra/packer-builder-vsphere/issues/63.

I hope someday this plugin will be part of core packer since it is more avanced that my try to provide a such feature.

[adarobin]

I just spun up a nested ESXi 6.5 (yes I know, I'll try 6.7 next) and installed a vib file I made to configure the necessary things for Packer. It looks like authentication is failing for VNC:

```m-c02tv2lthtd7:rhel-vmware-packer adarobin$ packer build -only=RHEL6 -var-file=/Users/adarobin/rhel-vmware.json rhel-vmware-packer.json
RHEL6 output will be in this color.

==> RHEL6: Downloading or copying ISO
RHEL6: Downloading or copying: file:///Users/adarobin/Code/rhel-base-image/isos/rhel-server-6.10-x86_64-dvd.iso
==> RHEL6: Uploading ISO to remote machine...
==> RHEL6: Creating required virtual machine disks
==> RHEL6: Building and writing VMX file
==> RHEL6: Starting HTTP server on port 8939
==> RHEL6: Registering remote VM...
==> RHEL6: Starting virtual machine...
RHEL6: The VM will be run headless, without a GUI. If you want to
RHEL6: view the screen of the VM, connect via VNC with the password "OcyuQOnV" to
RHEL6: vnc://192.168.128.130:5900
==> RHEL6: Waiting 10s for boot...
==> RHEL6: Connecting to VM via VNC (192.168.128.130:5900)
==> RHEL6: Error handshaking with VNC: no suitable auth schemes found. server supported: []byte{0x1}
==> RHEL6: Stopping virtual machine...
==> RHEL6: Destroying virtual machine...
Build 'RHEL6' errored: Error handshaking with VNC: no suitable auth schemes found. server supported: []byte{0x1}

==> Some builds didn't complete successfully and had errors:
--> RHEL6: Error handshaking with VNC: no suitable auth schemes found. server supported: []byte{0x1}

==> Builds finished but no artifacts were created.
```

I'll try 6.7 next.

[adarobin]

Same error with 6.7

```m-c02tv2lthtd7:rhel-vmware-packer adarobin$ packer build -only=RHEL6 -var-file=/Users/adarobin/rhel-vmware.json rhel-vmware-packer.json
RHEL6 output will be in this color.

==> RHEL6: Downloading or copying ISO
RHEL6: Downloading or copying: file:///Users/adarobin/Code/rhel-base-image/isos/rhel-server-6.10-x86_64-dvd.iso
==> RHEL6: Uploading ISO to remote machine...
==> RHEL6: Creating required virtual machine disks
==> RHEL6: Building and writing VMX file
==> RHEL6: Starting HTTP server on port 8619
==> RHEL6: Registering remote VM...
==> RHEL6: Starting virtual machine...
RHEL6: The VM will be run headless, without a GUI. If you want to
RHEL6: view the screen of the VM, connect via VNC with the password "jCoa8zkE" to
RHEL6: vnc://192.168.128.173:5900
==> RHEL6: Waiting 10s for boot...
==> RHEL6: Connecting to VM via VNC (192.168.128.173:5900)
==> RHEL6: Error handshaking with VNC: no suitable auth schemes found. server supported: []byte{0x1}
==> RHEL6: Stopping virtual machine...
==> RHEL6: Destroying virtual machine...
Build 'RHEL6' errored: Error handshaking with VNC: no suitable auth schemes found. server supported: []byte{0x1}

==> Some builds didn't complete successfully and had errors:
--> RHEL6: Error handshaking with VNC: no suitable auth schemes found. server supported: []byte{0x1}

==> Builds finished but no artifacts were created.
```

[mkarg]

In _our_ case (the one this issue originated from), the last line is "Starting HTTP server on port 8619" but there never is a line "Registering remote VM" nor "Starting virtual machine". In fact, packer directly tries to start VNC, which is pretty weird. Again, using 6.0 it works, the problem is onyl with 6.7.

[rickard-von-essen]

@adarobin 6.5 is known to work. It fails because you require authentication of a type we don't support.

See #5580

[rickard-von-essen]

Please supply the information requested in the issue template:

• Packer version (packer version)
• Host platform (uname -a etc.)
• Debug log output from PACKER_LOG=1 packer build template.json.
  Please paste this in a gist.
• The _simplest example template and scripts_ needed to reproduce the bug.
  Include these in your gist.

[adarobin]

@mkarg I was able to build on 6.7 in my environment once I set vnc_disable_password to true. I'm curious to see what the differences in our templates are.

@rickard-von-essen It appears that ESXi 6.5+ ignores any VNC password configured. I ran with -debug and I was able to connect with vncviewer after Packer's authentication failed with no password even though vnc_disable_password was not set to true. I'm going to do a PR for the documentation.

[rickard-von-essen]

@adarobin great!

[DanHam]

This is an issue that's been know about for quite some time - see #5580. As noted, the docs for the VMware ISO builder haven't been updated to let people know about this - I struggled for a while with this myself! If I remember rightly the issue affects ESXi 6.0 and up...

There are a number of other issues with the VMware ISO docs when building on vSphere/ESXi that I was going to include in a PR but unfortunately haven't had the time to do - sorry @mkarg!

Mac users should note that the VNC viewer shipped with OSX can't be used when no password has been set! TigerVNC works fine though.

[DanHam]

@mkarg Just in case, if you are using a vDS and have problems, the solution is to set the port binding type for the port group to 'ephemeral' - See issue #2715 and THIS VMware doc for background. Thought I'd better mention this as this is the next issue I ran into after solving the VNC one!

@adarobin This issue and the solution is also missing from the docs if you fancy rolling that into your PR...

For completeness I needed to set the following in vmx_data to connect my VM to the vDS:

"vmx_data": {
    ...
    "ethernet0.virtualDev": "vmxnet3",
    "ethernet0.present": "TRUE",
    "ethernet0.addressType": "generated",
    "ethernet0.startConnected": "TRUE",
    "ethernet0.dvs.portgroupId": "dvportgroup-55",
    "ethernet0.dvs.switchId": "50 0b 04 db 81 66 35 5e-13 7c 09 fb 38 5e 08 93"
}

[adarobin]

@DanHam #6486 was already merged and documents the VNC issue.

I don't use vDS with Packer presently, so I have no environment to confirm if that fix works.

[mkarg]

vnc_disable_password did help on 6.7, but only if we completely disable the firewall. We are now investigaging why this is the case, as the VNC port range was explicitly opened.

[DanHam]

@mkarg

That debug log you originally had in your comment was telling you that you need to configure the firewall rules on your ESX server to allow Packer to connect in over VNC.

You need to ensure the rules persist across reboots - by default ESXi will remove any customisations to firewall rules when you reboot. See THIS gist for my notes on how to do this...

I've used those notes to configure my own system. Clearly, any changes you make to your system are at your own risk... 😉

[mkarg]

@DanHam We meanwhile found out that all works well unless we reboot ESXi. So the actual sole problem simply is that the services.xml gets overwritten (or "reset") on ESXi reboot. We are now investigating ways to work around that.

[adarobin]

@mkarg @DanHam I made a vib for ESXi that will configure the firewall for Packer and persist across reboots if it helps you any. It also configures GuestIPHack and enables SSH.

Edit: No warranty express or implied, might destroy your system, etc...

[DanHam]

@mkarg

We are now investigating ways to work around that.

EDIT: That gist I put in the comment above could help you out - GIST. Actually, that vib looks so neat I'm gonna bin that gist!!!

Otherwise @adarobin solution of using a vib looks really neat!!

[arizvisa]

Sorry about this being sorta late, but is this because go-vnc doesn't support TLS* for the authentication scheme? What authentication schemes does ESXi 6.7 support?

[adarobin]

@arizvisa Since VNC isn't officially supported on ESXi, I don't think the authentication schemes are published anywhere.

For that matter, I am able to authenticate to ESXi 6.5 and 6.7 without a entering a password even though a password is specified in the virtual machine config.

[arizvisa]

@adarobin, one way to test would be to try specifying the different auth schemes individually and seeing which ones work.

I don't have an esx instance to test against right now (or I'd do it myself), but manpage for tigervnc offers:

−SecurityTypes sec-types

Specify which security scheme to use for incoming connections.
Valid values are a comma separated list of None, VncAuth, Plain, TLSNone, TLSVnc, TLSPlain, X509None, X509Vnc and X509Plain.

Default is VncAuth,TLSVnc.

Another way (which I think depends on your vnc client) is that with vncviewer you can specify -Log *:stderr:100 to output everyting to stderr, or -Log *:file:100 -LogFile /path/to/file.log to output to a log file (https://www.realvnc.com/en/connect/docs/viewer-parameter-ref.html). The VNC protocol's handshake includes the exchange of the available authentication mechanisms, so this way the client can choose what it supports. So... I'd expect vncviewer to output what's supported and emit what it succeeded with.

Looking at go-vnc's source, it looks like there aren't many authentication schemes supported but fortunately mitchellh@ wrote it so that it's easily pluggable. So, adding support for more schemes shouldn't require any more work than just the authentication.

[arizvisa]

If somebody wants to let me know if the issue is due to go-vnc not supporting a particular auth protocol for esxi6.7 (TLS*) and what auth protocols esxi6.7 supports for vnc (via my above message), lmk.

I'm willing to implement it for go-vnc because I'm a protocol junkie and it'll actually coincide with my real job.

[adarobin]

@arizvisa ESXi 6.5 is only offering None for the authentication type. I don't have 6.7 handy to test, but I suspect it is the same.

For ESXi 6.5
```m-c02tv2lthtd7:opt adarobin$ vncviewer -Log *:stderr:100 192.168.128.130:0

TigerVNC Viewer 64-bit v1.9.0
Built on: 2018-08-21 17:06
Copyright (C) 1999-2018 TigerVNC Team and many others (see README.rst)
See http://www.tigervnc.org for information on TigerVNC.

Tue Aug 21 17:11:10 2018
DecodeManager: Detected 8 CPU core(s)
DecodeManager: Creating 4 decoder thread(s)
TcpSocket: Connecting to 192.168.128.130 [192.168.128.130] port 5900
CConn: connected to host 192.168.128.130 port 5900
CConnection: reading protocol version
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
CConnection: processing security types message
CConnection: Server offers security type None(1)
CConnection: Choosing security type None(1)
CConnection: processing security message
CConnection: processing security result message
CConnection: processing security result message
CConnection: Authentication success!
CConnection: reading server initialisation
CConnection: initialisation done
CConn: Using pixel format depth 24 (32bpp) little-endian rgb888
CConn: Using Tight encoding

For ESXi 6.0
```m-c02tv2lthtd7:opt adarobin$ vncviewer -Log *:stderr:100 192.168.128.128:0

TigerVNC Viewer 64-bit v1.9.0
Built on: 2018-08-21 17:06
Copyright (C) 1999-2018 TigerVNC Team and many others (see README.rst)
See http://www.tigervnc.org for information on TigerVNC.

Tue Aug 21 17:17:28 2018
 DecodeManager: Detected 8 CPU core(s)
 DecodeManager: Creating 4 decoder thread(s)
 TcpSocket:   Connecting to 192.168.128.128 [192.168.128.128] port 5900
 CConn:       connected to host 192.168.128.128 port 5900
 CConnection: reading protocol version
 CConnection: Server supports RFB protocol version 3.8
 CConnection: Using RFB protocol version 3.8
 CConnection: processing security types message
 CConnection: Server offers security type VncAuth(2)
 CConnection: Choosing security type VncAuth(2)
 CConnection: processing security message

Tue Aug 21 17:17:34 2018
 CConnection: processing security result message
 CConnection: processing security result message
 CConnection: Authentication success!
 CConnection: reading server initialisation
 CConnection: initialisation done
 CConn:       Using pixel format depth 24 (32bpp) little-endian rgb888
 CConn:       Using Tight encoding

[arizvisa]

Awesome. Thanks @adarobin for doing that.

But ah man, that sucks. i guess there's another protocol that we'll need to use. I saw some references to PCoIP and port 903. I'll do some digging. If there's another protocol to dick with I can probably intersect
research on that with my regular job and work on getting it done.

Thanks again guys!

[arizvisa]

Actually, it apparently is simpler than I had originally thought. Apparently it's using mks which is just a different authentication scheme (maybe it can be integrated into the auth mechanism for go-vnc somehow despite it being a differing protocol).

Someone actually implemented mks in python too:
https://github.com/rgerganov/mks with the authentication consisting in https://github.com/rgerganov/mks/blob/master/authd.py

[adarobin]

Another idea I just had would be tunneling the VNC connection over the SSH connection we already have open to the host. This would help keep things secure and negate the need to even open VNC in the ESXi firewall.

[arizvisa]

yea. probably a better solution to not expose it to just anybody. i personally wouldn't trust VMware's VNC implementation against vulnerabilities. I mean, most of the potentially vulnerable surface is post-auth, but still.. you never know.

i'll be lookin' into implementing mks for go-vnc (or packer) to support remote console against esxi properly, so expect that sometime in the future.

[anlx-sw]

i also just ran into the issue and wanted to add that I really like @adarobin 's suggestion to tunnel via ssh.

this would maybe also allow to transfer a kickstart / preconfiguration file (containing e.g. root user credentials for the new server) to the server.

I think to rely on a configuration which weakens the esxi's security by allowing basically anyone without any credentials to connect to the vm's should not be the way how this is done.

[mikemol]

This impacts ESXi 6.5 as well; we have to crack open ports on the ESXi host and use _unauthenticated_ VNC sessions in ESXi 6.5, which is a real problem.

[jdub]

A complicating factor for some users on ESXi 6.5+:

• you have to open a firewall rule to allow VNC connections
• custom firewall rules on vSphere-managed hosts may only be added with a custom VIB
• you can't install "CommunitySupported" VIBs when using Secure Boot

[mikemol]

To be clear, if you install a custom ("CommunitySupported") VIB, you're letting go of VMWare support.

[adarobin]

Per #8984 it looks like vSphere 7.0 drops VNC support completely.

[SwampDragons]

Yeah, I think the esxi builders are going to have to be essentially deprecated in favor of the recently-merged vsphere ones by the time 7.0 drops, and we'll have to move those builders to using a non-VNC way of getting the boot_command into the vm.

[arizvisa]

driver_esx and related were a hack anyways.

[swerveshot]

Yeah, I think the esxi builders are going to have to be essentially deprecated in favor of the recently-merged vsphere ones by the time 7.0 drops, and we'll have to move those builders to using a non-VNC way of getting the boot_command into the vm.

Back when the vsphere-iso builder was maintained by jetbrains they used a USB HID feature to enter input for the build via the console. I don't know if this feature carried over when HashiCorp took over.

[AndrewSav]

As far as I can see it did.

[SwampDragons]

We're working on tweaking the vsphere-iso builder to work with individual esxi hosts' web clients rather than only working with vCenter; since the vsphere-iso builder uses the usb keyboard instead of VNC, that will probably be our recommended path forward here.

[swerveshot]

We're working on tweaking the vsphere-iso builder to work with individual esxi hosts' web clients rather than only working with vCenter; since the vsphere-iso builder uses the usb keyboard instead of VNC, that will probably be our recommended path forward here.

Indeed you are, see #9791. Today I successfully tested a build of the vsphere-iso builder that works with a single ESXi host. The USB HID scan codes work great.

Demo here: https://www.youtube.com/watch?v=vIerwfvCXGg&list=PLHg__Q91ZGND0xLY27JV__GeMO9vBIQTx&index=2

[AndrewSav]

Can some one please clarify why is it better to use individual ESXi hosts rather than vCenter, it is not clear to me.

[chewrocca]

Can some one please clarify why is it better to use individual ESXi hosts rather than vCenter, it is not clear to me.

My guess would be you can run ESXi with a free license and not have to run a separate vCenter.

[swerveshot]

Can some one please clarify why is it better to use individual ESXi hosts rather than vCenter, it is not clear to me.

However vCenter adds some features that don't come with ESXi itself one would mostly use vCenter to manage a cluster of ESXi servers. If you only need a single server you don't need the licence and you don't need the extra resources required to run vCenter.

@chewrocca Yes, you could run ESXi server without a license. However you wouldn't be able to run Packer against that server because the API calls on an ESXi server without a license are read-only.

[sylviamoss]

As a follow up to what @SwampDragons and @swerveshot replied here about using vsphere-iso with a single ESXi host, here are the binaries that you can use to test if the changes for https://github.com/hashicorp/packer/issues/9791 works for you too.
binaries link

[nimDevOps]

I have been working with packer and vmware-iso builders on remote ESXi for a while now. 6.0, 6.5 worked fine with GuestIPHack and manually adding VNC port range to /etc/vmware/firewall/service.xml file. Then I upgraded ESXi to 6.7, 6.7U2 and finally to 7.0. Now I'm trying to run packer and create CentOS 7 vm and ESXi firewall blocks it.
I could not add port range to service.xml, I got Operation not permitted error:
chmod: /etc/vmware/firewall/service.xml: Operation not permitted as somebody mentioned above I would need a real license not trial to have RW access, will try it shortly...

After some researching found one interesting post How to create persistent firewall rules on ESXi. The main idea there is to create and import custom VIB file. Then custom VIBName.xml will be created under /etc/vmware/firewall and here you go: custom firewall rule set is there.

Wrapping-up: Is it possible to create PackerVNC.vib to be imported into vmware as persistent firewall rule set?

[adarobin]

@nimDevOps I made a vib which is already linked to somewhere near the top of this issue. That said, VNC support is removed from 7.0 completely so it won't help you.

[nimDevOps]

Thanks @adarobin ! I guest I will need to try vsphere-iso then.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.