Packer: Packer crashes when provisioning a Windows VM to Azure using ssh

[kirannhegde]

Overview of the Issue
Here are my environment details:
-Windows Server 2019 LTSC English(This is the system from where the Packer is run)
-Packer version: 1.6.0
-Go lang 1.14.4

I am trying to create a Windows server 2019 image on Azure using Packer. I have been able to get it working with WinRM. However, my experience has been that WinRM is flaky and I would like to build the image over SSH. We have a large number of chocolatey packages to install (in excess of 50). A lot of them require reboots(we make use of the windows-restart provisioner). All of these packages are installed via PowerShell provisioners. It's been my experience that installing such a large number of packages using WinRM is error-prone. In the past, I have had success using SSH for creating golden images on XenServer. I have successfully used SSH for communication as well as running the provisioners in my XenServer environment. Hence, I am trying to use SSH for initial communication with Azure as well as installing all of the different packages using Powershell provisioners.
When I try using SSH communication to build a Windows server 2019 image on Azure, the ssh_username and ssh_password parameters are being completely ignored by Packer. Ultimately, the packer run terminates with a crash in Packer.exe. In the logs, I see an error of the following nature:
_"The secret retrieved from https://pkrkvfxfgy2w60v.vault.azure.net/secrets/packerKeyVaultSecret/b55be864bc3e482cb7e8284b6c63953f is an empty string_

This is happening in spite of specifying a ssh username and ssh password.

What I am trying to do is something like this:
Step 1: Using Packer, I create an image that has Microsoft port of OpenSSH installed. In this step, I create a windows username and a password for this user name. I add this user to the "Administrators" group on the local system.
An image is created. I use WinRM for step 1(i have to use WinRM here as SSH is not enabled by default on Windows servers)
Step 2: In step 2, I reference this image built-in step #1 to create another image. I use the .json file pasted earlier for this step.
As part of this step, I did like to use SSH.

The packer log file can be found at:
https://gist.github.com/kirannhegde/4db1296be5bfddcc9823b0d37952196a

Here is my .json file used for step 1 of the Packer run:

{
  "builders": [{
    "type": "azure-arm",

    "client_id": "{{user `client_id`}}",
    "client_secret": "{{user `client_secret`}}",
    "subscription_id": "{{user `subscription_id`}}",
    "tenant_id": "{{user `tenant_id`}}",

    "build_resource_group_name": "{{user `build_resource_group_name`}}", 
    "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
    "managed_image_name": "Packer-BuildAgent-cvad-step1-EnableSSH-{{isotime \"200601020304\"}}",

    "os_type": "{{user `os_type`}}",
    "image_publisher": "{{user `image_publisher`}}",
    "image_offer": "{{user `image_offer`}}",
    "image_sku": "{{user `image_sku`}}",
    "image_version": "{{user `image_version`}}",


    "communicator": "winrm",
    "winrm_use_ssl": true,
    "winrm_insecure": true,
    "winrm_timeout": "3h",
    "winrm_username": "packer",

     "azure_tags": {
        "dept": "Packer-Engineering",
     "org": "Packer-SES-Build",
        "task": "Packer-SES Build agent"
    },

    "os_disk_size_gb": "{{user `os_disk_size_gb`}}",
    "vm_size": "{{user `vm_size`}}",
    "virtual_network_name": "{{user `virtual_network_name`}}",
    "virtual_network_subnet_name": "{{user `virtual_network_subnet_name`}}",
    "virtual_network_resource_group_name": "{{user `virtual_network_resource_group_name`}}"


  }],

  "provisioners": [
       {
          "type": "powershell",
          "inline": [
            "net user kiranh abcdefg@12345 /add",
             "net localgroup administrators kiranh /add",
             "Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0",
             "Start-Service sshd",
             "Set-Service -Name sshd -StartupType 'Automatic'",
             "New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22"
           ],
           "elevated_user": "packer",
           "elevated_password": "{{.WinRMPassword}}",
           "execution_policy": "unrestricted"
       },
       {
          "type": "powershell",
          "inline": [
            "Add-WindowsFeature Web-Server",
            "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit",
            "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
        ]
       }
  ]
}

Here is my .json file used for step 2 of the Packer run:

{
  "builders": [{
    "type": "azure-arm",

    "client_id": "{{user `client_id`}}",
    "client_secret": "{{user `client_secret`}}",
   "subscription_id": "{{user `subscription_id`}}",
    "tenant_id": "{{user `tenant_id`}}",

    "build_resource_group_name": "{{user `build_resource_group_name`}}", 
    "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
    "managed_image_name": "Packer-BuildAgent-cvad-step2-InstallPackages-{{isotime \"200601020304\"}}",

    "os_type": "{{user `os_type`}}",
    "custom_managed_image_name": "{{user `custom_managed_image_name`}}",
    "custom_managed_image_resource_group_name": "{{user `build_resource_group_name`}}", 

    "communicator": "ssh",
    "ssh_username": "{{user `ssh_username`}}",
    "ssh_password": "{{user `ssh_password`}}",
    "ssh_wait_timeout": "3h",

    "azure_tags": {
        "dept": "Engineering",
    "org": "SES-Build",
        "task": "SES Build agent"
    },

    "os_disk_size_gb": "{{user `os_disk_size_gb`}}",
    "vm_size": "{{user `vm_size`}}",
     "virtual_network_name": "{{user `virtual_network_name`}}",
     "virtual_network_subnet_name": "{{user `virtual_network_subnet_name`}}",
     "virtual_network_resource_group_name": "{{user `virtual_network_resource_group_name`}}"


  }]

Is there anything I could do to get the ssh communicator working for windows Azure VM's?

Regards,
Kiran Hegde

[nywilken]

Hi @kirannhegde thanks for reaching out. We will try to take a look at this and see what's going on as soon as we have a chance. I see Packer is crashing when trying to access the Storage Profile information for the image but it is not immediately obvious why.

Re: Is there anything I could do to get the ssh communicator working for windows Azure VM's?
Have you tried reaching out to the Packer community forum?

When it comes to help with this kind of initial configuration, community forum or mailing list are generally more useful. Issues opened here on the Packer issue tracker are only viewed by a small handful of developers who work on the tool, and we don't always have the most depth or experience when it comes to custom issues with particular build configurations.

[kirannhegde]

Hello @nywilken Thanks for the quick response. Let me know if you did like any additional information. In the meantime, I will reach out to the community to see if they have a solution.

Regards,
Kiran Hegde

[burninmedia]

Got this same issue, I'm seeing it with Windows 2012. Looking forward to a resolution.

[paulmey]

The Windows flow assumes WinRM and as such tries to provide a certificate to the VM to use for that purpose. Apparently, the builder is smart enough not to create the certificate in key vault, but not smart enough to not try to configure the Windows VM for WinRM...

[jsturtevant]

Does the Windows flow have a hard requirement on WinRM? If the issue with configuring the windows VM for winRM was fixed do you know any reason ssh wouldn't work?

[ChulHul]

I'm also running into this issue. Any chance it'll be fixed soon?

[kirannhegde]

Hello @burninmedia @ChulHul @jsturtevant When I initially posted this issue, I was having issues provisioning over WinRm. However, I managed to get everything working using WinRm, and now I no longer rely upon SSH for my provisioning. Is there something that you are trying to do specifically with SSH that can't be done over WinRM? I am asking this so that I could share my experience with WinRm that could be helpful.

Thanks,
Kiran Hegde

[ChulHul]

I was hoping to be able to run packer from a linux machine to build out windows images (with the windows image using the powershell provisioner).

[SwampDragons]

Packer's WinRM client works from Linux too :)

[ChulHul]

Hah, it worked, i just needed to open up some ports. Thanks alot.

[kirannhegde]

Hello, @ChulHul Glad that you have got it working now. Just for my knowledge,what ports did you have to open up to get it working?

Thanks,
Kiran Hegde

[Codypinto23]

@ChulHul also, wanted to follow-up and ask which ports you needed to open up to get this to work? Thanks!