Packer: Deprovision step not working with Windows on Azure

[atakacs90]

Overview of the Issue

The deprovision step gets stuck in an infinte loop with the following error:

1589606815,,ui,error,==> azure-arm: Get-Service : Cannot find any service with service name 'WindowsAzureTelemetryService'. 

1589606815,,ui,error,==> azure-arm: At C:\Windows\Temp\script-5ebf7849-741e-b3ca-88f5-82f1f7d320aa.ps1:3 char:11 

1589606815,,ui,error,==> azure-arm: + while ((Get-Service WindowsAzureTelemetryService).Status -ne 'Runni ... 

Reproduction Steps

Run a packer with azure-arm builder.

Packer version

1.5.6

Simplified Packer Buildfile

    {
    "provisioners": [
        {
        "type": "powershell",
        "inline": [
            " # NOTE: the following *3* lines are only needed if the you have installed the Guest Agent.",
            "  while ((Get-Service RdAgent).Status -ne 'Running') { Start-Sleep -s 5 }",
            "  while ((Get-Service WindowsAzureTelemetryService).Status -ne 'Running') { Start-Sleep -s 5 }",
            "  while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { Start-Sleep -s 5 }",

            "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit /mode:vm",
            "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
        ]
        }
    ]
    }

Operating system and Environment details

Windows

[atakacs90]

@danielsollondon did something change on the Microsoft side?

[jeremiahsnapp]

I noticed the same thing started happening to me yesterday. The VM just no longer has that service installed. I can't find any explanation yet about why that changed.

[SwampDragons]

This looks like something that is pretty squarely on the Azure side. I don't think there's anything Packer can do to force this to work, but I'll leave the issue open for now so you all can communicate about it.

[HarishVaranasi]

Yes, it would appear that the service is no longer coming in from microsoft.
I am not sure what the Telemetry service does to the image. But let us see if they put it back in the subsequent builds.

[danielsollondon]

Hi All, apologies, there was a recent change to the agent, you will need to update the sysprep command:

Write-Output '>>> Waiting for GA Service (RdAgent) to start ...'
while ((Get-Service RdAgent).Status -ne 'Running') { Start-Sleep -s 5 }
Write-Output '>>> Waiting for GA Service (WindowsAzureTelemetryService) to start ...'
while ((Get-Service WindowsAzureTelemetryService) -and ((Get-Service WindowsAzureTelemetryService).Status -ne 'Running')) { Start-Sleep -s 5 }
Write-Output '>>> Waiting for GA Service (WindowsAzureGuestAgent) to start ...'
while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { Start-Sleep -s 5 }
Write-Output '>>> Sysprepping VM ...'
if( Test-Path $Env:SystemRoot\system32\Sysprep\unattend.xml ) {
  Remove-Item $Env:SystemRoot\system32\Sysprep\unattend.xml -Force
}
& $Env:SystemRoot\System32\Sysprep\Sysprep.exe /oobe /generalize /quiet /quit
while($true) {
  $imageState = (Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\State).ImageState
  Write-Output $imageState
  if ($imageState -eq 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { break }
  Start-Sleep -s 5
}
Write-Output '>>> Sysprep complete ...'

I will ask for the packer docs to be updated.

[SwampDragons]

Is there a docs page we can link to that is guaranteed to have an up to date sysprep example? That seems more future-proof than us trying to update our documentation every time something changes on the Azure side.

[danielsollondon]

@SwampDragons - agreed, one doc is in the process of being published that has this command, let me come back to you in 1 week, when I have a location.

[HarishVaranasi]

The Telemetry service is still not coming in, has anyone done a successful build yet ?

[atakacs90]

I have.
Either remove the Telemetry service line, or use what @danielsollondon suggested.
Microsoft told me that the telemetry service was merged into the RdAgent service.

[mjiderhamn]

Is there a docs page we can link to that is guaranteed to have an up to date sysprep example?

https://docs.microsoft.com/sv-se/azure/virtual-machines/linux/image-builder-json#default-sysprep-command maybe?
(Seeing linux is part of the URL is somewhat confusing though)

[centran]

This does seem like an issue on the Azure side. I believe what is happening is the guest agent is updating something on the image as it is being built that causes their market images to not generalize. The sysprep log is complaining about a missing vmagent dll and packer will just keep looping IMAGE_STATE_UNDEPLOYABLE if you used the MS Azure guide for packer or hashicorp's guide.

The reason it might be hard to reproduce is that both guides go immediately to sysprep with no other steps and thus there is a race condition where sysprep can finish before the guest agent updates. It might also be an issue with how Azure sysprep their images every month for patch tuesday. They might be doing something that only shows up occasionally.

This issue can currently be replicated by placing a "pause_before": "5m" (sometimes it works with as little as 2 minutes) in the powershell provisioner that syspreps the image. That gives the agent enough time to update and cause the issue.

Immediately generalizing a Windows image in packer is kind of pointless and defeats the purpose. A hackish way I am dealing with this is the very first thing to provision is stopping Windows Guest Agent and RDagent services. Then proceed with how you customize the image but if you use a provisioner that restarts Windows you need to stop those services again (I did not test restarts).

After you are done customizing you need to restart the services and then immediately run sysprep after their status comes back as running.

This hack can be done by adding two powershell provisioner blocks., One as the very first thing run and the second right before the sysprep/generalize. (I have a pause on the second one that starts the services back up so that any customization scripts before that have time to finish. That pause might not be needed for someone else.)

{
    "type": "powershell",
    "inline": [
        "while ((Get-Service RdAgent).Status -ne 'Running') { Start-Sleep -s 5 }",
        "while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { Start-Sleep -s 5 }",
        "& net stop WindowsAzureGuestAgent",
        "& net stop rdagent"
    ]
}

{
    "type": "powershell",
    "pause_before": "1m",
    "inline": [
        "& net start WindowsAzureGuestAgent",
        "& net start rdagent",
        "while ((Get-Service RdAgent).Status -ne 'Running') { Start-Sleep -s 5 }",
        "while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { Start-Sleep -s 5 }"
    ]
}