Packer: hyperv-vmcx: Intermittent failures on detecting if VM exists

[FeodorFitsner]

While using hyperv-vmcx builder we are getting the following error quite often:

Packer progress:

hyperv-vmcx output will be in this color.

1 error(s) occurred:

* Failed detecting if virtual machine to clone from exists: PowerShell error: Hyper-V\Get-VM : Hyper-V encountered an error trying to access an object on computer 'AAAAAA' because the object was
not found. The object might have been deleted, or you might not have permission to perform the task. Verify that the
Virtual Machine Management service on the computer is running. If the service is running, try to perform the task
again by using Run as Administrator.
At C:\Users\Administrator\AppData\Local\Temp\2\powershell475679111.ps1:3 char:9
+ return (Hyper-V\Get-VM | ?{$_.Name -eq $vmName}) -ne $null
+         ~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (:) [Get-VM], VirtualizationException
    + FullyQualifiedErrorId : ObjectNotFound,Microsoft.HyperV.PowerShell.Commands.GetVM

We have around 50 VMs on Hyper-V host and looks like iterating through all VMs is very inefficient.

Replacing that line with Hyper-V\Get-VM -Name $vmName -ErrorAction SilentlyContinue will solve the problem.

Let me know what you think and I'll do a pull request. Thank you!

[SwampDragons]

Hi, thanks for reaching out. If we set that line to have an error action of silently continue it feels like we might as well delete the line. Before we go that route, I'll investigate whether we need that line/whether there's a better way of getting that information. I'm glad you have a workaround in the meantime, though.

[FeodorFitsner]

Hi @SwampDragons, thanks for looking into that! By that replacement command I meant you guys could fix it in Packer like that. Unfortunately, the command is hard-coded in Packer sources, so we don't have a workaround.

[SwampDragons]

Ah, I had assumed you'd already build a version of Packer with the patch applied for yourselves.

[SwampDragons]

Can you tell me a little more about your setup? The fact that it's happening intermittently feels very strange to me, unless you're running the builder with different permissions at different times, or the host machine is still loading powershell modules when you kick off the build.

[FeodorFitsner]

It is Windows Server 2019 with around 50 VMs running. When the script does Hyper-V\Get-VM | ?{$_.Name -eq $vmName} it actually fetches the details of all VMs and then iterates through the array just to find a single VM with a matching name. I think the problem is intermittent because the script operates on a timeout edge. If someone has 100 or more VMs the script would be timing out all the time. On the other hand, Hyper-V\Get-VM -Name $vmName fetches the details of exact VM and it always takes the same time.

[SwampDragons]

Ah, I see. I'll build you a patch with Hyper-V\Get-VM -Name $vmName but without the -ErrorAction SilentlyContinue and we'll see how that works.

[FeodorFitsner]

Thank you!

[SwampDragons]

You can find builds of Packer containing this patch here: https://app.circleci.com/jobs/github/hashicorp/packer/20639/artifacts

If this resolves the flakiness for you, I can merge.

[FeodorFitsner]

Thank you, but for some reason I'm getting 404 while trying to access the link you provided.

[SwampDragons]

Ah, I think I got opted in to a new dashboard in circleci which made that link bad for users who aren't logged in. Try this one. https://circleci.com/gh/hashicorp/packer/20639#artifacts/containers/0

[FeodorFitsner]

This link works, thank you!

[SwampDragons]

Just to clarify -- did you get a chance to test the build yet, or just download it successfully?

[FeodorFitsner]

Yep, it works, thank you! Sorry for the delay with reply!

[FeodorFitsner]

Is it going to be included in the next Packer release?

[SwampDragons]

Yes, it'll go out with 1.5.0, which should be released in mid to late December.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.