Windowsserverdocs: Working at all in 2019?

[sheik124]

Anyone successfully using DDA on 2016 v1709 or on 2019?

https://social.technet.microsoft.com/Forums/azure/en-US/9a556653-88e0-4f98-a6d2-aa55f033546a/win-server-2016-v1709-element-not-found-when-starting-vms-with-passthrough-gpus?forum=virtualmachingmgrhyperv

I am having the same issue.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 06fc7832-1655-f552-de28-b99bf2ab092f
• Version Independent ID: 2a99f3c3-3472-1a5a-0a22-b99615108980
• Content: Deploy graphics devices using Discrete Device Assignment
• Content Source: WindowsServerDocs/virtualization/hyper-v/deploy/Deploying-graphics-devices-using-dda.md
• Service: na
• Product: windows-server-threshold
• Technology: hyper-v
• GitHub Login: @chrishuybregts
• Microsoft Alias: chrihu

[riverar]

No issues with Windows Server 1903 18362.175, though this is not the same SKU.

[sheik124]

That's the GUI-less Windows "as a service" Server, right? I thought about
imaging, installing Hyper-V Server (free), and then using Veeam to turn my
current WS2019 install into a VM. Not sure if it would work though.

I can get DDA working with other peripherals, NVMe SSDs, and my USB3
controllers, but it hasn't worked on any graphics cards I've tried.

On Mon, Jul 29, 2019, 4:05 PM Rafael Rivera [e-mail]
wrote:

No issues with Windows Server 1903 18362.175, though this is not the same
SKU.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/windowsserverdocs/issues/3083?email_source=notifications&email_token=AARUVZWIXE4ZYRKFSUSAPG3QB5ERDA5CNFSM4ICI6B72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3B3GEA#issuecomment-516141840,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARUVZXAKJWYB73R7WUXUW3QB5ERDANCNFSM4ICI6B7Q
.

[ultnrg]

No issues with Windows Server 1903 18362.175, though this is not the same SKU.

Are you able to share your hardware configuration for working GPU passthrough using DDA?

I've just installed Server 19 1903 with the same error message posted above on three different hardware platforms. Everything worked fine with the same hardware on Server 16 1607, but any later version of 16 or any version of 19 has refused to work on any of the 3.

[riverar]

Hi @ultnrg I'm currently using an AMD Ryzen box, hosting several Gigabyte GeForce GTX 1660 cards, running Windows Server 1903 18362.239. How about you? AFAIK, this won't work properly with AMD graphic cards due to lack of SR-IOV support (but I haven't confirmed that).

EDIT: Let's continue working in the forums https://social.technet.microsoft.com/Forums/azure/en-US/9a556653-88e0-4f98-a6d2-aa55f033546a/win-server-2016-v1709-element-not-found-when-starting-vms-with-passthrough-gpus?forum=virtualmachingmgrhyperv

[aburlak]

Just installed Windows Server 2019 semi-annual (1909) as a Hyper-V host, got the 'Element not found' error when trying to start a VM with GPU attached.

Setting two policy settings below has resolved the error for me (no reboot was required, either).

Registry key path (create it if not present): HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\HyperV

Registry values:
RequireSecureDeviceAssignment = 0 (REG_DWORD)
RequireSupportedDeviceAssignment = 0 (REG_DWORD)

It looks like this policy is yet another "whitelisting"/opt-in mechanism - vendors have to "opt in" for the DDA support for GPU virtualization.
Setting this policy allows the GPU to work even if not supported by the vendor in this configuration.

More information is here:
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/dispmprt/ns-dispmprt-_dxgkarg_getvirtualgpuprofile

[AlpineWhite]

Just installed Windows Server 2019 semi-annual (1909) as a Hyper-V host, got the 'Element not found' error when trying to start a VM with GPU attached.

Setting two policy settings below has resolved the error for me (no reboot was required, either).

Registry key path (create it if not present): HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\HyperV

Registry values:
RequireSecureDeviceAssignment = 0 (REG_DWORD)
RequireSupportedDeviceAssignment = 0 (REG_DWORD)

It looks like this policy is yet another "whitelisting"/opt-in mechanism - vendors have to "opt in" for the DDA support for GPU virtualization.
Setting this policy allows the GPU to work even if not supported by the vendor in this configuration.

More information is here:
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/dispmprt/ns-dispmprt-_dxgkarg_getvirtualgpuprofile

I wanted to clarify this statement. It allows the device to be seen by the guest OS. However, something is still blocking actual utilization. I've gone through Ubuntu 18.04, 19.04, 19.10, RHEL8, Fedora 31, Debian 9, Debian 10, Debian EDU 11, and Windows 10 Pro. No OS will properly load GPU drivers. Linux hosts fail to boot when using the GPU and Windows throws an error 43 after drivers are installed.

HW:
Dell R720
2x E5 2670v2
128GB DDR3 1600
Server 2019 Datacenter
Tested with Nvidia P2000, GTX Titan, and AMD RX580.

Like others, I have no problem passing through USB3 devices, SSDs, network adapters. The problem happens only on GPUs.

I would love to see a fix or workaround from MS, but after 2.5 years (broke in 2016 v. 1709) my hopes are low. Software QA has been.... lacking the past couple years.

[riverar]

@AlpineWhite The driver issue you're seeing is due to NVIDIA and AMD deliberately checking for virtualization and "failing" when detected. Xen, Hyper-V, VMware, and others are explicitly unsupported. You can workaround this by changing the CPUID leaf strings the driver looks for (use a hex editor and find HyperV, for example) then resigning the drivers. It's sad, I know.

[AlpineWhite]

I knew that about Nvidia. AMD should be OK though. Here's a screenshot of an RX580 being passed through KVM without issue (and carrying all the signs of being a VM). The problem is definitely within Hyper-V.

[riverar]

@AlpineWhite Are you sure your hv_vendor_id isn't altered? Or kvm property set to off? I just checked your version of atikmdag.sys (25.20.15031.9002) and it's still checking for VMware, XenVM, KVM, AliyunKVM, and Microsoft Hv.

[RdGameboy]

You can workaround this by changing the CPUID leaf strings the driver looks for (use a hex editor and find HyperV, for example) then resigning the drivers. It's sad, I know.

@riverar Can you explain this process please? Or link to a guide? I've got everything up and running except for that darn code 43!

[riverar]

@RdGameboy Sure. Here's a rough guide for NVIDIA drivers:

1. Extract your drivers on disk.
2. Navigate to /path/to/extracted/drivers/Display.Driver and expand -R nvlddmkm.sy_ nvlddmkm.sys.
3. Open nvlddmkm.sys in a hex editor (e.g. XVI32)
4. Find XenVMMXenVMM
5. Observe XenVM, KVM, VMWare, and HyperV strings in the general vicinity. These are the CPUID leaf strings I was referring to. Change a few characters (overwrite, don't insert) for the platform you care about
6. Re-compress the file (e.g. makecab nvlddmkm.sys nvlddmkm.sy_)
7. Copy this over the old one in the driver package
8. Open a EWDK environment in the Display.Driver folder from step 2 and issue:
   inf2cat /os:10_X64 /v /driver:. signtool sign /n "Subject Name of Cert" /ac "CrossCertificate.cer" nv_disp.cat
9. Install your drivers

You can loosen some of the signing requirements (e.g. use of self-signed certs) by enabling Test Mode for the Windows 10 guest VM (i.e. bcdedit.exe -set testsigning on).

Hope that helps!

[RdGameboy]

@riverar
Sorry I spent about an hour trying to figure out how to do this on AMD drivers but was unable. I'm attempting to use an RX590, similar to an earlier poster who posted AMD issues who was using an RX580 successfully but you noted that their drivers were still checking for the VM.

I attempted to expand -R atikmdag.sys but kept getting an error that led me to believe that it was already expanded? So I opened up atikmdag.sys in XVI32 and was greeted to a mix of gobbledygook but upon further inspection I found Voltage information. Not what we're looking for but proof that it's uncompressed I hope? I tried searching for the terms XenVMMXenVMM, KVM, Hyper, and HYPER but none returned results. What should I try next? Thanks for the help!

EDIT:
Should've kept working! Found the appropriate string! I didn't realize that the search function had a next feature, first time using a hex editor, sorry! Anyway I found the string
CheckHypervisorType: Vendor ID = [%s]
Unfortunately the instructions are too vague for me and my ignorance of this topic :( How should I edit this string?

[LudicrousBloodyKill]

@RdGameboy Have you figured it out? I also happen to have the same problem and it's also with an RX 590.

[riverar]

@RdGameboy @LudicrousBloodyKill I just took a look. AMD checks for a machine manufacturer of “Microsoft Corporation” and model of “Virtual Machine”. Find “Virtual Machine” in XVI32 and change it to something else (e.g. “XXXXXXX Machine”).

Other platforms are in the same vicinity of the above (e.g. Xen, Qemu, XenVMM, KVMKVM, AliyunKVM, VMwareVMware).

[LordAoshi]

I go a Radeon WX3200 Pro and installed the drivers on the VM. On Windows server 2019 Hyper-V i updated the registry with the 2 strings mentioned above. After that I followed all the official microsoft installation guide for DDA Gpu and now it finally shows the gpu in the VM.
But there is still the Error 43... Although the driver from AMD was installed successfully ??
Any ideas?

Thanks ;)

[aburlak]

It worked for me with Windows Server 2019 v.1909 (January 2020 refresh) and Radeon VII, which was the only GPU.

I disabled the boot animations and the BasicDisplay.sys driver, so that the guest VM takes over the GPU just after exiting the text-mode BOOTMGR prompts.

bcdedit /set {current} bootux disabled
bcdedit /set {current} bootmenupolicy legacy
bcdedit /set {current} novga on
bcdedit /set {current} quietboot on
bcdedit /set {bootmgr} displaybootmenu yes
sc config BasicDisplay start= disabled

I didn't install any drivers on the host (except making sure that the legacy VGA driver isn't loaded).

In addition to passing through the Radeon VII, I passed through the NIC (Aquantia AQC107), the USB controller, and the SATA drive with preinstalled Windows 10 guest.
It was all working fine (after setting those 2 registry keys), however, as with all such Radeon Vega/Vega2-based setups, it was affected by infamous "reset bug".

After playing with this for about a day - I moved on to a KVM-based configuration, because I needed more flexibility (and the ability to patch the code to intercept device reset, etc).

[LordAoshi]

so there is still no solution even for a passthrough?? I thought we were living in 2020.... but it seems we are behind decades... its just a shame....

[mr250]

@RdGameboy Sure. Here's a rough guide for NVIDIA drivers:

1. Extract your drivers on disk.
2. Navigate to /path/to/extracted/drivers/Display.Driver and expand -R nvlddmkm.sy_ nvlddmkm.sys.
3. Open nvlddmkm.sys in a hex editor (e.g. XVI32)
4. Find XenVMMXenVMM
5. Observe XenVM, KVM, VMWare, and HyperV strings in the general vicinity. These are the CPUID leaf strings I was referring to. Change a few characters (overwrite, don't insert) for the platform you care about
6. Re-compress the file (e.g. makecab nvlddmkm.sys nvlddmkm.sy_)
7. Copy this over the old one in the driver package
8. Open a EWDK environment in the Display.Driver folder from step 2 and issue:
   inf2cat /os:10_X64 /v /driver:. signtool sign /n "Subject Name of Cert" /ac "CrossCertificate.cer" nv_disp.cat
9. Install your drivers

You can loosen some of the signing requirements (e.g. use of self-signed certs) by enabling Test Mode for the Windows 10 guest VM (i.e. bcdedit.exe -set testsigning on).

Hope that helps!

Mr Riverar ! Please help :-)

signtool sign /n "Subject Name of Cert" /ac "CrossCertificate.cer" nv_disp.cat

What do you mean "Subject Name of Cert" and "CrossCertificate.cer" ?
In the AMD Drivers its alsosomething like C0354308.cat and U0354308.cat

BR
Tomas

[mr250]

@riverar
Im having progress here, i am on the last signing step, but i dont get the certs you mention...
Br
Tomas

[mr250]

@riverar
I did
bcdedit.exe -set testsigning on
And then tried to install the driver.
But now im getting

This device is not working properly because Windows cannot load the drivers required for this device. (Code 31)

[riverar]

@mr250 @LudicrousBloodyKill and others. Let's take this out of the Windows Server docs repository. I'll purchase an AMD GPU today and provide NVIDIA/AMD guidance on my blog ASAP. (Due to COVID19, shipping is slow. GPU will arrive next week Sunday.)

[mr250]

Wonderful !!
This makes me so happy, i cant find anything on the old interwebz regarding this.
I would really like to help out as well.

I need a guide so i can reproduce this when updating drivers.
BR
Tomas

[jkommeren]

Just installed Windows Server 2019 semi-annual (1909) as a Hyper-V host, got the 'Element not found' error when trying to start a VM with GPU attached.

Setting two policy settings below has resolved the error for me (no reboot was required, either).

Registry key path (create it if not present): HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\HyperV

Registry values:
RequireSecureDeviceAssignment = 0 (REG_DWORD)
RequireSupportedDeviceAssignment = 0 (REG_DWORD)

It looks like this policy is yet another "whitelisting"/opt-in mechanism - vendors have to "opt in" for the DDA support for GPU virtualization.
Setting this policy allows the GPU to work even if not supported by the vendor in this configuration.

More information is here:
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/dispmprt/ns-dispmprt-_dxgkarg_getvirtualgpuprofile

Just wanted to say, this worked for me. Bought a 50euro Quadro K2000 just for fun, because my AMD card was too old. Would not budge ("Element not found").

Must have tried a zillion things (v1809).

Added these two keys to registry (HyperV folder was not there): VM Booted, showed quadro k2000 with Error 43. Reboot (of VM) and all is golden.

I owe you a beer!

[DanoshNL]

@riverar , Did you by any chance got to find a solution for the AMD drivers? We're trying like hell to get a WX4100 working but same as @LordAoshi we did everything right and still get a error 43.
Thanks!

[riverar]

@DanoshNL I posted this last month https://github.com/riverar/Remove-HypervisorChecks but AMD support is still pending. You may be in luck though, Microsoft recently added GPU-P support to Windows/WSL which could be used in Hyper-V to simplify this problem. Will look into it.