Vulkan-docs: Extension request: preventing or acknowledging potential descriptor relocations at draw-time

[procedural]

Vulkan is all about pushing work off the clock, but, apparently, due to incomplete API design, it may cause descriptor relocations at draw-time:

https://github.com/gpuweb/gpuweb/issues/19#issuecomment-316512163

@ RafaelCintron oh, sorry about being unclear.
I'm more concerned about the fact that only a single ID3D12DescriptorHeap of each type can be actively bound at any time on the command list. This prevents scenarios (supported by Vulkan/Metal) where different rendering sub-systems (e.g. particle system, texture loader, etc) have their own descriptor pools, from which they allocate descriptors meant to be used together in a single draw call.

@ kvark , D3D devs tell me this is due to hardware limitations.

On at least on IHV, you can only bind one descriptor heap at a time with size maximum of 1 million. If an API wants to pretend that multiple can be bound at a time, that means that this IHV would have to potentially relocate descriptors from multiple application heaps into the one hardware heap at draw-time in the worst case (e.g. they didn’t get lucky and the app’s heaps happened to all fit into one hardware heap).

An app can certainly make a large descriptor heap in DX12 (a million entries is the largest size guaranteed to work assuming memory is available), and then dedicate subregions of that heap to individual “pools” that the application separately suballocates out of.

The problem is, say you have pool A, B and C each 500k descriptors. Only 2 will fit in a max 1 million entry heap. If you use A and B together, works fine. Then if you want to use B and C together you have to switch to another heap and likely copy all of B from the AB heap to a new BC heap. Assuming you use much smaller “pools” relative to descriptor heap size, and most pools fit in one heap, the number of times you’d have to switch heaps and shuffle descriptors around may not be huge. But D3D12 lets you see the cost of doing so.

Won't name names, but it's one of the big three IHVs, and what probably happens in their driver is this:

(                       RealHardwareDescriptorHeap                     )
------------------------------------------------------------------------
[         VkDescriptorPool        ]  [         VkDescriptorPool        ]
-----------------------------------  -----------------------------------
[VkDescriptorSet] [VkDescriptorSet]  [VkDescriptorSet] [VkDescriptorSet]

Currently, Vulkan doesn't know anything about the ( RealHardwareDescriptorHeap ), but D3D12 knows about it, and the advantage of D3D12 over Vulkan here is that it has an explicit command called ID3D12GraphicsCommandList::SetDescriptorHeaps that can do all the necessary work explicitly, meaning you can see the cost of switching descriptor heaps on that IHV's hardware and overlap some other work with it or minimize the number of heap switches, while Vulkan keeps fingers crossed and basically says "sure, bind descriptor sets from different pools however you want", making the cost implicit. This is OpenGL slots all over again, but dressed in structs this time around.

What's interesting is that IHV keeps silence for 4 years since the Vulkan 1.0 release, no extension was proposed to address this, or even mentioned anywhere. If this is not a problem, I will happily close this issue, but if it is, speak now and don't wait for the next API to say "Vulkan was good and all, but it hid the cost of certain operations... So welcome the new API called Kraken! It fixes all the problems of Vulkan, we promise!"

[Wunkolo]

Thankfully VkDescriptorPoolCreateInfo has a pNext inside of it already, avoiding the whole VkDescriptorPoolCreateInfo2 + vkCreateDescriptorPool2 pattern that other API-movements have had to suffer from.
I think Vulkan would benefit from having an extension that lets you know RealHardwareDescriptorHeap's properties and lets you designate the HardwareDescriptorHeap that the descriptor pool sits on top of, following the whole "tell me everything you need about this thing, up-front" pattern that Vulkan has and putting the task of optimal/minimum descriptorheap-state-changing to the user.

[procedural]

@Wunkolo right.

Since Vulkan doesn't have any sort of residency management for device memory or, in this case, descriptor heap memory, even two small things would help a lot:

1. Querying available memory size of the RealHardwareDescriptorHeap, like in VK_EXT_memory_budget but for the descriptor heap.
2. Control descriptor pools overallocation behavior explicitly, like in VK_AMD_memory_overallocation_behavior but for the descriptor heap. Currently, you can only hope that vkCreateDescriptorPool will return VK_ERROR_OUT_OF_DEVICE_MEMORY if you'll exceed the available memory of the RealHardwareDescriptorHeap, but you have no guarantees of that and descriptors may be relocated at draw-time behind your back.

[NicolBolas]

@procedural

Is this "RealHardwareDescriptorHeap" a truly global resource, like device memory? That is, do all application using the GPU pull from the same resource, with descriptors living permanently within this storage?

I'm not a D3D12 expert, but it seems to me from skimming the docs that ID3D12DescriptorHeap is not a global resource. Indeed, the Microsoft docs on the subject say something very suspicious:

All heaps are visible to the CPU. The application can also request which CPU access properties a descriptor heap should have (if any) – write combined, write back, and so on. Apps can create as many descriptor heaps as desired with whatever properties are desired. Apps always have the option to create descriptor heaps that are purely for staging purposes that are unconstrained in size, and copying to descriptor heaps that are used for rendering as necessary.

Emphasis added.

It seems to me that descriptor heaps are typically applied at the beginning of a command list's recording (or a very few times thereafter) through a call to CommandList::SetDescriptorHeaps. By the nature of this API, you are permitted to have multiple descriptor heaps with their own sizes used by different command lists at different times.

This suggests that, for hardware where "RealHardwareDescriptorHeap" is a thing, it doesn't seem to be a global resource. It seems to be an ephemeral resource, populated to execute a sequence of commands, and then populated to execute a different sequence of commands. The population is not a cheap operation, but it's also an operation that will be paid every time you execute that command list.

So in D3D12, it appears that you can have one ID3D12DescriptorHeap that has 1 million elements, and another one that has 1 million elements, and use them both of them. Just not at the same time, and not without cost. The cost will be paid when you call SetDescriptorHeaps. In Vulkan, it seems that the cost still happens, but it happens when you call vkCmdBindDescriptorSets, and is only paid when certain conditions are met.

As such, the simplest way to employ something similar in Vulkan is to just have vkBeginCommandBuffer be able to take a struct containing one or more descriptor pools. If you provide such things, then make it UB if you make a vkCmdBindDescriptorSets call that uses descriptors from some other pool. This makes it a one-time cost, always paid at the start of the CB, rather than hypothetically being paid at any vkCmdBindDescriptorSets call. Naturally, this would also make vkBeginCommandBuffer fail if too many descriptors are in the given pools.

Alternatively, you can have a SetDescriptorHeaps-like call, a vkCmdMakeDescriptorPoolsResident or something. This would make those descriptors available, and come with the attendant promise that no subsequent commands will use non-resident descriptors.

If you want to be even more D3D12-like, you can take a bunch of descriptor pools and associate them into some kind of aggregate pool object, so that you can ensure that they don't violate any size limitations.

My overall point is that the suggestions here seem wrong-headed. They seem to treat the limitation as equivalent to a global memory resource, rather than the more ephemeral resource restriction that D3D12 treats it as.

[procedural]

Is this "RealHardwareDescriptorHeap" a truly global resource, like device memory? That is, do all application using the GPU pull from the same resource, with descriptors living permanently within this storage?

Yes.

[procedural]

With exception of certain types of descriptors that may be stored somewhere else in some other form, like samplers.

[NicolBolas]

Can you provide some links that go into detail on the nature of this resource? The WebGPU conversation you linked to only talks about things in terms of the D3D12 feature (which represents the resource as ephemeral. You can have multiple heaps that in aggregate violate the limit; you're just restricted from using them at the same time).

[krOoze]

@NicolBolas Well in the thread Redmond engineer said it is HW limit, and claimed he consulted it with their HW people. Still bit in the stage of internet hearsay. We will see what varied non-MS engineers will say here...

Still 1M sounds bit like infinity to me. TBH if you try to sample anywhere near 1M resources in a shader I would expect LOST_DEVICE anyway. Being realistic, I would expect more hidden limits of this kind; every HW has some awkward quirks. And driver SW can add some weird limits of their own. I wish every such case could be neatly covered, but that is probably unrealistic; so I settle if it at least fails gracefully.

[jekstrand]

This extension request seems to make the assumption that RealHardwareDescriptorHeap is some sort of hardware concept that is fairly universal. The reality is descriptor heaps, as D3D12 uses them, are an API concept and not necessarily something in hardware. Vulkan chooses to use a different model which is potentially more flexible for implementations. Descriptor pools in Vulkan exist purely to help the driver manage memory more performantly by letting them use lock-free memory allocators within the pool rather than allocating directly from a central pool which might require a lock. They don't actually imply anything that must be bound to the pipeline at any particular time. Pushing towards the D3D12 model, while it would bring the APIs closer together, is just that: a change in the driver model.

Yes, there are some hardware limits around descriptor heaps but those are limits which are implied by some detail of the hardware and their existence does not mean that all hardware has descriptor heaps.

[jekstrand]

What's interesting is that IHV keeps silence for 4 years since the Vulkan 1.0 release, no extension was proposed to address this, or even mentioned anywhere. If this is not a problem, I will happily close this issue, but if it is, speak now and don't wait for the next API to say "Vulkan was good and all, but it hid the cost of certain operations... So welcome the new API called Kraken! It fixes all the problems of Vulkan, we promise!"

We talked about this a bit on today's working group call. The consensus is that no one has talked about fixing this because no IHVs think that it's actually a problem. We aren't seeing any real performance issues that making a change like this to our descriptor model would solve.

[NicolBolas]

@krOoze

Well in the thread Redmond engineer said it is HW limit, and claimed he consulted it with their HW people.

Yes, but "it" in this context is not clear. Or rather, the meaning of "it" is disputed.

Procedural is saying that all ID3D12DescriptorHeap objects source from a global resource. That is, throughout an entire application, the sum total of descriptors used by all ID3D12DescriptorHeap objects cannot exceed some fixed maximum. Each ID3D12DescriptorHeap object represents some subsection of this global resource, and for optimal performance, the global resource should not be altered during a frame's processing.

I am saying that this is not what the Direct3D 12 API is actually doing. From my read of Microsoft's documentation, it appears that you can have as many ID3D12DescriptorHeap objects as you want, and you can swap between them however much you like. However, each individual ID3D12DescriptorHeap object can only have so many descriptors, and that limit is the hardware limitation being discussed.

That is, imagine that there's a giant table of descriptors living in the GPU. Each time you call CommandList::SetDescriptorHeaps, the system copies the descriptors from your ID3D12DescriptorHeap into this table of descriptors, and this copying must happen before you can use any descriptors. This also means that any one ID3D12DescriptorHeap cannot exceed the size of this table of descriptors.

But this means that the GPU table of descriptors is not where ID3D12DescriptorHeap gets its storage from. The GPU table's descriptor list is ephemeral, being uploaded to at various points during the rendering process. It isn't fixed and static; each SetDescriptorHeaps call potentially represents copying into this table and invalidating any data previously there.

The limitation is about how many descriptors can be in use at one time, not how many can exist anywhere throughout your application.

[procedural]

We talked about this a bit on today's working group call. The consensus is that no one has talked about fixing this because no IHVs think that it's actually a problem. We aren't seeing any real performance issues that making a change like this to our descriptor model would solve.

Perfect, thank you, @jekstrand! :) As promised, I'm happily closing this issue then, since I was worried that it could have serious implications for some hardware.