Vulkan-docs: Replace abstract "queue families" with "engines" that map to real hardware

[janekb04]

Originally posted here: https://github.com/KhronosGroup/Vulkan-Docs/issues/569#issuecomment-657527265_ . Now a new issue because I think it is big enough.

[janekb04]

I would suggest simply abandoning queue families altogether (they would still be available for backwards compatibility), for reasons stated previously:

1. They do not map to hardware at all (and even if they do, its just a choice of the implementation, that is not exposed by it at all)
2. They are not explicit and low level, but a driver managed scheduling system instead
3. They provide no information whatsoever about:
   a) how many queues should actually be created
   b) are queues within a family competing for resources
   c) are families independent from each other
4. Different drivers interpret what families and queues are differently and implement them differently

[janekb04]

I think that instead of exposing abstract "queue families", real hardware should be exposed. I would imagine it like this:

A new type of object should be added: a vkPhysicalDeviceEngine that represents a physical part of the hardware. (Alternatively it could be a vkDeviceEngine if we want to keep multiple logical devices per physical device, though the changes proposed here make it no longer needed to have more than one logical device)
Examples include: the main graphics and compute engine, (multiple) Copy/DMA engines, maybe Video Encoding/Decoding Engines, and anything else that the hardware actually, physically has.
vkEnumeratePhysicalDeviceEngines(physicalDevice, count, pEngines) would be used to query the amount and information about the diffrent engines.
Each physical component should be a separate engine. As such, if a device has two Copy/DMA engines, both should be represented by a separate vkPhysicalDeviceEngine object instead of them being a single object with a count of two.
I imagine that the vkPhysicalDeviceEngine would either contain or allow for querying the information about:

1. the type of the engine (what type of physical hardware it represents - compute cores, copy engines etc.)
2. the capabilities of the engine (what kinds of operations could be performed by it - ex. can a copy engine be used to copy data intra-device or inter host)
3. any other information relevant to hardware like number of cores, bandwidth, etc.

[janekb04]

vkQueues would be created from a vkPhysicalDeviceEngine instead of a queue family. In detail:

1. It would be guaranteed that creating a single queue from an engine can fully saturate it, use it to its full potential, just like a single thread can fully use a single CPU core if it is bound to it by setting its affinity.
2. The application could choose which parts of hardware it actually needs and use only those. For example this could decrease power consumption as unneeded hardware would be "dormant"
3. It would be possible to create multiple queues per engine. There would be no limitation on how many of them could be created or it would be very high). This would allow to use a given engine in "multitasking" mode, where the driver would schedule the work on the engine similarly to how a single-core CPU can have multiple processes that seem to run concurrently.
4. Queues would be created and destroyed dynamically (after physical and logical device creation) just like normal CPU threads are. After all they would be just an abstract concept used to schedule work.

[janekb04]

@haasn Originally posted this on the issue this was migrated from:

What I like about ^ is that it avoids altogether the need to fix VkQueues at device creation time, which can create hurdles and unnecessary API overhead when attempting to share a VkDevice across multiple components/libraries.

That being said, it does raise the obvious question of what will happen to the 'unified graphics+compute' queue family. Shouldn't e.g. a modern AMD GPU's topology look more like:

• 1x graphics engine
• 4x compute engine
• 2x dma engine

Because even the 'unified' graphics+compute family currently schedules compute operations onto the same four compute pipes, right?

Therefore your reasoning would dictate they be considered separate engines, with no 'unified' engine for both. This mandates the use of cross-queue synchronization.

[janekb04]

@haasn I think that because modern hardware uses the same cores for graphics and compute there would be no need to have a separate engine for them. Instead there could be a GRAPHICS+COMPUTE engine, that we could create multiple queues on. That being said, I think it would be beneficial to be able to say which subset of functionality we want to use when creating a queue.

[janekb04]

We could create a single GRAPHICS+COMPUTE queue from the GRAPHICS+COMPUTE engine and potentially multiple COMPUTE only queues (from the same engine) for async compute, as the driver could schedule different queues however it finds the most optimal (though it should respect priorities).

[krOoze]

Yea, but I comiserate with the original authors of the system. It is just not that simple.

You are little bit being prescriptive that any current and future GPUs must be organized around some "Engines". And you are being prescriptive what the classes of the engines should be. I mean, shoud there be a Raytracing engine? ALU engine (for NV Tesnsor Cores)?

There is also a risk of making things so explicit it actually adds overhead. E.g. if every queue is distinct, it might need nominally coarser-grained synchronization between them than would otherwisely be needed, and as well perhaps splitting command buffers to smaller chunks which would add overhead, and as it currently works each queue needs separate submission (which could add submission overhead).

The GPUs do have engines for organizational purposes, but they are perhaps not as clear cut and discreet as you make them to be. xcoding engines perhaps use the services of compute engines. Graphics engines definitely use compute engines for fragment shading. Graphics engine might use copy engine. And everything possibly contests the same memory resource, as well as likely having some non-trivial limits that cannot be simply abstracted and exposed. As an app to get this right, you need perfect information to act on it, but if you cannot get it, then it is better to not have this explicitness at all.

Anyway:

ad 1: As I said above. Kinda hard goal due (often non-trivial) interdependencies to guarantee an engine can be saturated. Even in normal case one compute unit can disable another one due to exceeding some limit. If the engines were explicitly exposed as separate, it would be too surprising that one "engine" can starve another one.

ad 2: Might be bit naive to do it this way due to powerstates of the device. Perhaps one should not control the energy consumption by limiting the number of engines it uses. For one, you cannot know which other processes contests the resources to evealuate how many you need. Secondly, you might try use less engines, but it might push the whole device to some Boost powerstate, and in the end you might waste more despite your best intentions.

ad 3: Probably introduces internal synchronization to the driver, which is antithesis to Vulkan conventions. I do not necessarily see the advantage over a single queue and the app explicitly arbitrating its use across whatever user-space threads it may have. Having infinite queues you perhaps need mandatory priority system and whatnot other fluff in the driver itself.

ad 4: Possibly tricky, because synchronization primitives would probably work on the "real" queue, not your abstract queues. E.g. if you signal a semaphore it might stop all work, not just the work in the specific logical queue. Might be tricky to keep the GPU "well-fed" using this more flexible system. You cannot much "schedule work" using the system, quite the opposite, you might have less control over how things actually get scheduled. E.g. if you have four of your logical queues, and submit stuff to each, then you have actually no idea what happens with the workload.

[NicolBolas]

It would be possible to create multiple queues per engine. There would be no limitation on how many of them could be created or it would be very high). This would allow to use a given engine in "multitasking" mode, where the driver would schedule the work on the engine similarly to how a single-core CPU can have multiple processes that seem to run concurrently.

I don't much like this. If hardware does not directly support "multitasking mode", this means that the implementation has to support it for you. Which means that different queue submit operations have to be internally synchronized by the driver. If your goal was to make the API more "explicit", then this suggestion fails at that goal. Especially if there's no way for a user to tell whether that's going on.

Overall, I think it would be better to leave the Vulkan abstraction as is and instead provide a way to ask an implementation important questions about how its queue families, queues, and hardware constructs interrelate. Or at the very least, if there's going to be a change, we need to start with a full and complete understanding of what the user needs to know about the hardware and its relationship with submission operations.

[critsec]

(Speaking for myself, not on behalf of the WG.) An important goal of Vulkan is to give apps a reasonable way to write code that is portable across implementations, including ones that didn't exist when the app was written. The more possible configurations they have to deal with, and the less they're able to test on representative configurations, the harder that is. That's worth keeping in mind.

It would help to understand what real-world problems are being caused by the current system (see the XY Problem). Without understanding the problem, it's hard to evaluate possible solutions. The original post has a list of things the author doesn't like about the current system; 3(a-c) come closest to describing concrete, objective issues, but it would still help if those could be connected to real impact: what things does someone want to do that we can't be done, or loses meaningful efficiency, because of this?

Re. points 2 and 4: The driver is part of the implementation; there's many cases where the driver design is tightly intertwined with the hardware and/or operating system design, and is not as arbitrary or changeable as you might think. "Different drivers interpret what families and queues are differently and implement them differently" is an inherent result of the fact that they're having to map these to significantly different underlying hardware architectures and capabilities, and the abstraction we have is a result of that. When you have to build something that bridges systems where nearly all scheduling is done in the driver, systems where nearly all of it is done in the hardware (with a rigidly fixed set of scheduling concepts and behaviors), and several points on the spectrum between, it ends up not being very useful to try to distinguish what things are hardware and which are software because for practical purposes the answer is that it's both.

[krOoze]

@critsec TBF, we largely went over all the problems of the current system in the #569 Issue.

The original system kinda lacks use cases. I.e. in what situation to use what queue family and when to use a separate queue. Only straightforward is the fused queue family and using only one queue. But even there a guarantee is not given it can actually saturate the whole device. And it is non-obvious the Copy capability of that queue might be different from the DMA Copy capability of a dedicated queue family. Basically the queue families feel so abstract that no information can be assumed about them. And so, no reasoning about what is the best way (or what is the bad way) to use them is possible.

[NicolBolas]

@krOoze "TBF, we largely went over all the problems of the current system in the #569 Issue."

While true, I think it would be useful to lay down the entire problem set in a single, clear post, since that thread has information all over the place. Especially if the issues can be stated in a problem-focused way. Something like a series of statements similar to the following: "my workload is X, I've seen GPUs that have queue configurations Y and Z, and it's really unclear how to efficiently load X into both Y and Z."

I think the most important thing that should be presented is under what circumstances does the obvious interpretation of queues and queue families not achieve optimal or near-optimal performance?

[TomOlson]

@janekb04's "engines" proposal is kind of similar to some ideas that floated around early in the design of Vulkan 1.0. There were a lot of console programmers in the room, and they really just wanted a (very slightly abstracted) block diagram and register map. Previous comments have pointed out some of the problems with this. For various reasons, there end up being a lot of quirky interactions between nominally independent engines, and the problem of mapping a computation onto them quickly goes from "hard" to "intractable". That's not to say that there is no hope at all - some work is being done in this area - but it's very R&D and not something we can put into the API any time soon. So, we are stuck with the current abstraction for at least the next couple of years.

That takes us back to the larger discussion here and on #569. We definitely agree that the queue abstraction is a bit high-level and loosely specified. We also know that we don't provide much guidance for how to use queues and queue families. What we don't know is, how much of a problem is this causing? As @critsec said, it isn't clear to us what use cases people are having trouble with, or what kinds of performance impacts people are seeing. Our intent is that for "normal" use cases, you shouldn't be spending much time tuning your use of queues - do the fairly-obvious thing and you should get decent results. If that's not the case, we'd like to know about it.

So, +1 to @NicolBolas's suggestion in the preceding comment - a crisp statement of the problems people are having would be very useful to us, especially if you can quantify cost in development flexibility or perf. Might want to do that on a new issue since #569 is getting kind of long. I suggest closing this issue since we can't really used the "engines" proposal for the forseeable future. We do appreciate the effort and the discussion.

[janekb04]

@TomOlson I agree that concrete examples would be best. I will close this issue now and open a new one when I will have gathered enough information to be able to compose a list of specific cases.