Vulkan-docs: New extension for linking fences with OS events/file handles

[ratchetfreak]

Currently the only way to wait on the device for a Fence is vkWaitForFences.

However all of this will block the current thread and prevent other OS events (like IO a disk read finishing up, a key press from the user, ...) being handled.

To let other OS events get through to the thread you would need to associate the fence with a Kernel object that can be waited on in bulk (like WairForMultipleObjects or select/epoll).

This would for example be with a vkLinkFenceToWin32EventHandleEXT(vkDevice, vkFence, HANDLE); where if the fence gets signaled the associated event also gets signaled. (only 1 event per fence and vice versa)

Or instead the notification could be done with a IO completion queue:

struct vkLinkFenceToWin32CompletionQueueInfoEXT{
    VkStructureType  sType;
    void* pNext;
    HANDLE completionQueue;
    ULONG_PTR completionKey; 
    OVERLAPPED *overlapped; 
}

vkLinkFenceToWin32CompletionQueueEXT(vkDevice, vkFence, vkLinkFenceToWin32CompletionQueueInfoEXT*);
//or as part of the pNext in fence create info

The completionKey and OVERLAPPED pointer are passed back to GetQueuedCompletionStatus after the fence is signaled. The OVERLAPPED struct is not copied (to mimic the rest of the win32 api concerning the struct which allows the user to put data after the struct).

I'm not familiar enough with linux programming to judge what the best method for linking fences to file handles is; eventfd(2) per fence or a pipe(7) acting as a queue? I'll leave that up to others to judge.

[ratchetfreak]

With the external_semaphore_* extensions released recently this issue can be resolved by allowing the application to wait on the fd/Handle. Perhaps by adding a flag to let the application signal that it wants to do that.

[cubanismo]

Methods to do this with fences are under discussion. However, waiting on semaphores from the CPU is not supported. They are intended for GPU-side waits.

[zimond]

1.0.54.0 adds vkGetFenceFdKHR, I guess it's the way to go?

[ratchetfreak]

@daiheitan looks like however it's still not documented allow use with WaitForMultipleObjects or select/epoll

[zimond]

@ratchetfreak Yeah... Seems the latest Nvidia driver has this extension, but with no select/epoll support. I'm still testing the API.

[Ralith]

Any news on this? Would be very convenient.

[expipiplus1]

VK_KHR_external_fence isn't really intended for this use case. As far as I understand it, it's intended to be used for using fences across multiple instances/devices.

Having said that, it would be very convenient to extend VkExternalFenceHandleTypeFlagBits with an entry requiring support for epoll to facilitate application usage of these fds.

~(perhaps best as a separate issue: does anyone know what a "Linux sync file" is (as mentioned in the docs for VkExternalFenceHandleTypeFlagBits)?). It's https://www.kernel.org/doc/html/v5.4/driver-api/sync_file.html~

[expipiplus1]

Was this ever resolved inside Khronos? It would be handy to use the OS's waiting primitives instead of Vulkan's.

I noticed recently that @critsec suggests here (https://stackoverflow.com/questions/42721870/means-for-calling-back-from-vulkan-to-host-application) that this is possible with the FENCE_FD handle type did you mean SYNC_FD?

Methods to do this with fences are under discussion. However, waiting on semaphores from the CPU is not supported. They are intended for GPU-side waits.

I guess timeline semaphores would be a slight complexity.

[critsec]

Getting a SYNC_FD handle type using vkGetFenceFdKHR on Linux/Android will give you a fd that is compatible with epoll. This is definitely a use case that VK_KHR_external_fence is intended for.

I don't think there's an equivalent for Windows yet; the OPAQUE_WIN32 and OPAQUE_WIN32_KMT handle types aren't compatible with WaitForMultipleObjects.

A fallback is to use a separate utility thread that waits using vkWaitForFences and then signals something that is compatible with what you want to do. This is clearly less than ideal (extra thread scheduling delays, etc.) but works well enough in some cases.

[cubanismo]

Yeah, the FDs you pull out of fences on the NV Linux driver aren't going to do much if you poll() them. If we did have an API to get the right FD, because of the way our HW/driver manages fences (potentially other drivers/HW as well, not sure), the usage would have to be something like:

VkFence fence = createAndSignalSomeFence();
vkGetFenceFdKHR(..., fence, ... handleType = VK_EXTERNAL_FENCE_HANDLE_TYPE_POLL_FD ..., &fd);

while (true) {
   poll(..., fd, ...);
   // Might wake up spuriously, beyond the usual EINTR reasons. Check actual fence status:
   if (vkGetFenceStatus(dev, fence)) break;
}
// Fence is done.

And similar for timeline semaphores. Another place this gets complicated is the interactions with other external handle types and by implication, WSI. E.g., if you try to do this with a fence you pass to vkAcquireNextImageKHR(), it may not work as you expect by the time we fold in the "payload" concept.

That said, if there are demonstrated use cases where this would be better than the alternatives (E.g., spin up a waiter thread), we could pursue an extension, but I haven't heard much demand thus far, so I haven't pursued it. Most people seem OK with dedicating a thread to it, or at least happy enough that they haven't made it their highest priority Vulkan spec issue. If you need it on windows, things are a little more complicated too, but it's probably doable with win32 event objects there.

[expipiplus1]

Getting a SYNC_FD handle type using vkGetFenceFdKHR on Linux/Android will give you a fd that is compatible with epoll. This is definitely a use case that VK_KHR_external_fence is intended for.

At the least, would it be worth enshrining this in the spec, perhaps under a physical device property?

Another place this gets complicated is the interactions with other external handle types and by implication, WSI. ...

Thinking about this a little more, perhaps a more general, platform agnostic solution would be to have callbacks executed by Vulkan when waiting is done, something like

void myCallback(void* myData, uint32_t semaphoreCount, VkSemaphore* semaphores) {
  // touch my fd or whatever
}

void myProgram() {
  VkWaitSemaphoreCallbackInfo waitCallbackInfo;
  waitCallbackInfo.sType = VK_STRUCTURE_TYPE_WAIT_SEMAPHORE_CALLBACK_INFO;
  waitCallbackInfo.pNext = NULL;
  waitCallbackInfo.callback = myCallback;
  waitCallbackInfo.data = NULL;

  VkSemaphoreWaitInfo waitInfo;
  waitInfo.sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO;
  waitInfo.pNext = &waitCallbackInfo;
  waitInfo.semaphoreCount = 1;
  waitInfo.pSemaphores = &mySem;
  waitInfo.pValues = &myValue;
  const VkResult r = vkWaitSemaphores(dev, waitInfo, 0);

  switch(r) {
    case VK_SUCCESS:
      // myCallback has been called
    case VK_TIMEOUT:
      // myCallback may not have been called yet
  }
}

It may not be appropriate to reuse vkWaitSemaphores for this, as it's now quite a distinct operation (building up state in the driver). On the other hand VkSemaphoreWaitInfo is precisely the type which should go with VkWaitSemaphoreCallbackInfo.

This would also neatly push the handling of spurious wakeups back into the driver which already has to deal with it.

That said, if there are demonstrated use cases where this would be better than the alternatives (E.g., spin up a waiter thread), we could pursue an extension, but I haven't heard much demand thus far, so I haven't pursued it. Most people seem OK with dedicating a thread to it, or at least happy enough that they haven't made it their highest priority Vulkan spec issue.

The desire here, https://github.com/vulkano-rs/vulkano/issues/666, is to integrate Vulkan more cleanly with various Rust job-systems/io-managers (although there's nothing Rust-specific about this).

My use case would be to use Vulkan with concurrent Haskell without having my threads robbed having to service waits, the GHC event manager is good at waiting on FDs however.

I suppose that in general this would enable Vulkan to be integrated into event systems without them having to handle anything Vulkan-specific.

Certainly these can all be done with waiter threads but like @critsec says, this is clearly less than ideal.

If you need it on windows, things are a little more complicated too, but it's probably doable with win32 event objects there.

Not the driver's concern any more using the callback approach :wink:

[Ralith]

A callback invoked while a thread is blocked isn't any better than the status quo; you could just wait then call the thread. The appeal of epoll/whatever compatibility is that it allows you to compose waiting for a vulkan resource with other asynchronous operations within a single thread.

Personally I've addressed my needs here by just polling timeline semaphores in various patterns. It's not the most elegant solution, but conversely, it's much more portable than a platform-specific multiplexing scheme.

[expipiplus1]

A callback invoked while a thread is blocked isn't any better than the status quo; you could just wait then call the thread. The appeal of epoll/whatever compatibility is that it allows you to compose waiting for a vulkan resource with other asynchronous operations within a single thread.

One can build this epoll compatibility on top of callbacks though:

myFd = eventfd(...)
vkWaitSemaphores(... callback which writes to myFd)
epoll_wait( read myFd and any other async operations )

At some point the driver's going to have to call write or SetEvent, is it any matter if this is done in an application-supplied callback? (I think the only case where this would matter would be if the driver does this from some privileged context)

[Ralith]

If your vkWaitSemaphores call doesn't block, that could work, but I'm not sure how an ICD could implement that, except by spawning a background thread on its end, which seems a bit silly.

[expipiplus1]

Some drivers do have userspace threads kicking around, but in most cases you're probably right. Certainly with epoll/WaitEvents support would allow one to implement this themselves anyway.

For the case of timeline semaphores, eventfd objects keep a 64bit value inside them, so that could be used (although the application would wake on every change and have to check themselves).

[cubanismo]

Getting a SYNC_FD handle type using vkGetFenceFdKHR on Linux/Android will give you a fd that is compatible with epoll. This is definitely a use case that VK_KHR_external_fence is intended for.

At the least, would it be worth enshrining this in the spec, perhaps under a physical device property?

I believe this is already implied. The spec specifies the properties of FDs returned for SYNC_FD (Basically states it's compatible with either the Android fence FD or Linux sync FD kernel primitives for a given system, IIRC), and I believe Linux itself defines the operations possible on a sync FD.

Thinking about this a little more, perhaps a more general, platform agnostic solution would be to have callbacks executed by Vulkan when waiting is done

That's just going to mean the driver spinning up that extra thread for you opaquely, without the ability for you to control its affinity, priority, etc. No value-add for doing this in the driver, so not a good candidate for Vulkan functionality. Just spin your own thread as suggested above, use that to make a callback to the rest of your code, signal an event FD, etc.

[oddhack]

@ratchetfreak could this be closed, based on the most recent comment from @cubanismo ?