Vulkan-docs: Request clarification for interaction between semaphore wait and barrier

[chrisvarns]

I'm trying to understand exactly how the initialLayout transition in a renderpass is sychronised with the WSI semaphore. I encountered the _Swapchain Image Acquire and Present_ section in https://github.com/KhronosGroup/Vulkan-Docs/wiki/Synchronization-Examples which seems to be based on a note in the spec (extended KHR version) section 6.4.2, about using semaphore waits to guarantee that the WSI semaphore wait operation happens-before the layout transition. I am trying to understand how it sets up a dependency such that the transition happens-after the semaphore wait operation, as it isn't immediately obvious to me.

The note says to set both the srcStageMask and dstStageMask to VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT, which it says establishes an execution dependency between the wait and the barrier:

the semaphore signals after the presentation operation completes, the semaphore wait stalls the VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT stage, and there is a dependency from that same stage to itself with the layout transition performed in between.

The "dependency from that same stage to itself" is the main source of confusion for me, as the rest of the spec talks in terms of synchronization scopes, and it's not immediately clear from the note what exactly is in the first sync scope of the barrier such that it actually happens-after the wait. The strong implication is that the wait operation itself is in the first sync scope, which would imply that the wait operation is defined to execute in the stage specified in the pWaitDstStageMask parameter of vkSubmitInfo, and therefore gets included in the first sync scope of the barrier when that stage is specified as the srcStageMask, but elsewhere in the spec it says it doesn't execute in any defined pipeline stage.

I am struggling to find the language in the spec that concretely clears this up for me, maybe someone can point me in the right direction?

Thanks in advance

[krOoze]

Ah, synchronization...

Anyway, so our situation is we called vkAcquire with semaphore Sa, and we want to render something to that image (with a render pass). Then we want to vkPresent that image. And you are interested in spec quoted justification for the conventional way to do this (as you should). Right?

Firstly the spec already offers informational note\clarifications on how to do this. The first is the Note in 6.4.2, second quoted below in point nr. 7, and third is:

Note

When the presentable image will be accessed by some stage _S_, the recommended idiom for ensuring correct synchronization is:

• The VkSubmitInfo used to submit the image layout transition for execution includes vkAcquireNextImageKHR::semaphore in its pWaitSemaphores member, with the corresponding element of pWaitDstStageMask including _S_.

• The synchronization command that performs any necessary image layout transition includes _S_ in both the srcStageMask and dstStageMask.

Or as I loosely describe the scheme in my mini-tutorial. So, that's the informal stuff that strongly advises us what to do. Now, what is happening, per the spec, is:

1. We do the vkAcquire with Sa, which then have a signal pending on it. When it gets signaled it means the PE is done with the Image and won't access it anymore, so we are allowed to write into it. Hopefuly nothing surprising, and no spec quote necessary.

2. We vkQueueSubmit our draw command buffer. We provide the signal-pending Sa in waitSemaphoreCount, and we provide pWaitDstStageMask which would be the first stage that writes into the Image (let it be VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT). So, the Semaphore is waited on no later than right before that stage begins. What a Semaphore wait does is described thoroughly:

   When a batch is submitted to a queue via a _queue submission_, and it includes semaphores to be waited on, it defines a memory dependency between prior semaphore signal operations and the batch, and defines _semaphore unsignal operations_ which set the semaphores to the unsignaled state.

   the second _synchronization scope_ is limited to operations on the pipeline stages determined by the destination stage mask specified by the corresponding element of pWaitDstStageMask.

   The second _access scope_ includes all memory access performed by the device.

   Batch is practically the thing you submit for execution with VkSubmitInfo. I.e. the array of cmdbuffers that have associated array of wait Semaphores, and array of signal Semaphores.

   What a _synchronization scope_ and _access scope_ is is described in the 6.1. Execution and Memory Dependencies. I strongly suggest you read it so many times that nothing else exists in your mind. :p

3. The cmdbuffer had only like vkBeginRenderPassInstance, vkDraw, vkEndRPI. The renderpass (subpass external dependencies) is where all our pipeline bariers are. We defined subpass dependency from VK_SUBPASS_EXTERNAL to our only subpass. Let that dependency have srcStageMask=dstStageMask=VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT. We also provided VkAttachmentDescription::initialLayout and VkAttachmentReference::layout.

   It is practically a normal Pipeline Barrier as any other (and I am not gonna quote how that works). Interesting part is we actually made an _execution dependency chain_ with the pWaitDstStageMask == srcStageMask. Interesting part is the motivation why we did so (the Semaphore already makes a dependency, so why??):

   Automatic layout transitions away from initialLayout happens-after the availability operations for all dependencies with a srcSubpass equal to VK_SUBPASS_EXTERNAL, where dstSubpass uses the attachment that will be transitioned.

   So what we have actually described is a dependency from Sa signal to the layout transition, and then from the transition to our draw. So now have an image in LAYOUT_COLOR_ATTACHMENT, and has an _execution dependency_ and is also _made visible_ (i.e. memory dependent) on our vkDraw. OR not so fast:

4. In the renderpass we defined LOAD_CLEAR (we probably want to clear the image before we draw new stuff in it), which works this way:

   The load operation for each sample in an attachment happens-before any recorded command which accesses the sample in the first subpass where the attachment is used. [...] Load operations for attachments with a color format execute in the VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT pipeline stage.

   So, actually we made dependency to the Load Operation. And subsequently the Load itself makes a dependency from itself to "any recorded command which accesses the sample" (i.e. our draw).

5. The vkDraw is executed, and the rest is pretty similar to the above, just in reverse order:

6. We defined STORE_STORE op on the Image (to preserve our drawing (reverse of 4., exact spec quote is near the description of Load Op).

7. (Reverse to 3.), we also defined VkAttachmentDescription::finalLayout, and an "to external world dependency". I.e. from our only subpas to VK_SUBPASS_EXTERNAL, with likely srcStageMask=COLOR_ATTACHMENT_OUTPUT and dstStageFlag=VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT. The subsequent Semaphore Sd will chain on BOTTOM (a Semaphore signal catches anything), AND it won't stall any other work we might be doing in the cmdbuffer, as the spec advises us:

   Note

   When transitioning the image to VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR or VK_IMAGE_LAYOUT_PRESENT_SRC_KHR, there is no need to delay subsequent processing, or perform any visibility operations (as vkQueuePresentKHR performs automatic visibility operations). To achieve this, the dstAccessMask member of the VkImageMemoryBarrier should be set to 0, and the dstStageMask parameter should be set to VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT.

8. In vkSubmit we provided a semaphore Sd which will signal when the batch (our cmdbuffer) is completed.

9. We provide the signal-pending Sd Semaphore to vkPresent, which will wait on it before trying to read the Swapchain Image.

So in summary, we (and the API behavior) made a valid dependency:
last PE read → semaphore signal (vkAcquire) → semaphore wait (vkSubmit) → layout transition (via external subpass dependencies) → render pass Load OP → vkDraw → render pass Store Op → layout transition → semaphore signal(vkSubmit) → semaphore wait (vkPresent) → first PE read.

Hope that makes some things clear (rather than the opposite :D), and points to the documentation of it in the spec where you needed. I think nr. 3 and nr. 7 answer your specific question.

[chrisvarns]

Thanks for your help, you have confirmed my understanding to me :) so I am 100% happy my understanding of the practical side of "how do i synchronize this layout transition" is correct.

I strongly suggest you read it so many times that nothing else exists in your mind. :p

Indeed this was my homework over the weekend, I have read chapters 6 and 7 pretty exhaustively now 👍

There is an outstanding academic question though, or maybe I've fallen into a hole in the spec, I'm not sure, of how exactly the semaphore wait operation is included in the first sync scope of the subpass dependency. I understand that:

• pWaitDstStageMask = VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT includes all operations in that stage in the second sync scope of the wait operation, i.e. the wait happens-before all operations in that stage
• srcStageMask = VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT limits the first sync scope of the dependency to prior operations that occured in that stage
• semaphore wait operations themselves do not execute in a defined pipeline stage

There appears to be a gap in the spec, or I have just not seen the relevant language yet, as we are having to make a bit of a jump to then say that:

• srcStageMask = VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT includes the wait in the first sync scope of the subpass dependency, and therefore the layout-transition happens-after the semaphore wait operation

which appears to be by magic because the wait operation is not specced to execute in that stage. I'm happy that this is the correct thing to do, but I am struggling to find the supporting language in the spec

As an aside, some of the related language in the notes only occurs in the KHR version of the spec which caught me out initially :)

Thanks for your help!

[chrisvarns]

It's almost as if including a stage in a sync scope adds the _stage itself_, not just the operations within it.

[krOoze]

We need not add any Stage.
We are defining an _Execution Dependency Chain_ (chapter 6.1. Execution and Memory Dependencies):

[...] a chain exists if the intersection of B_S in the first dependency and A_S in the second dependency is not an empty set.

Our "first dependency" (S₁) is the Semaphore Wait Operation submitted from vkSubmit. The B_S1 is pWaitDstStageMask (of everything in Submission Order after the Wait Op).

Our "second dependency" (S₂) is a Synchronization Op (between srcStage and layout transition) caused by our external Subpass Dependency (recorded on vkCmdBeginRenderPass). The A_S2 is our srcStage (of everything in Submission Order before the Subpass Dependency Op).

So if we used VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT, then the intersection is STAGE_COLOR (of everything in Submission Order after the Wait Op and before the Subpass Dependency Op).

So we formed a Dependency Chain (i.e. the two Dependencies behave as if there was only one from our semaphore signal to the layout transition).

[chrisvarns]

Thanks for the help, I will digest this and reread the bit about chains.

The AS2 is our srcStage (of everything in Submission Order before the Subpass Dependency Op).

This is my sticking point, right now it is not clear to me why/how this is guaranteed to be a non-empty set, although you seem convinced it is. What operations occur in that stage after the semaphore wait and before the layout transition? It reads to me like BS1 and AS2 are potentially empty sets, in which case there would be no chain.

[krOoze]

@chrisvarns Oh I see. No, that could use clarifying. You could interpret the "of everything in Submission Order after the Wait Op and before the Subpass Dependency Op" as an empty set.

I am only convinced of it because I also read the old version (≤ 1.0.34) of the spec as well, where it was made clear the supplied stage mask is the thing that matters:

A pair of consecutive execution dependencies in an execution dependency
chain accomplishes a dependency between the stages A and B via
intermediate stages C, even if no work is executed between them that
uses the pipeline stages included in C.

That was apparently lost in the rewrite.

[chrisvarns]

Nice one! Yes that is the language I was looking for 👍

Thanks for your patience :)

[oddhack]

@chrisvarns could this issue be closed at this point? It sounds like you got the answers you were after.

[chrisvarns]

@oddhack I got the answers, and people googling will hopefully land here and get them too, but it is worth reintroducing the language to the spec imo (if it hasn't already, I have not checked), as anyone who reads the spec post 1.0.34 may have the same confusion I did.

[Tobski]

@chrisvarns The language @krOoze specified covers this case - the original paragraph, or anything like it, wouldn't fit into the spec after the rewrite. It specifically dealt with only a subset of the wide variety of synchronization in the spec, which is largely why it was removed. I do agree that it's not the easiest language to read however, and we should write more/better developer facing documentation of how things work here, rather than modifying the spec IMO.

@alonorbach are we tracking the need to write developer facing documentation for synchronization anywhere? We should possibly raise an issue if not.

[krOoze]

@Tobski That assumes the spec is correct, complete, and unambiguous and the problem is the language is only too formal for inexperienced developer.

Can you produce spec quote which you think covers the case? If there is none, or if I can too easily weasel out of it by misinterpreting it, then it is IMO still technically spec Issue.

[chrisvarns]

@Tobski sorry I thought I replied to this. Fwiw my perspective is that the spec does not currently define that a chain of dependencies between sets of operations A -> B -> C is actually enforced if B is an empty set of operations. The language @Krooze referenced from a previous version of the spec does define this, and its removal in my opinion results in the behaviour becoming undefined.

[Tobski]

Ack I also forgot to reply to this!

@chrisvarns @krOoze
So it's specifically the empty set case that's at issue here? Sorry that got a little lost in the discourse above, I was under the impression that it was the idea of _execution dependency chains_ that was being suggested needed to come back, which I can point at language for (if I try hard enough... 😅).

The intent is that chaining empty sets like this works, at least for the semaphore case, but I can't think where/whether we'd have language for that. I'll have a look and report back. If there's missing/ambiguous spec language, I think I now understand enough to suggest a fix at least - thanks to both you and @krOoze for bearing with this 😄

[Tobski]

Huh, I actually found a single sentence that clears this up which... actually surprised me. We were clearly very thorough!

So we have this definition of execution dependency chain:

An execution dependency chain is a sequence of execution dependencies that form a happens-before relation between the first dependency’s A' and the final dependency’s B'. For each consecutive pair of execution dependencies, a chain exists if the intersection of BS in the first dependency and AS in the second dependency is not an empty set.

Note in particular the final sentence there:

For each consecutive pair of execution dependencies, a chain exists if the intersection of BS in the first dependency and AS in the second dependency is not an empty set.

BS and AS are defined as the scopes of the synchronization commands, _before the relevant sets of operations are intersected with them_. So as long as the two scopes both included the same pipeline stages, for example, then there's an execution dependency chain, irrespective of whether operations are present or not.

This is all from the start of "6.1 Execution and Memory Dependencies" in the Vulkan spec: https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/chap6.html#synchronization-dependencies

[krOoze]

@Tobski Well, yes. We were also thorough and covered that one sentence above too :D.

I was perhaps not clear how I could weasel out of that particular spec quote.

Synchronization scope is a set of pipeline stages and some free-form text (where the problem lies).

Let's say for our simple example there is a semaphore with pWaitStage PS. And first thing submitted is a pipeline barrier with srcStage also PS.

Now notice the spec language for those two synchronization primitives:

Semaphores:

In the case of vkQueueSubmit, the second synchronization scope is limited to operations on the pipeline stages determined by the destination stage mask specified by the corresponding element of pWaitDstStageMask. Also, in the case of vkQueueSubmit, the second synchronization scope additionally includes all commands that occur later in submission order.

Barriers:

the first synchronization scope includes all commands that occur earlier in submission order. [...] the first synchronization scope is limited to operations on the pipeline stages determined by the source stage mask specified by srcStageMask.

Ok, pWaitDstStageMask ∩ srcStageMask is still PS, all is well.

But intersection of "all commands that occur later in submission order" and "all commands that occur earlier in submission order" can be said is an empty set if there is no action commands recorded inbetween. Therefore I could claim there was no dependency chain created.

[chrisvarns]

Ok so old language:

A pair of consecutive execution dependencies in an execution dependency
chain accomplishes a dependency between the stages A and B via
intermediate stages C, even if no work is executed between them that
uses the pipeline stages included in C.

New language:

For each consecutive pair of execution dependencies, a chain exists if the intersection of BS in the first dependency and AS in the second dependency is not an empty set.

The penny has dropped. I must have misread that BS and AS are the scopes themselves, and the intersection is the intersection of the scopes, not of any operations. Enough time has passed since we had this conversation that it's not all in my head, but it looks like that resolves this issue for me. 👍

[Tobski]

Thanks @chrisvarns glad this is resolved to your satisfaction - sorry it took so long!

Now to finish up with you @krOoze 😛

So the scopes are defined as potential sets of things, not actual things:

The synchronization scopes define which other operations a synchronization command is able to create execution dependencies with.

So when it says "scope includes all commands that occur later in submission order" it means "there's a synchronization point here, and any/all commands later in submission order will be affected by it". What it does not mean is "the actual commands that are submitted are the scope". The actual operations are never included in the synchronization scope, those are resolved as the two halves of a dependency (A' and B').

Does that help clear things up?

[chrisvarns]

Thanks to @krOoze and @Tobski for your patience on this one :) I read the whole thing back and it is clear to me now.

[krOoze]

@Tobski Well, yea. I never imagined Vulkan would work otherwise (which would be weird).
Still, considering how convoluted you explanation sounds, it wouldn't hurt to clarify in spec slightly. The note currently there does more harm than good IMO. Could be replaced with the original or otherwisely improved.

Either way do not keep this Issue open for my sake.

[Tobski]

@krOoze you've mentioned a few notes above - which one are you referring to? I'm always keen to remove words from the specification ;)

[Tobski]

Also I'm pretty sure my explanation was less convoluted than yours :P

[krOoze]

You had to claim it is actually some metadescription to claim it does not mean what it says. :D

I'm always keen to remove words from the specification ;)

my man

Anyway, this note in the sync chapter:

An execution dependency is inherently also multiple execution dependencies - a dependency exists between each subset of A' and each subset of B', and the same is true for execution dependency chains. For example, a synchronization command with multiple pipeline stages in its stage masks effectively generates one dependency between each source stage and each destination stage. This can be useful to think about when considering how execution chains are formed if they do not involve all parts of a synchronization command’s dependency. Similarly, any set of adjacent dependencies in an execution dependency chain can be considered an execution dependency chain in its own right.

I think the current note contents are not that useful. Especially the last part. You can say "dependency" only so many times in one sentence before it stops making sense. I think it tries to make a mental model of how chains work but ultimately the wording seems worse than the formal text to me. Also it seems to convolute what could be two topics\paragraphs. (And well, it misses the stuff in this Issue, which I think is useful)

[Tobski]

Ah yes that note, if I recall correctly, I added that in order to appease something @jeffbolznv thought was unclear, though I don't remember him being particularly fond of this note as a resolution either - I think we mostly ended up accepting it due to attrition...

I'd be ok with removing the note if @jeffbolznv is ok with it?

I'm also not opposed to somehow reinforcing that the scope defines the _potential_ things a command can synchronize with, I'm just not sure how to do it off the top of my head. Suggestions welcome - maybe as a separate issue so that @chrisvarns can stop getting emails in his inbox 😅

[chrisvarns]

Don't mind me :)

On Wed, 22 May 2019, 23:24 Tobski, [e-mail] wrote:

Ah yes that note, if I recall correctly, I added that in order to appease
something @jeffbolznv https://github.com/jeffbolznv thought was
unclear, though I don't remember him being particularly fond of this note
as a resolution either - I think we mostly ended up accepting it due to
attrition...

I'd be ok with removing the note if @jeffbolznv
https://github.com/jeffbolznv is ok with it?

I'm also not opposed to somehow reinforcing that the scope defines the
potential things a command can synchronize with, I'm just not sure how
to do it off the top of my head. Suggestions welcome - maybe as a separate
issue so that @chrisvarns https://github.com/chrisvarns can stop
getting emails in his inbox 😅

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/KhronosGroup/Vulkan-Docs/issues/812?email_source=notifications&email_token=AAFTYVMLQXII2NBKRTF7RZTPWXBYJA5CNFSM4FYEHP5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWAQYPQ#issuecomment-494996542,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFTYVLGOJ64GVEBQZL6AJDPWXBYJANCNFSM4FYEHP5A
.

[jeffbolznv]

I haven't been following this discussion. I'm not particularly hung up on that note, if there's reason to remove it. I don't recall the motivation for it. It seems like it's describing a useful way to think about things, but I'm not sure it's really all that illuminating.

[Tobski]

@jeffbolznv I did a git blame to figure out the history - seems we were trying to call out explicitly how you consider the chaining of by-region dependencies, since those are multiple dependencies instead of one dependency - but that language has actually been clarified a fair bit since then, so possibly this is no longer as useful?

[jeffbolznv]

Yeah, I don't get anything about by-region dependencies from this note, so OK with me if you want to remove it.

[Tobski]

I would like to formally request that we nuke the note from orbit and close this issue: https://github.com/KhronosGroup/Vulkan-Docs/pull/1048