Vulkan-docs: vkCmdPipelineBarrier vs external subpass dependencies

[dylanede]

As far as I understand it, though without any explicit mention in the docs, the memory dependency created by VK_SUBPASS_EXTERNAL dependencies applies to all memory. There doesn't seem to be a way of restricting it to specific resources. Given that vkCmdPipelineBarrier supports specifying specific resources, it would appear to be the more natural fit in some cases (used prior to and after a render pass in a command buffer). In the case where vkCmdPipelineBarrier is used before beginning a render pass, does this mean that a render pass no longer needs to define an external dependency (using VK_SUBPASS_EXTERNAL) on previous commands in the queue (assuming that the accessed resources, masks etc. match the ones specified in the barrier command)? And from a driver perspective is this likely to produce better or worse performance, given that the dependency is no longer baked into the render pass ahead of time, but instead provided as part of command buffer recording?

[NicolBolas]

As far as I understand it, though without any explicit mention in the docs

If it is not explicitly mentioned, how do you "understand it" that way? If you believe that VK_SUBPASS_EXTERNAL applies to all memory, then you should be able to point to something in the standard that declares this to be so. That's what a standard is for, after all.

Remember that in the render pass model, attachments, for the duration of the render pass instance, do not exist in regular GPU memory. You can't read from them with normal reading operations, and you can't write to them with normal writing operations. You can only read or write to them through attachment operations.

As such, subpass dependencies don't need memory dependencies; they always affect the visibility attachments involved in the dependency. Attachments used in the dependent subpass are made visible to commands in the dependent subpass. The standard makes that clear (though it says it backwards; attachment contents become undefined if there are no appropriate dependencies).

I see nothing about VK_SUBPASS_EXTERNAL in the standard which states or implies that it represents a full memory barrier across all memory objects.

Also, it should be noted that if you don't explicitly define external dependencies, the API will define default external dependencies, both on the start and the end. This ensures the visibility of data in attachments.

[dylanede]

The reason I took that meaning from it is that in the synchronisation chapter, it is explained how synchronization primitives restrict the scope of resources that are involved in execution and memory dependencies, often with access masks and pipeline stage masks. It also mentions that some synchronization primitives will further restrict the scope, for example by allowing resources to be specified explicitly, such as vkCmdPipelineBarrier. The only constraints I could find in the standard on the resources affected by an external subpass dependency are those specified by the access and pipeline stage masks in the dependency, from which I came to the conclusion that the scope of resources given by the intersection of the access and pipeline stage masks is the scope that the memory dependency extends to. Presumably I have missed the section that describes further constraints (such as to the set of attachments bound in the render pass, which is what I was expecting to find).

[ghost]

Subpass Dependencies are... tricky. When I was trying to integrate the synchronization chapter changes we made last year into the renderpass chapter I got confused numerous times.

But here's what I understand the state of things as:

• Subpass dependencies are indeed a memory barrier that applies to all resources.
  
  
  
  • Effectively they are a pipeline barrier with a VkMemoryBarrier
  
  
  • Except they also count as including a VkImageMemoryBarrier for attachments referenced by both src+dst subpasses, with layouts as described by those two subpasses
    
    
    
    
    
    • External subpasses count as referencing all attachments with layouts described by the initial/finalLayout parameters of the render pass.
    
    
    
  
  
• Subpass dependencies have a lot of pre-planning associated with them, so the driver is free to move them around more.
  
  
  
  • This usually makes them more efficient than an equivalent pipeline barrier (barring driver tomfoolery).
  
  

The thing to remember is that rarely is a resource-specific memory barrier more efficient than a global one (assuming the same stage and access masks) - reason being that for the most part a cache flush is a cache flush - splitting one global flush into multiple resource-specific flushes is almost universally bad.

In general, I'd recommend using external subpasses instead of pipeline barriers, simply because the driver has the opportunity to optimise where the synchronization happens.

[dylanede]

Thank you, that's cleared things up for me. Would it be worth making this information (especially your first bullet point) more explicit in the documentation? Also, I know that it doesn't appear to be in the "style" of the docs, but examples would really help in the synchronisation and render pass chapters. Synchronisation is so important that confusion should be minimised as much as possible.

[ghost]

I think this is the sort of thing that would need a serious renderpass chapter tidy-up to make more obvious, which I'm not signing up to do I'm afraid :)

Potentially there's space for a note - if you have any suggestions about where that might go it would help. I have to go look at the renderpass docs for an unrelated issue today anyway, so I'll see if I can figure something out whilst I'm there.

[NicolBolas]

I would say that the place to put such information is where it lays out the meaning of VkSubpassDependency. It should simply say that it makes all memory modified by the srcStageMask visible and available to the dstStageMask. As well as making modifications to all attachments available and visible to the dstStageMask stages.

BTW, I would be concerned if subpass self-dependencies automatically act as a memory barrier for all resources. It would make more sense to limit self-dependencies to just the attachments, with the specific barrier call specifying which resources are available.

[ghost]

I would say that the place to put such information is where it lays out the meaning of VkSubpassDependency. It should simply say that it makes all memory modified by the srcStageMask visible and available to the dstStageMask. As well as making modifications to all attachments available and visible to the dstStageMask stages.

I mean it kind of already says that - using the access and synchronization scope language which is common to all synchronization. I'm loath to clarify it there because what makes subpass dependencies special compared to all other synchronization methods, I guess? Then again, people do find this particular case unclear, so maybe it's worth doing... I'd kind of like to know what other people think. So I'll leave this open for a bit and ping a couple of people about it.

BTW, I would be concerned if subpass self-dependencies automatically act as a memory barrier for all resources. It would make more sense to limit self-dependencies to just the attachments, with the specific barrier call specifying which resources are available.

Self dependencies do effectively nothing on their own - they only denote that you'll be adding a dependency via a vkCmdPipelineBarrier in the near future. The vkCmdPipelineBarrier is what determines the actual dependencies.

[NicolBolas]

I'm loath to clarify it there because what makes subpass dependencies special compared to all other synchronization methods, I guess?

They aren't mentioned in Chapter 6. That chapter says that there are exactly 4 "synchronization primitives": events, fences, semaphores, and barriers. Subpass dependencies are not on that list.

Second, there's the fact that every one of those synchronization primitives makes it clear how memory visibility/availability work. In the case of vkCmdWaitEvents and vkCmdPipelineBarrier, you provide explicit memory barriers. You also provide barriers when waiting on semaphores in batches. vkWaitForFences does not take barriers, but the specification also _explicitly says_ that waiting on fences does not constitute a memory barrier at all.

Subpass dependencies are different due to the lack of similar language. There's nothing that says that a non-self dependency represents a memory barrier, only an execution barrier. The closest you get to that is:

it defines a memory dependency between the subpasses identified by srcSubpass and
dstSubpass.

But that doesn't state which memory dependencies it defines. I suppose the simplest way to resolve this is to change it to say:

it defines an execution dependency and a global memory dependency between the subpasses identified by srcSubpass and dstSubpass.

With a link to the "global memory barriers" section. And possibly with a description that says it acts as a VkMemoryBarrier with srcAccessMask and dstAccessMask set to their counterparts in VkSubpassDependency.

[ghost]

They aren't mentioned in Chapter 6. That chapter says that there are exactly 4 "synchronization primitives": events, fences, semaphores, and barriers. Subpass dependencies are not on that list.

The very next paragraph says:

In addition to the base primitives provided here, Render Passes provide a useful synchronization
framework for most rendering tasks, built upon the concepts in this chapter. Many cases that would
otherwise need an application to use synchronization primitives in this chapter can be expressed
more efficiently as part of a render pass.

Which is supposed to be that information. I guess it could be more directly included rather than being a footnote. I considered pulling it into the sync chapter, but the render pass chapter is too messy to do that to in its current form, and I didn't want to rewrite ANOTHER chapter whilst I was rewriting the sync chapter 😨

Subpass dependencies are different due to the lack of similar language. There's nothing that says that a non-self dependency represents a memory barrier, only an execution barrier. The closest you get to that is:

In https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VkSubpassDependency there's very explicit language about the synchronization scope and access scope, which is identical to the language of the other sync primitives. "Memory Barrier" is a type of structure that defines an access scope, not a fundamental definition. Access scopes are the thing that define how memory is moved around. Since subpass dependencies don't take a memory barrier parameter, they just have access masks which define the access scope instead.

I think the issue here is that you're looking for "memory barrier" to be a defined term. A "memory dependency" is very clearly stated as the combination of an execution dependency and some amount of memory management. I'm not quite sure why you think "execution dependency and a global memory dependency" would clarify anything though.

Having said all that, whilst I can argue with you on individual points, I can't argue the fact that the explanation of subpass dependencies is somewhat inobvious, and a pain in the ass to pull out. The problem I think is that to properly fix it, as I've stated before, we'd need to rework the renderpass chapter - and I have no desire to go down that rabbit hole again right now (not ruling it out for the future though).

So I'm happy to add some sort of note to clarify things, but I don't want to add more attempted clarification to the core spec, because using different language to elsewhere would just inevitably confuse things further (that's what the sync chapter looked like pre-rewrite, and it was awful). It sounds like I should add a note somewhere that makes reference to VkMemoryBarrier, since the behaviour does mirror that to a degree.

[ghost]

Proposed text for the note:

For non-attachment resources, the memory dependency expressed by subpass
dependency is nearly identical to that of a slink:VkMemoryBarrier (with
matching pname:srcAccessMask/pname:dstAccessMask parameters) submitted as a
part of a flink:vkCmdPipelineBarrier (with matching
pname:srcStageMask/pname:dstStageMask parameters).
The only difference being that its scopes are limited to the identified
subpasses rather than potentially affecting everything before and after.

For attachments however, subpass dependencies work more like an
slink:VkImageMemoryBarrier defined similarly to the slink:VkMemoryBarrier
above, the queue family indices set to ename:VK_QUEUE_FAMILY_IGNORED, and
layouts as follows:

• The equivalent to pname:oldLayout is either the pname:layout parameter
  of pname:srcSubpass, or pname:initialLayout if pname:srcSubpass is
  ename:VK_SUBPASS_EXTERNAL.
• The equivalent to pname:newLayout is either the pname:layout parameter
  of pname:dstSubpass, or pname:finalLayout if pname:dstSubpass is
  ename:VK_SUBPASS_EXTERNAL.

[oddhack]

This should be fixed in the 1.0.59 spec update.