Vulkan-docs: Make vk.xml more concise for other languages

[ghost]

(first of all _many_ thanks for even having made an xml spec
c, this makes it a thousand times easier to make a wrapper for vk)

I would like to propose a few improvements to the vk.xml spec format.

• so many xml element tags are labeled as "type", but have absolutely completely different behavior, most of them could be named by their category eg. <type category="basetype">typedef <type>uint32_t</type> <name>VkSampleMask</name>;</type> could be <basetype>typedef <type>uint32_t</type> <name>VkSampleMask</name>;</basetype>
• many elements contain random bit of specific c code that are not grouped within another xml element or xml attribute, this makes my parsing both harder, more prone to error, and increase the chance that my code won't be able to parse the next version of the spec. eg. <type category="bitmask">typedef <type>VkFlags</type> <name>VkFramebufferCreateFlags</name>;</type> could be simply <type category="bitmask"><type>VkFlags</type><name>VkFramebufferCreateFlags</name></type> (removed the semicolon and typedef), functionpointer category types are specially guilty of this.
• pointers are also kindof akward to parse. eg. <type>VkBufferCopy</type>*
• some of the enums are akward to convert eg.<enum value="(~0ULL)" name="VK_WHOLE_SIZE"/> could be written as <enum value="18446744073709551615" type="uint64_t" name="VK_WHOLE_SIZE"/> (same for 1000.0f)
• command validity seems to be specific for whatever code documentation khronos is using. Would be great if it could be a bit more cross-language friendly. (low priority)
• There is no reliable way to link between vk.xml commands/types and the online reference https://www.khronos.org/registry/vulkan/specs/1.0/apispec.html

There are other improvements that could be made to help cross-platform wrapping but if you're interrested in improving that would be a good start.

ps: I may be ignorant on some of these matters.

many thanks for all your hard work

[NicolBolas]

so many xml element tags are labeled as "type",

To do otherwise would create a number of XML elements that have the exact same structure (more or less) yet have different names. That makes parsing such XML rather more difficult. It also makes it more difficult to create new categoryies in the future.

And really, is it that much more difficult to say type[category = "basename"] than to say basename in XPath, or the equivalent logic in whatever parsing system you're using? The parsing burden here is pretty trivial.

many elements contain random bit of specific c code that are not grouped within another xml element or xml attribute

And the fact that such text is not inside an element or attribute makes it easy enough to _ignore_. Personally, I don't like that they're there, but such text can safely be ignored.

<enum value="18446744073709551615" type="uint64_t" name="VK_WHOLE_SIZE"/>

If you're going to write it out, it should be as hex: 0xFFFFFFFFFFFFFFFFULL.

[ghost]

• <type category="include">#include &lt;<name>xcb/xcb.h</name>&gt;</type>
• <type requires="X11/Xlib.h" name="Display"/>
• <type category="define">#define <name>VK_MAKE_VERSION</name>(major, minor, patch) \ (((major) &lt;&lt; 22) | ((minor) &lt;&lt; 12) | (patch))</type>
• <type category="basetype">typedef <type>uint32_t</type> <name>VkSampleMask</name>;</type>
• <type requires="vk_platform" name="void"/>
• <type category="bitmask">typedef <type>VkFlags</type> <name>
• <type category="handle"><type>VK_DEFINE_HANDLE</type>(<name>VkInstance</name>)</type>
• <type name="VkAttachmentLoadOp" category="enum"/>
• <type category="funcpointer">typedef void (VKAPI_PTR *<name>PFN_vkInternalAllocationNotification</name>)( <type>void</type>* pUserData, <type>size_t</type> size, <type>VkInternalAllocationType</type> allocationType, <type>VkSystemAllocationScope</type> allocationScope);</type>
• <type category="struct" name="VkOffset2D"> <member><type>int32_t</type> <name>x</name></member> <member><type>int32_t</type> <name>y</name></member> </type>

Absolutely different behavior and most of them widely different structure

And really, is it that much more difficult to say type[category = "basename"] than to say basename in XPath, or the equivalent logic in whatever parsing system you're using? The parsing burden here is pretty trivial.

in my case, it actually is, I'm not using XPath, I have to setup a bunch of ugly switch to check how to interpret the data, the data type that models the type xml element has to account for a handful of completely unrelated data which leads to a data structure that is completely unreadable. It's also a question of readability, I read <type> yet I see that basicaly everything is a type but they don't all do the same thing, also there are type tags inside type tags and its for a totally different reason.

<member optional="true">const <type>VkPhysicalDeviceFeatures</type>* <name>pEnabledFeatures</name></member>

It's cool that your code is fine. But the code I need to pick up pointers like these is ugly and error prone and hard to read. Other people might want to use vulkan too you know.

If you're going to write it out, it should be as hex: 0xFFFFFFFFFFFFFFFFULL.

hex or dec doesnt matter, but don't put the data type (ULL) with the constant it makes my life harder to separate the two.

[NicolBolas]

Absolutely different behavior and most of them widely different structure

But your suggestion did not make a distinction between categories that do have "widely different structure" and those that do not. A basetype and a handle are more or less the same thing. Whereas a funcpointer and a basetype are really two different elements masquerading as one.

in my case, it actually is, I'm not using XPath, I have to setup a bunch of ugly switch to check how to interpret the data, the data type that models the type xml element has to account for a handful of completely unrelated data which leads to a data structure that is completely unreadable.

I don't know what language you're using, but I can write a Lua script that can handle those cases pretty trivially:

local type_procs =
{
  {test = function(element)
    return element.att["category"] == "struct"
  end, func = ProcessStruct},
  ...
}

One would simply take each type element and iterate through type_procs. If the test function returns true, execute the func. This is, structurally speaking, the exact kind of code I would use to parse through any XML: a jump table with functions that test if the element matches something. It's just doing a bit more matching logic than just testing the element name.

Python, Perl, and other scripting languages probably have similar constructs to make this kind of thing easy.

But the code I need to pick up pointers like these is ugly and error prone and hard to read. Other people might want to use vulkan too you know.

So, considering that Vulkan's API is specified _in the C language_, how exactly would you suggest that vk.xml specify Vulkan's API? That member definition you quoted, for example. What XML would you prefer to see?

[ghost]

But your suggestion did not make a distinction between categories that do have "widely different structure" and those that do not. A basetype and a handle are more or less the same thing. Whereas a funcpointer and a basetype are really two different elements masquerading as one.

Sounds like you're saying the spec SHOULD be hard to read ?

So, considering that Vulkan's API is specified in the C language, how exactly would you suggest that vk.xml specify Vulkan's API? That member definition you quoted, for example. What XML would you prefer to see?

It is a C api, but why bother to make a xml spec if the goal would be to only use it for C.

basetype

<type category="basetype">
    typedef <type>uint32_t</type> <name>VkSampleMask</name>;
</type>

to

<basetype>
    <name>VkSampleMask</name>
    <type>uint32_t</type>
</basetype>

orders of magnitude cleaner, so this is a C typedef of uint32_t to a type called VkSampleMask, the python code to generate the C api is as easy to write, albeit a line or 2 bigger, but for others it allows us to simply make the type traduction of uint32_t to whatever we use in our source and create a simple function to create the typedef our way. The only information I need to know is {"It's a typedef", "the name of the new type", "the type its based on"}. This is beautifully generic for many languages.

functionpointer

<type category="funcpointer">
typedef void (VKAPI_PTR *<name>PFN_vkInternalAllocationNotification</name>)(
    <type>void</type>*                                       pUserData,
        <type>size_t</type>                                      size,
        <type>VkInternalAllocationType</type>                    allocationType,
        <type>VkSystemAllocationScope</type>                     allocationScope);
</type>

omg this is unparseable, extracting the parameters name is a HUGE pita.

<funcpointer>
<returntype>void</returntype>
<name>PFN_vkInternalAllocationNotification</name>
<args>
    <arg name="pUserData", datatype="void*">
    <arg name="size", datatype="size_t">
    <arg name="allocationType", datatype="VkInternalAllocationType">
    <arg name="allocationScope", datatype="VkSystemAllocationScope">
</args>
</funcpointer>

Literally a million time easier to make a wrapper for that type. As easy to generate the C decl.

structs

(struct in general are not bad but)

<type category="struct" name="VkMemoryBarrier">
    <member><type>VkStructureType</type>        <name>sType</name></member>
    <member>const <type>void</type>*            <name>pNext</name></member>
    <member optional="true"><type>VkAccessFlags</type>          <name>srcAccessMask</name></member>
    <member optional="true"><type>VkAccessFlags</type>          <name>dstAccessMask</name></member>
</type>

that pointer outside the inner <type> makes my parsing code stupid and error prone.

<struct name="VkMemoryBarrier">
    <member><type>VkStructureType</type>           <name>sType</name></member>
    <member  const="true"><type>void*</type>            <name>pNext</name></member>
    <member optional="true"><type>VkAccessFlags</type>          <name>srcAccessMask</name>
    <member optional="true"><type>VkAccessFlags</type>          <name>dstAccessMask</name></member>
</struct>

also the target language may not have a concept of const.

commands

commands, in general are well made (notice that they use their own xml element and not <type category="command">), however they do exhibit the same annoying pointer behavior as the struct example above.
eg. https://github.com/KhronosGroup/Vulkan-Docs/blob/1.0/src/spec/vk.xml#L3081

conclusion

Also at this point, I have to ask right. Are you actually arguing that the spec is easily readable, well organized and easily parseable ?

Like if one of the dev closes this ticket with "not fixing for backward compatibility reason", I would just shut my mouth that's totally understandable (it also gives me a guarantee that the current format of the spec won't change much.) However if this gets closed because Khronos is satisfied with the quality of the xml spec (from a generic perspective). I'm sorry but that would just be plain wrong. The spec is nice, I'm absolutely grateful it's even there but there are obvious improvements that can be made on it and pretty much ALL wrappers would benefit from it.

[krOoze]

Well, it's a relative of ol' issues #6 and #19 and #83.

They said vaguely, it was/is too hard to make/maintain. So we should be glad Vulkan is out now ( even if it was promised for xmas :'( ) and not troubling itself with some pesky xml to this day.

@hydroflame Seems you have good ideas. I like to think, it's open for a reason. Maybe start some alter_vk.xml + header generator then, with all the niceties and language-impartial xmlness you speak of.

[ghost]

Some of the suggestions on this thread are good and deserve looking into.

The layout of all the function pointers, commands etc where they're just C snippets with embedded xml tags was originally used to avoid having to lay out the header in the generators. However we're now formatting in the xml anyway for whitespacing issues. I think moving to something that is closer to pure xml would be a good and non-invasive step to make it easier to create other languages.

There are some problems with the suggestions (the type tagging is important to determine dependencies, so you still want to separate a VkType from whether it's a pointer or not), but this should all be addressable whilst still making the whole thing a bit cleaner.

Tobias

[NicolBolas]

First, some technical considerations:

<arg name="pUserData", datatype="void*">
<member  const="true"><type>void*</type>            <name>pNext</name></member>

This is inconsistent. For function arguments, each definition is a self-contained element, with the data being attributes. But for members, each definition puts the data of interest into sub-elements. A parameter declaration and a member declaration store the same kind of information. So they should store it in the same way. Even if they have different element names, they ought to use the same data model.

In the best case, a user ought to be able to use the same function to process either element.

Also, if the goal is to make processing for non-C languages easier, then your goal should be to make it possible to process 100% of the information through XML alone. That is, you should not need any textual processing. As such, you need to remove _all_ vestiges of C.

That means pointer types should not be <type>typename*</type>. The user shouldn't have to parse the type string to figure out if a type is a pointer or not. It should be some attribute on the type element or something else that's detectable through XML, rather than requiring text parsing. Perhaps:

<arg name="pUserData" basetype="void" decorator="pointer"/>

---

Sounds like you're saying the spec SHOULD be hard to read ?

No, I'm saying that I don't agree that the current specification is "hard to read". More clearly, my point is that the _specific issue_ you cited, the use of a category attribute, did not make the spec "hard to read".

basetype, handle, and several others category types are all minor variations on the same idea. There is no need to give them different element names, even if the processor needs to output different things for them.

And quite frankly, if your XML parsing system makes it significantly more difficult to detect <type category='basetype'> instead of <basetype>, then I respectfully suggest that you are using a poor XML parser.

Are you actually arguing that the spec is easily readable, well organized and easily parseable ?

I feel that a distinction should be made between "easy to read" and "hard to read". Just because something is not "easy" doesn't mean it is _hard_; there is a lot of room between these two.

vk.xml is well organized in terms of its data. It's parsing format is adequate, with information presented in a consistent, parseable format. It is, in all ways, functional.

How "easy" it is to process depends entirely on how you intend to use it. vk.xml's primary purpose is to be used to generate vulkan.h. At that goal, it is _exceedingly_ efficient. Indeed, one does not even need an XML parser to generate such a header; the XML tags can in most cases be blindly stripped. Any other fixup could probably be done with a clever regex, not an XML parser.

Using it to do anything other than generating vulkan.h requires _effort_. It requires parsing beyond mere XML element/attribute stuff in several circumstances. Even generating good C++11 (replacing typedef with using, for example) requires more work than is strictly required of the task.

But...

The alternative is to develop a format that makes it _harder_ to turn into vulkan.h. The current system can just strip XML tags for the most part. A more platform-neutral vk.xml would require the C generator to do _actual work_. Maybe that's a worthwhile tradeoff.

But you should not pretend that it is _not_ a tradeoff.

Personally, I don't _like_ many of the choices made in the file. But I do not dislike them enough to say that everyone should have to rewrite their _existing_ parser frameworks. Especially those belonging to Vulkan.

[ghost]

I agree with your first point. Type parsing should be consistent wether its a struct member, a command argument, or a funcpointer declaration.

Also, if the goal is to make processing for non-C languages easier, then your goal should be to make it possible to process 100% of the information through XML alone. That is, you should not need any textual processing. As such, you need to remove all vestiges of C.

FYI that makes it harder for me to generate my wrapper and I totally agree on that point.

the XML tags can in most cases be blindly stripped.

again, why bother to make a xml spec ?

A more platform-neutral vk.xml would require the C generator to do actual work.

Yeah, but most of the changes are gonna be to turn "stripping xml tags from the inner element" to using sprintf

the python code to generate the C api is as easy to write, albeit a line or 2 bigger, (my quote)

But Khronos has always had "cross-platform" in mind, I feel that cross-language is in the same spirit. A more concise document does mean a bit more work for the C generator, no doubt. But it also means a TON less work for the others.

then I respectfully suggest that you are using a poor XML parser.

Quite possible, It's fairly new, I'm not gonna defend it.

Personally, I don't like many of the choices made in the file.

These are examples, they are most likely not perfect and I hope that Khronos won't just shove them in vk.xml without reflecting on what actually, on a fundamental level, they are trying to describe.

also, thank you @TobiasHector for the consideration

I should've checked other issues before posting this but hey looks like every other wrapper dev are complaining about the exact same things :D.

@TobiasHector would it be possible to give me a very very rough estimate of when this might be addressed ? I absolutely don't mind if this is "longterm" but If it just so happens to be shortterm I might put a hold to my wrapper so I don't waste my time.

[ghost]

Oh, it would also be nice if the padding/align of structs was somewhere to be found so I can guarantee that the structs you define match the C structs

[ghost]

@TobiasHector would it be possible to give me a very very rough estimate of when this might be addressed ? I absolutely don't mind if this is "longterm" but If it just so happens to be shortterm I might put a hold to my wrapper so I don't waste my time

@hydroflame Unfortunately it's probably "medium term" :smile:. I don't want to make a big breaking change to the xml without making thoroughly sure it's the right thing - so it'll take some time to come together. Also have to get people to agree on the change, in particular @oddhack - and ideally I'll solicit feedback from you guys.

Oh, it would also be nice if the padding/align of structs was somewhere to be found so I can guarantee that the structs you define match the C structs

This is defined by the C-standard and the compiler, so it's not something we strictly control, we usually just rely on the compiler to generate the fastest possible code. I suspect the easiest thing to do would be to create a program that compiles against the header and outputs the size/alignments of each struct - which is probably not too difficult to do?

[ratchetfreak]

This is defined by the C-standard and the compiler, so it's not something we strictly control, we usually just rely on the compiler to generate the fastest possible code. I suspect the easiest thing to do would be to create a program that compiles against the header and outputs the size/alignments of each struct - which is probably not too difficult to do?

Technically it's the platform's ABI that defines the struct layouts and padding.

This of course means that the platform is responsible for defining a good ABI and making sure everything is sane...

[ghost]

Technically it's the platform's ABI that defines the struct layouts and padding.

Well if you're going to be precise, sure :stuck_out_tongue:

[ghost]

create a program that compiles against the header and outputs the size/alignments of each struct - which is probably not too difficult to do?

Nope no problem, at least I know what I have to do, thanks

[NicolBolas]

I want to do a 180 on my position here. I've taken a closer look at registry.rnc and vk.xml. And I have to say... they seriously need an enema.

Even ignoring the issue of porting to other languages, there are just so many odd things in there. For example, take the enum tag. It can appear in two places. But it has different meanings and content models in those places. And that'd be fine...

Except that registry.rnc doesn't _recognize_ that fact. It shoves both content models into the same element. Not because RelaxNG is incapable of allowing two elements with the same name to have different content models based on context (it very much can do this, and do it easily). But because the schema was simply poorly written.

If vk.xml is going to be fixed for language issues, then it needs to be fixed for _all_ of these kinds of issues. We don't want to make people constantly rewrite their processing code just to make things nicer. We need to have a single change to a more flexible format, then keep to that format.

[ghost]

Ok I've noticed something else that's super weird. So @vulkan devs, you say you need the <type> element because it has the name attribute and you use that in the <feature> element to find all the function/types/enums to generate based on target platform. That's fine, however, if you look at type>require>command, the element has a name attribute BUT the command element does NOT have a name attribute, its hidden in command>proto>name. Pick one convention :P

[ghost]

ok ok here's another really weird thing. You guys define the <feature> element to declare all the stuff you're gonna need. vk.xml#5023 declares a section of structs that are declared but not used in the API. Listen, it doesn't even matter why that's there but not used, the important point is that the structs that ARE used are NOT declared in the <feature> section. Why ? XD

[ghost]

It's worth pointing out (again!) that a lot of this came from the way the OpenGL and OpenGL ES had their registry laid out. We didn't really redesign it for Vulkan, and Vulkan had a lot of complexity that this schema just simply wasn't designed for. Equally, a lot of use cases that weren't factored in originally have come up. Subsequently this format has evolved over time and it's not exactly what you'd call pretty, or consistent - but it IS functional.

So yes, I'd like to change it - but it requires careful planning this time around, and we only want to break all the generators once, not twice (or more!) :) So unfortunately, expect this to take some amount of time.

[ghost]

(That said, I will start looking at it once the asciidoctor transition has been done)

[NicolBolas]

Would you be willing to accept contributions from the community on this? For example, we could iterate on an alternative schema and registry file to get commentary on how well it performs for various needs. Then once that's done, we could submit it for Khronos approval.

[ghost]

AFAIK we're still waiting for the CLA situation to be resolved for the spec before we're able to take substantive things from the community - so I'm not 100% sure whether this would be ok or not in lieu of that. I'll raise it in the WG to see if we'd be on reasonable ground accepting schema change contributions.

Obviously if it's ok to do that - then yes any input is certainly going to be valuable!

[NicolBolas]

Has there been any movement on this? Specifically about "the CLA situation" and accepting contributions from the community.

[krOoze]

@NicolBolas I was aking them something like that in issue #328. Seems, we are not there yet.
EDIT: vk.xml seems to be under some different licence, so could perhaps accept modifications even now...

[SilverPhoenix99]

There's pull request https://github.com/KhronosGroup/Vulkan-Docs/pull/180 related to issue https://github.com/KhronosGroup/Vulkan-Docs/issues/83 mentioned in this issue https://github.com/KhronosGroup/Vulkan-Docs/issues/210#issuecomment-217855480

[NicolBolas]

@SilverPhoenix99: That change doesn't really make the spec easier to process for non-C languages. It removes one value, which was not even part of the RelaxNG schema.

I have been making progress on a complete rewrite of the registry.rnc schema format to make processing much more regular, minimize the amount of C-parsing, and basically making the RelaxNG actually functional. I've been writing Lua scripts to convert from the old vk.xml format to the new one, and once finished, I'll write a script to reverse the process.

In so doing, I can prove that the new format contains 100% of the information of the old one by invoking the existing processing infrastructure on the round-tripped format. And/or just by comparing the XML files.

As I've written the converter, I found that processing vk.xml was harder than I thought. Unless you are writing declarations _exactly like C_, you have to do a bunch of regex work to extract useful information. Something as simple as having a C++ interface that takes gsl::spans instead of dynamic arrays means having to do real processing work.

[SilverPhoenix99]

That change doesn't really make the spec easier to process for non-C languages. It removes one value, which was not even part of the RelaxNG schema.

Sorry, I didn't check it, but it's not a problem. I understand if it doesn't match your requirements.

As I've written the converter, I found that processing vk.xml was harder than I thought. Unless you are writing declarations _exactly like C_, you have to do a bunch of regex work to extract useful information.

I understand your pain. I've done a similar thing in Ruby and had to manually parse some text with regexes too. If some unusual C code is added to the xml, it might break easily.

I've been writing Lua scripts to convert from the old vk.xml format to the new one, and once finished, I'll write a script to reverse the process.

That's great! I would love to see how the new vk.xml turns out. If possible, can you give some examples of the changes?

[NicolBolas]

That's great! I would love to see how the new vk.xml turns out. If possible, can you give some examples of the changes?

I guess that's kinda the question at this point: is it OK to publicly create a project (that isn't necessarily a fork of this one) that transforms vk.xml into a different format? The license in vk.xml seems to suggest that this is perfectly fine so long as the copyright info is maintained, but krOoze's #328 issue makes it unclear if that's allowed.

The general gist of the changes is merely what we've talked about in this issue. The design goals for my format are, in rough priority-order:

1. Ensure that 100% of the current information in vk.xml is captured in the new format. Transformation back and forth should be lossless.
2. Make it easier to generate bindings for non-C/C++ languages. Extracting information from the current registry requires being able to parse C/C++ type declarations. This should be avoided where possible.
3. Make the RelaxNG schema actually verify the basic integrity of the document. The current registry.rnc doesn't provide much verification, even of the basic structure of the registry. For example, according to the schema, the <type category="basetype"> element can include structure <member> child elements, even though such a thing has no meaning and cannot be correctly processed by tools.
4. Make the format more consistent. For example, the name of a type element sometimes appears in an attribute and sometimes in a <name> sub-element within the element. This makes processing such declarations more difficult. However, do note that improving consistency can sometimes lead to information being in two places.
5. Make the format more logically structured. For example, the current schema allows the content model of an element to radically change because of an attribute being set to a certain value.

The various <type category="whatever"> elements are really completely different elements, and in many cases have entirely different conceptual content models. Yet they still claim to be type elements. This requires processing code to be radically different for the same element, based on the presence or absence of an attribute.

1. Minimize changes needed to the existing processing infrastructure, where they do not conflict with the above. If the existing infrastructure is going to be able to process the new format, it will have to change somewhat. But using different element names is typically an easy fix. By contrast, using a completely different structural layout is not. The latter should be avoided unless there is a genuine need.
2. Layout the RelaxNG schema in a way that makes it easier to extend via RelaxNG's own extension/inclusion mechanisms. This also makes it easier to extend when adding new kinds of data. It should be possible to search for where an attribute or element is defined just by knowing the attribute/element's name.
3. Make the format amenable to transformation to and processing in non-XML formats like JSON. This means avoiding lots of "markup"-style formatting.

[krOoze]

@NicolBolas Let's not get paranoid here. There seem to be an "explicit written permission" in the vk.xml file.

The issue/my answer is generally about the CLA, the (non)existence of contributor process (they probably do not even read PRs to avoid legal problems) and the spec copyright (the one in the spec text and the Asciidoc source files).

[NicolBolas]

That seems reasonable.

My attempt is up on GitHub for those who are interested. The format itself is "finished" (modulo bugs, missing stuff, etc), but the converter is not. I haven't gotten to feature and extension elements yet. But everything else is there.

[oddhack]

It's unlikely we would adopt significant changes to the current XML format any time in the forseeable future, but people are certainly welcome to rewrite / reinterpret it. That's why it's under an OSS license.

As I've noted previously, the choice to have mixed-mode C declarations in the XML is _intentional_. It is very much based out of lessons learned with the older SGI .spec file format. Obviously it is possible to parse this information and convert it to bindings for other languages, as you have done.

There is no progress to report on the CLA situation. When there is it will be announced but there is no point in continuing to ask about it. We're well aware of the problem.

[NicolBolas]

It's unlikely we would adopt significant changes to the current XML format any time in the forseeable future

What about basic sanity changes, like not having sibling elements use the same element name when they have completely different contents? Or upgrading registry.rnc to accurately reflect the actual restrictions specified in the format's documentation (to the limit that RelaxNG can, of course)?

Obviously it is possible to parse this information and convert it to bindings for other languages, as you have done.

It should be understood that I have not in fact done this. That is, what I have written is _just enough_ of a C type parser to get through vk.xml as it currently exists. Which means that while I can parse "const Typename *", I cannot parse "Typename const *".

I imagine that I am not the only person to take such shortcuts when it comes to writing this sort of thing. That's the downside of having a format that is not well-specified like this: people will take shortcuts and become dependent on the current way things are specified.

[oddhack]

I can't tell you anything more than I have before. I get that you don't like the schema. Nonetheless it works for the intended purposes and has been adopted by others for their purposes without so much dissatisfaction. We might tweak it in minor ways as time allows, and if it's not disruptive to the existing users. I'm glad you have been able to transform it to better meet your needs.

[Sairony]

There's a lot of dissatisfaction about the format, in #vulkan on freenode at least consensus seems to be that vk.xml is a complete mess. I don't see why some effort couldn't have been put into it, there's no redeeming factor for the current format at all. And now since it's already released into the wild it will be hard to change. "Good enough" is the mentality that gave us windows.h. Instead of a few giving it some thought & looking a bit ahead, now everyone has to suffer. We parse @NicolBolas version since it's vastly superior, but now we have to live with the fact that we're relying on the converter being maintained.

[NicolBolas]

There's a lot of dissatisfaction about the format, in #vulkan on freenode at least consensus seems to be that vk.xml is a complete mess.

I don't think people grousing on IRC counts as "a lot of dissatisfaction". Unless those people doing the grousing or the projects they maintain are significant or important in the graphics community, I don't think Khronos is going to be very interested in improving the format to appease them.

I'm more curious as to why this issue hasn't been closed as WNF. Oddhack's statement "it's unlikely we would adopt significant changes to the current XML format any time in the forseeable future" pretty much sums up Khronos's position. So it's best to accept that we have the format we have and do our best with it.

[Lucretia]

The issue is this, the original OpenGL spec files were "designed" to allow generating to strongly typed languages, I'm assuming Ada here as there were bindings originally for Ada on SGI machines. When John created the new GL xml spec (which is what the Vulkan one is derived from), he tried to continue with the separation, but fixated on C dumped a load of C crap in it that could've just been generated by the generator after reading in the xml., now every other binding generator has to ignore this stuff and parse typedefs.

The solution is simple, remove anything specific to C to the generator, add type information and make sure all parameters and returns have specific types. The generators can then create their own type map. I mentioned this in a thread on gl.org somewhere and it was dismissed as "this is a C API," so what? Not everyone wants to use C.

The xml file should easily support the generation of bindings to any language without having to parse language specific stuff. Parsing XML is enough of a pain.

[oddhack]

The XML schema will evolve modestly in the future as driven by new requirements in the API and toolchain, and by uses in other Khronos APIs as those evolve. It isn't likely to be fundamentally changed, however.

[chapmanworld]

^^ Just wanted to add support for everything hydroflame mentions here.

I'm writing a generator for Delphi/Pascal, and the structure of the XML is tricky to parse for all the reasons mentioned.

In particular, numeric constants are very difficult because of type-cast constants defined as expressions. My options with these constants are to either assume that they'll never change (and hard-code a test for that), or to write a C/C++ numeric constant parser which is really non-trivial. Particularly when expressions are used to define the constant...

<enum value="(~0U-1)" name="VK_QUEUE_FAMILY_EXTERNAL"/>

In this case, I must parse a type-cast unsigned integer, bitwise not, and the expression -1.
If this were represented by a literal value, it would be trivially easy to parse and would not require changes to the XML structure, unless you also wanted to specify the data-type .

[Lucretia]

@chapmanworld exactly, other parsers shouldn't have to parse, or have to skip C/C++ code. This XML file should be language agnostic, all that language specific crap in the XML should go into the C/C++ generators, not like it's hard to do or anything.

[chapmanworld]

@Lucretia I agree entirely, however, with so many C/C++ dwelling members of the Khronos group - I understand the disregarding attitude towards other languages. I added my comment here mostly as a gripe, not really expecting anything to change (but always hopeful).

I also said nothing of the macros embedded in the XML document, where macro's are almost exclusively a C/C++ feature - but with only a handful of them, I just went the hard-coding route there too.

[Lucretia]

@chapmanworld The original OpenGL spec files were written in an agnostic way, they even had type maps, probably because they were generating Ada bindings for GL back then. Now, no care is taken, they just espouse the same "it's a C API," well, get this, not everyone uses or even wants to use C.

Yup, the macros stuff, even more crap imo. Best to just match against the relevant tag names and bin them. But even that can be difficult. Again, C/C++ specifics should go into the C/C++ generator. This is (really very) basic software design.