Galaxy: Improve Galaxy XSD for better IDE autocompletion/help/validation support

[davelopez]

Hi!
I think it would be great to have real-time auto-completion, help and validation during XML tool wrappers development (in addition to the good old planemo linter). This can be very useful for new tool developers to ease the learning curve as they can discover the API, read the documentation for best practices and avoid syntax and structural errors as they type.
Many IDEs like IntelliJ, VSCode and others can provide this functionality, but, right now, it seems there is no full support for some valid elements like macros (i.e. the expand element as reported in #3029) and many elements are missing documentation or more restrictive type definitions.

I am no expert in XSD, but I would like to contribute to improving it if I can :)

So far, the first difficulty I found is about how to support multiple expand elements inside the tool element.
Right now, the current tool element definition uses the <xs:all> indicator to define its children in any given order. Unfortunately, this indicator does not allow more than 1 occurrence of any of its child elements, so the expand element will be allowed just once.
After some research, looks like there are some ways to overcome this problem:

1. Replace <xs:all> with <xs:choice maxOccurs="unbounded">: this will allow having multiple expand elements but at the cost of also allowing multiple occurrences for the rest of the elements which I do not think is acceptable.

2. Replace the <xs:all> with <xs:sequence>: this will allow specifying the minOccurs and maxOccurs for each child element without the maxOccurs="1" limitation at the cost of having to define the child elements in the specific order, which probably would make sense since the coding style best practices already define this order.

I would probably go for option 2, but since I am fairly new to Galaxy I am unaware of the consequences of replacing this <xs:all> indicator and if this may break some other tools relying on the current XSD like planemo, the autogenerated documentation, etc.

Thanks in advance for your inputs!

[jmchilton]

It is very hard to model the expand in the XSD because it can literally appear anywhere in the document. The tool element is just one problem - every child element of that element may contain an expand at any position. Even if you update every element of the schema to allow an expand at every position - an invalid document might be missing an element that comes in as a result of the expansion process - so that schema would still not reflect what Galaxy XMLs are valid or invalid. Planemo does an expansion and then validates against the XSD - I don't know of another way of doing that. The macro languages just makes the XML language very un-modelable by XSD. If you want to improve editor support I'd look at non-XML ideas - like maybe a language server to expand the macros and then apply existing XML parsing and validation?

[davelopez]

Thank you for your fast reply @jmchilton!

I totally agree that the perfect solution would be to implement a custom LSP that could expand the macros and then validate, but I was hoping on a solution that can be used with the existing LSP for XML like this one as I have no previous experience with LSPs.
Maybe we can just provide a "good enough" support for macros in the XSD that can deliver a "helpful experience" without many validation errors thrown to the user, always considering that planemo will be used in the end to verify that the final expanded XML is valid.

What do you think?

[davelopez]

Finally, I had the time to look into the LSP documentation and started a pet project to learn also a bit of Python and Node.js in the process :)
Here is the repo for the language server and a small Visual Studio Code extension that uses it. It is still in early development, but it may be a start to something more useful in the future :)
Thanks again for the idea!

[dannon]

@davelopez That looks fantastic, very cool!

[davelopez]

Thanks so much @dannon! glad you like it :)

[hexylena]

Hi @davelopez, this is fantastic to see!! I've wanted this since 2015 but couldn't convince anyone to let me spend time on it. I'm so glad you've done it! This will make tool development so much easier. (There are some ideas in #1195 for further additions if you want to build some more VSCode extensions!)

[davelopez]

Thanks, @hexylena! It still needs lots of work to become something useful, so any ideas or feedback from more experienced tool developers are greatly appreciated!