Roslyn: Add Grammars To Documentation

[AdamSpeight2008]

Would it be possible to add the language grammars for C# and VB.net to the available documentation?

[AdamSpeight2008]

Asking because I can only seem to find VB11 and C#5

[gafter]

We're still investigating how best to ship the language specifications.

[AdamSpeight2008]

@gafter
I working on creating text file of just the grammars as based on the two *.doc that ship with VS2015RC. VB.net I've done already (-licence), current working on the C# one _but it uses a different form of grammar specification, which I'm converting to be the same as the VB.net one_

If I were to make the publicly available I think it is only appropriate to the include copyright / license and attribution within them. I don't know what license etc (will cover it).

Another option is to have another Git Repo dotnet/roslyn grammars/ to which I could submit a PR to.
Or should I do PR to this repo with them in documentation section?

[gafter]

@AdamSpeight2008 You'll have to consult with your own lawyer for legal advice on what part of the VS2015RC documentation you can copy, make derivative works of, and publish. I am not aware of any permission grant in those documents that would allow you to do what you're suggesting, but even if I were, I am not a lawyer (and even if I were, I'm not _your_ lawyer).

However we are investigating re-releasing the specifications in the open.

[AdamSpeight2008]

My goal is to have the language grammar itself accessible via an api
One approach is attributes on the parse functions.

<grammar("hh ::= ( ('0'|'1') '0'-'9' ) | ('2' '0'-'2')">
<grammar("mm ::= ('0'-'5')('0'-'9')>
<grammar("ss ::= ('0'-'5')('0'-'9')>
<grammar("time ::= hh ':' mm (':' ss)?>
<grammer("literal-time ::= '#' time '#'">
Function Parse_DateLiteral ....

Then have it statically verified it matches the grammar.

[AdamSpeight2008]

I've started a repo.

[niemyjski]

+1 please release the grammars

[billhenn]

Guys the formal specs/grammars were requested for the new C#/VB versions back in May here and VS 2015 RTM still ships with outdated old version specs. When and where can we get the updated specs/grammar? We really need this for our products so that we can accurately make changes to our C#/VB parsers.

[gafter]

@MadsTorgersen @ljw1004 Do either of you have any insight on when or how our customers will have access to the formal grammars for C# and VB?

[billhenn]

Guys, any word on this? We have customers asking for us to update our C#/VB parsers to understand the latest syntax and really need the formal grammar updates. Otherwise we have no reliable specs to go on.

[niemyjski]

+1

From: Bill Henning [email protected]
Sent: Wednesday, February 10, 2016 11:09 AM
Subject: Re: [roslyn] Add Grammars To Documentation (#3169)
To: dotnet/roslyn [email protected]
Cc: Blake Niemyjski [email protected]

Guys, any word on this? We have customers asking for us to update our C#/VB parsers to understand the latest syntax and really need the formal grammar updates. Otherwise we have no reliable specs to go on.

—
Reply to this email directly or view it on GitHub.

[gafter]

@MadsTorgersen The natives are restless.

[ejsmith]

Yes, I don't understand why the entire language has gone open source, but the spec hasn't been updated. Seems kind of ridiculous.

[gafter]

@ejsmith Yes, I also hate it when people don't give me enough stuff for free.

[niemyjski]

@gafter pretty uncalled for and rude comments from a .NET Foundation member...

Just found out what I was looking for Contributor Code of Conduct sub item #4

Public or private harassment

[ejsmith]

Yeah, I'm not trying to start trouble, but C# 6 has been out for quite a while now. And C# is an open ECMA standard. Seems like the spec should be available.

[gafter]

@niemyjski How can agreement be construed as harassment? ;)

As you can see I reopened this issue and assigned it for attention. We are working with ECMA, but ECMA standards are not generally published with an open license.

[billhenn]

@gafter Thank you for reopening the issue. For anyone like my company that is developing third-party tooling for the C# and VB languages, it's really important to have the official specs and grammar up-to-date so that we can properly support all the great work you guys are doing in language enhancements.

[nickjudson]

+1: For products which use .NET languages within a scripting environment, this is important!

[ljw1004]

While waiting for an official C#6 spec, I put up this temporary unofficial copy: https://github.com/ljw1004/csharpspec/blob/gh-pages/README.md

It incorporates this hyperlinked version of the grammar: http://ljw1004.github.io/csharpspec/csharp.html

And this version in ANTLR format: https://github.com/ljw1004/csharpspec/blob/gh-pages/csharp.g4?raw=true

[billhenn]

Hi @ljw1004, thanks for the unofficial copy. We need to be able to "diff" the updates for 6.0 as compared to 5.0, so that we can update our custom C# parser appropriately. In the past, we would take the Word document spec for a version and the next version and would diff the grammar portion of the Word document. It doesn't look like your unofficial Word document spec contains the grammar though, so it's difficult to know what is new/changed. Do you happen to have any sort of "diff" that we could examine that would compare 6.0 to 5.0?

[ljw1004]

@BillHenning here's the diff from C#5 to C#6: https://github.com/ljw1004/csharpspec/pull/1

You can look at just the .g4 file to see the grammar changes.

NOTE: the C#5 grammar I use here denotes the same language as the official C#5 spec, but I modified some of the production rules to make better use of ANTLR primitives such as kleene-star * (zero or more of) and plus + (one or more of). The official C#5 spec only used ? (zero or one of).

[billhenn]

Thanks @ljw1004, you rock. I know I'm totally pushing my luck, but you don't happen to have similar for VB 14, do you?

[ljw1004]

@BillHenning No sorry no VB yet. That's my job and I'm a bit behind. I'll see what I can bang out this week. You can follow progress at
https://github.com/ljw1004/vbspec/blob/gh-pages/README.md

[nickjudson]

@ljw1004 Any update on the VB version? Thanks.

[danipen]

@ljw1004 Could you tell us something about the release date of the official C#6 grammar? Thanks!!

[AnthonyDGreen]

@nickjudson the VB spec has been uploaded to a new repo @ https://github.com/dotnet/vblang/
@danipen the C# spec has been uploaded to a new repo @ https://github.com/dotnet/csharplang/

[zspitz]

@ljw1004 @AnthonyDGreen Would there be any value in making an ANTLR4 compliant grammar for VB.NET? Since (presumably) the Roslyn compiler wouldn't make use of such a grammar, the only benefit would be to make the rules written into the spec verifiably unambiguous.

While the current version linked to from the VB.NET spec README has the .g4 extension, it bears only a vaguely superficial resemblance to a valid ANTLR4 grammar. It causes errors and warnings when passed through ANTLR4.

In addition, character ranges and classes are denoted like so:

AlphaCharacter
    : '<Unicode classes Lu,Ll,Lt,Lm,Lo,Nl>'
    ;

which ANTLR interprets as a rule which matches the literal text <Unicode classes Lu,Ll,Lt,Lm,Lo,Nl>. The actual notation should be something like:

AlphaCharacter
    : [\p{Lu}\p{Ll}\p{Lt}\p{Lm}\p{Lo}\p{Nl}]
    ;

Also, presumably XML literals should be defined using different lexical modes.

NB. I haven't looked at the C# ANTLR grammar; does it also have similar problems?

[AdamSpeight2008]

I initially opened this issue, because of the lack of an up to date specification / grammar accessible to external members. The lang repos didn't exist at the time, the situation is a little better today. The contents of the specification still need bring in sync with released version of the languages.

[zspitz]

@AdamSpeight2008 Since the point of the grammar is not to generate a fully realized parser from the grammar, perhaps the grammar could be generated by running some script over the specification and extracting the ANTLR code blocks into a separate file?

[billhenn]

It's unfortunate this was closed. At the time I write this, the grammar specifications here are now multiple release iterations behind production C#/VB.

[CyrusNajmabadi]

@BillHenning you are welcome to open an issue to ask for this. Adam closed out all his issues.