Roslyn: Please provide the C# grammar, lexer, parser files

[Chris2011]

As I wanted to try to integrate C# into netbeans, I need the grammar or maybe better the antlr generated lexer and parser files. Is it possible to provide those files for C# 6.0 or for the new comming 7.0, if it's available? It would be very helpful.

As you can see here: https://github.com/ChristianWulf/CSharpGrammar

there are such files, but these are outdated.

Regards

Chris

[AdamSpeight2008]

@Chris2011
It is possible to download the language specification document for C#5 and VB11. If you any addition of VS prior to VS2015 it also included with the installation.

[Chris2011]

So as I understand it right, there is no official grammar file for the C# specification to use? I have to read the spec from the document? Only what I found is a completed summary from the things which are needed from the document: https://msdn.microsoft.com/de-de/library/aa664812(v=vs.71).aspx

For my perspective, it should be an official grammar (.g) file available and updated. So then you can generate the ANTRL parts (.lexer, .parser) from this official file. It is very hard to update an unofficial document or self created grammar file I mean by each own. This this should be a general file for all and should be updated. Only my opinion.

[AdamSpeight2008]

@ljw1004 https://github.com/ljw1004/vbspec
And https://github.com/AdamSpeight2008/Grammars which is based from the publicly available sources.

(C#) https://www.microsoft.com/en-us/download/details.aspx?id=7029
(VB) https://www.microsoft.com/en-us/download/details.aspx?id=15039

http://blogs.msdn.com/b/lucian/archive/2010/04/19/grammar.aspx

[Chris2011]

Ok and this is for C# 5.0. Do you whether there is a plan on the roadmap when an update for the C# 6.0 Spec is possible?

[monoman]

@Chris2011 You can always reverse engineer the custom parsers/lexers that Roslyn uses, they are not generated from a grammar, specially not from an ANTLR grammar. Although some source code generation occur in the build process it is mainly in the AST part of the code not in the parsing/lexing bits.

[Chris2011]

Of course this should be possible, but this is too much effort for me alone I think, not to much effort in general. But thx for the hint.

[dsaf]

@Chris2011

If you are OK with third-party sources and exotic notations, the JetBrains Nitra guys have a C# 5 grammar:

https://github.com/JetBrains/Nitra/tree/master/Grammars/CSharp/CSharp.Grammar/CSharp

http://confluence.jetbrains.com/display/Nitra/Home

[paul1956]

@AdamSpeight2008 the grammar you list has the following External Directive. I have never seen VB support external source. Is this something I have missed, or an error in the grammar. This would be a great feature if VB did support it.

//  13.2.2    External Source Directives

Start                       ::=   ExternalSourceStatement*
ExternalSourceStatement     ::=   ExternalSourceGroup  |  LogicalLine
ExternalSourceGroup         ::=   '#'  ExternalSource  '('  StringLiteral  ','  IntLiteral  ')'  EoL
                                    LogicalLine*
                                  '#'  "End"  ExternalSource  EoL

[AdamSpeight2008]

It is not an error in the grammar.

https://msdn.microsoft.com/en-us/library/szc67b5s.aspx

-------- Original Message --------
From:Paul M Cohen [email protected]
Sent:Fri, 05 Feb 2016 00:07:32 +0000
To:dotnet/roslyn [email protected]
Cc:Adam Speight adam.[email protected]
Subject:Re: [roslyn] Please provide the C# grammar, lexer, parser files (#8379)

@AdamSpeight2008 the grammar you list has the following External Directive. I have never seen VB support external source. Is this something I have missed, or an error in the grammar. This would be a great feature if VB did support it.

// 13.2.2 External Source Directives Start ::= ExternalSourceStatement* ExternalSourceStatement ::= ExternalSourceGroup | LogicalLine ExternalSourceGroup ::= '#' ExternalSource '(' StringLiteral ',' IntLiteral ')' EoL LogicalLine* '#' "End" ExternalSource EoL

—
Reply to this email directly or view it on GitHub.

[Chris2011]

@dsaf thx, I will have a look there too.

[KvanTTT]

The modern grammar with C# 5 & 6 support available in an official ANTLR grammars repository (C# runtime): https://github.com/antlr/grammars-v4/tree/master/csharp

[Chris2011]

Very nice, thx :)

[Chris2011]

After researching a bit and help from an other developer, I used JavaCC to parse a .jj file (based on a java grammar example file and rewrite it a bit for C#). You can see the result here: https://github.com/Chris2011/CSharp-for-netbeans/blob/develop/src/org/chrisle/netbeans/plugins/csharp4netbeans/filetypes/cs/jcclexer/cs.jj

It was based on this documentation: https://msdn.microsoft.com/en-us/library/aa664671(v=vs.71).aspx.
Then I saw, that this is very old and not maintained anymore, so I have to rewrite the .jj file and I will use this documentation: https://msdn.microsoft.com/en-us/library/x53a06bb.aspx

Maybe someone can help here to fix some stuff, which is not correct inside of it? Would be great. I started with it and the base is JAVA and I removed what is not working in C# and added stuff from C# to it. I don't know how contextual keywords will work (like get, set, var, etc.) so I added them to the normal keyword list.

Other progress you can see here: https://github.com/Chris2011/CSharp-for-netbeans/tree/develop/src/org/chrisle/netbeans/plugins/csharp4netbeans/filetypes/cs

Would be great that someone can help with fixiing the jj file, if someone want or give some other hints, etc.

[Chris2011]

FYI, I switched from JavaCC to ANTLR4.

[gafter]

We did a lot of work to write a parser that people wanting to parse C# can use. We didn't do it by producing a grammar and using some kind of tool, but rather we wrote it by hand. We do not plan to attempt to do it another way and hope to get a parser that behaves identically to the one in Roslyn. We only intend to maintain one parser.

[Chris2011]

@gafter so as I understand you right, I can forget to use the ANTLR version until someone will update this one from the official, handwritten one from you/your team, right? That handwitten lexer/parser whatever is your own format and similar to roslyn?

So that means, more or less, I have to use roslyn (however) or the LSP (if syntax highlighting is possible) to get the full features of newest C#, right?

[CyrusNajmabadi]

That handwitten lexer/parser whatever is your own format and similar to roslyn?

The handwritten lexer/parser is Roslyn :)

The format is just a C# module that lexes/parses and produces the SyntaxTree that powers the Roslyn compilers/IDEs.

You can use this directly as just a library. Or you can access to higher level versions of this through projects like OmniSharp. Or you can get it by using hte LSP and calling into some existing server that already hosts this stuff.

My recommendation woudl be the latter. Otherwise, if you just try to use a raw lexer/parser, you're going to spend a huge amount of time building up parity with projects that have already invested millions of hours here that you could just piggy-back off of :)

[Chris2011]

Big thx to you @CyrusNajmabadi. I totally agree with you. I thought it is enough having the grammar but as I can see and already saw that there is no real, good lexer and it is a big discussion about that, this topic is not that simple and can't be handled with a single grammar file. So I will try to change my code and use whatever is better.

Also I already looked into the OmniSharp project, some months ago, but the documentation wasn't that good. The last comment was: Please have a look into the Vim or Emac implementation... That was semi helpful.

Maybe the LSP will be the best solution for me, as a beginning project.

Thx for all the info and input. :)