Antlr4: ANTLR4 go-lang target

[anidotnet]

Is there any plan or ongoing effort for a go-lang target?

[sharwell]

Not to my knowledge. I looked into it briefly but abandoned the effort before I had anything working. It's safe to assume there will not be a Go target until the day someone donates a fully-working independent project implementing a Go target, similar to what happened with the C# and Python targets.

[anidotnet]

Thanks. Out of curiosity, may I know what caused you to abandon the effort? Was there any specific hurdle to overcome? I am asking because if I ever try it I want to know what issues I can expect. Moreover, @sharwell I saw that you have created a go grammar for goworks, so can it be possible to create some kind of java to golang translator with it to do the heavy lifting?

[sharwell]

I had some doubts about the future of Go. The port would be an enormous amount of work that I couldn't be sure would have long-term benefits, especially compared to some of the other targets which I can be involved with.

[markmcd]

I thought I'd add my support, just in case someone with the spare cycles comes here considering the project.

[wangjild]

@sharwell is there any other specification or guide for create new language target? except for
https://theantlrguy.atlassian.net/wiki/display/ANTLR3/How+to+build+an+ANTLR+code+generation+target

[pboyer]

I'm currently working on this. I should have it open-sourced within the month. Not all of the tests pass yet, but I'm close. Let me know if you'd like to contribute.

[anidotnet]

@pboyer Great to know! Sure I would like to help when my time permits. Currently busy with so much of daily job stuff, so could not commit anything.

[pboyer]

Just so anyone knows who finds this discussion, this is the best place to find info on building a runtime target:

https://github.com/antlr/antlr4/blob/master/doc/creating-a-language-target.md

The awesome part about the system that @parrt @sharwell and @ericvergnaud have built is that the tests are actually generated from templates and executed across processes. This makes an implementor's life so much easier.

[willfaught]

Is there a reference for the target template syntax?

[parrt]

Hiya Will. It's http://www.stringtemplate.org syntax

[willfaught]

Thanks @parrt.

---

I see template functions that aren't used elsewhere in the template, like pythonTypeInitMap in the Python 3 template. Are those dead code, or are these functions called elsewhere?

[parrt]

Sometimes they are directly instanced from code not from other templates.

[willfaught]

All the existing targets appear to have, and make heavy use of, inheritance, which seems to complicate things. Any guidance/tips on how those templates are structured around inheritance and how that structure would change for a target without inheritance (like C)?

[willfaught]

I was thinking of converting the C# target, as it's typed like Go, but the Python ones seem simpler and Python has an instance-as-parameter concept similar to Go.

[parrt]

Honestly, it's hopeless so I'd fake inheritance if you can. The data inheritance wouldn't be bad but the virtual methods might be. Maybe fake inheritance with a "delegation" pointer?

[willfaught]

Any way to figure out what template func params mean/where they come from, e.g. ParserFile(...namedActions...) and Parser(...sempredFuncs...)?

It's not clear why there are separate Parser and Parser_ template funcs. Parser just seems to fill in the string "parser_ctor" for the Parser_ parameter ctor. I'm wondering if there's something going on there I'm not seeing.

[kkbkris]

Dear Will Faught,

I was actually meddling around with these inheritance structures that were implemented using a scripting language, which was specifically JavaScript in my situation if you needed to know, but I do gather that this is of not really of any help at all. Otherwise it is kind of a similar situation, at a stretch.

Yours Sincerely,
Kristian Robert David Stacey
Student reading Computer Science and Electronics Masters Degree in Engineering and Masters Degree in Science at the University of Bristol

On 28 Apr 2016, at 20:17, "Will Faught" [email protected] wrote:

All the existing targets appear to have, and make heavy use of, inheritance, which seems to complicate things. Any guidance/tips on how those templates are structured around inheritance and how that structure would change for a target without inheritance (like C)?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[parrt]

Most templates match up with a model object in code generator. Look for ModelObjectWalker or some such. see it's comment.

[jwkohnen]

Are there three parties independently hacking on this now? I'd like to help out, but only on an open joint venture.

[willfaught]

@pboyer Have you pushed your Go target work? Can you point us to it?

[willfaught]

Depending on how it's written, with the built-in Go parser it should be pretty straightforward to generate a C target from it.

[pboyer]

@willfaught @wjkohnen @parrt I should give an update.

I have a working Go runtime and generator templates. 80-90% of the test cases pass and I'm currently using it for a non-trivial task at work.

The situation being what it is, I am going through the process at my employer (Autodesk) of open-sourcing the work. I actually just started this, so I can't say how long it will take. I'm guessing ~2 weeks.

So, if you can wait that long, we should have a working Go target fairly soon. As I said, there's still a few test cases to get to 100%, but it's not a bad starting point. I'll try and give an update soon.

[parrt]

Wow. I had no idea you had gotten so far! That's great news. Then a C target could be around the corner as well :) C++ is progressing nicely thanks to @DanMcLaughlin, @mike-lischke, and @davesisson woot!

[willfaught]

Hi @pboyer, how's the Go target going? I don't mean to push, I'm just really interested in trying it out. I'd be happy to help if needed.

[pboyer]

@willfaught Thanks for following up. Turns out OSS bureacracy is rather involved. Legal, business reviews are done. Should be out soon.

[pboyer]

Hey @willfaught @parrt @sharwell @ericvergnaud,

Here's the fork including a go language target:

https://github.com/pboyer/antlr4

To run the tests, do the usual:

mvn install

To compile for Go, do the usual with:

-Dlanguage=Go

The new stuff:

• runtime/Go - Obviously the run time for go, based off a mixture of the JavaScript and Java versions.
• tool/src/org/antlr4/v4/codegen/GoTarget.java - Some directives for target code generation.
• tool/resources/org/antlr4/v4/tool/templates/codegen/Go - Runtime templates!
• runtime-testsuite/resources/org/antlr/v4/test/runtime/go/ - Test templates!

Please send me any issues there. I'll try and start adding issues to the fork to keep track of them. You'll notice I haven't synced with the main repo in a while. :P There's still a good bit of work to get 100% test completion. There's also a lot of room for improvement for the quality of the output code.

@parrt It is my hope that this work can someday find it's way into this repo. Is that possible? I figure this needs to live outside of mainline _at least_ until the test completion is fixed, but I'm not sure what additional expectations exist (given this happens fairly rarely).

[willfaught]

Great work, @pboyer! I don't see an issues tab there. Can you enable it?

[pboyer]

Done! I forgot that it's not on by default.

[parrt]

Woot! great news, thanks @pboyer !! I definitely intend to integrate it if I can get somebody to individually verify it all works ;)

[willfaught]

@pboyer What's the pb directory for?

[base698]

Amazing work, just tried to use go tool yacc--a fate worse than death. I'm excited about the prospects this could bring.

[parrt]

Can anybody else independently verify the go target is working? Do all the unit tests work as they do for other targets?

[willfaught]

There are 8 or 9 tests that still fail in pboyer/antlr4, the fork that adds the Go target. I've fixed as much as I can, but the rest of the failures require understanding how ATNs and all that work. There hasn't been any activity by the other contributors in a while; not sure what happened there.

Answering #1216 and similar future issues would help.

See https://github.com/pboyer/antlr4/pull/40 for a summary of the remaining work. gitter.im/pboyer/antlr4 may also be of interest.

[willfaught]

You can see the latest test failures here: https://travis-ci.org/pboyer/antlr4

Example:

Failed tests:
  testPositionAdjustingLexer(org.antlr.v4.test.runtime.go.TestLexerExec): expected:<...ns',<6>,1:0](..)
  testZeroLengthToken(org.antlr.v4.test.runtime.go.TestLexerExec): expected:<[@0,0:[4=''xxx'',<1>,1:0](..)
  testReturnValueAndActionsList1_2(org.antlr.v4.test.runtime.go.TestLeftRecursion): expected:<...pr (expr a) , (expr [c) >> (expr x])) <EOF>)(..)
  testReturnValueAndActionsList1_4(org.antlr.v4.test.runtime.go.TestLeftRecursion): expected:<...* (expr b)) , (expr [c) , (expr (expr x) * (expr y)) >> (expr r)]) <EOF>)(..)
  testBasic(org.antlr.v4.test.runtime.go.TestListeners): expected:<(a 1 2)(..)
  testTokenGetters_1(org.antlr.v4.test.runtime.go.TestListeners): expected:<(a 1 2)(..)
  testAmbiguityNoLoop(org.antlr.v4.test.runtime.go.TestFullContextParsing): expected:<[alt 1(..)
  testLoopsSimulateTailRecursion(org.antlr.v4.test.runtime.go.TestFullContextParsing): expected:<line 1:3 reportAttemptingFullContext d=3 (expr_primary), input='a(i)'(..)
  testTokenAndRuleContextString(org.antlr.v4.test.runtime.go.TestParseTrees): expected:<[a[,] s](..)

[pboyer]

@parrt All of the 4.5.2 tests now pass with this change:

https://github.com/pboyer/antlr4/pull/64

There will be more work to synchronize with antlr/antlr4 and clean up, but expect a PR for the Go language target soon. :)

[parrt]

woot!

[mike-lischke]

@pboyer Does this include tests for all other targets? Did you change anything in the test setup (template file etc.) that would need to be considered by the other language targets?

I'm working currently on the C++ target and had to change a few things which affects testing all other targets (this has mostly to do with additional parameters required for the code template), so I'd much appreciate if you could merge my branch into yours, for which I have a pending pull request already (should go smoothly if you are in synch with the upstream repo) and see if any of your tests got broken (and fix them if so). I'd be willing to do the same with your branch, of course, if this is required.

[pboyer]

Hi @mike-lischke We tried to avoid changing the test templates very much. So, I don't think this should be a problem. Thanks for the heads up, though and I'll let you know if I run into any difficulties.

[owenthereal]

@pboyer Awesome! When will the Go target be ready?

[pboyer]

@jingweno All of the language tests run in github.com/pboyer/antlr4. Right now, I'm just trying to make the API as idiomatic and consistent as I can before submitting the PR. The system is very usable at this point if you build from source.

[johnnadratowski]

@pboyer - I was trying to use your new Golang target last night, and I hit a bit of a snag:

This could very well be user error, as I have no experience with Antlr. I'm really excited to try this out though!

[robbert229]

You will need to make some serious changes to the cypher grammar to make it work with antlr. The grammar itself has values like return which causes errors because of the reserved keyword return.

I ran into the same issue while attempting to make a parser/lexer via antlr for cypher last week. I haven't cleaned everything up yet myself...

@johnnadratowski

[johnnadratowski]

@robbert229 - Once again, I'm very much a novice at this, but it seemed that after I resolved some simple name conflicts it was able to properly parse it. I can send you what I did, it was super simple (literally just renaming 2 items).

Then again, perhaps I was getting those compilation errors because of deeper issues with the Cypher antlr spec...

I really have no clue on this stuff, I was using this as a learning activity (just wanted to output syntax-highlighted cypher for a project I'm working on)

[robbert229]

@johnnadratowski The two changes that you made I made as well, they were my first step towards getting the go runtime working. ( still haven't gotten it working yet )

Additionally just wanted to say that I love your go bolt driver!

[johnnadratowski]

@robbert229 - Thanks, depending on how this initial learning went, I was thinking about adding some pre-flight cypher checks in the driver.

If you're interested in collaborating on antlr and/or the driver let me know.

[daflame76]

Can anyone get the Go language target to work with the grammars?

C:\Users\Jeremy\Documents\Projects\grammars-v4\golang>java -cp c:\users\Jeremy\.m2\repository\org\antlr\antlr4\4.6-SNAPSHOT\antlr4-4.6-SNAPSHOT.jar org.antlr.v4.Tool -Dlanguage=Go Golang.g4
error(134): Golang.g4:341:0: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:101:17: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:294:6: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:344:10: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:376:6: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:382:10: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:401:16: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:431:6: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:443:29: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:592:14: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:627:6: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:137:23: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:540:22: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:294:17: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:596:31: symbol type conflicts with generated code in target language or runtime
error(134): Golang.g4:80:23: symbol type conflicts with generated code in target language or runtime

C:\Users\Jeremy\Documents\Projects\grammars-v4\golang>

[daflame76]

Nevermind, fixed it myself. Apparently the Go language definition wasn't written with consideration for having a language with a keyword called type to actually parse it.

[willfaught]

That possibility of keyword conflicts isn't good. Maybe there should be a
production name prefix in generated methods to prevent keyword conflicts?
RuleFoo or something.

On Thu, Dec 8, 2016 at 8:54 PM DaFlame [e-mail] wrote:

Nevermind, fixed it myself. Apparently the Go language definition
wasn't written with consideration for having a language with a keyword
called type to actually parse it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/antlr/antlr4/issues/659#issuecomment-265933587, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD5Vq-tEAHxq26z1f3pVfqJzzHr8im3ks5rGN8JgaJpZM4CKlVd
.

[jwkohnen]

The situation is not ideal, but not limited to Go. I've learnt that the hard way, when I tried to use a grammar with "super" as a keyword (or rule name?). Target language was Java.

[sharwell]

For C# we solve this by converting reserved keywords into other identifiers that are allowed in the language. For example, if you name a rule base, you'll get @base as the method name in generated C# code.

One possibility for Java would be including $ as a prefix or suffix in such a case (but limited to Java keywords there of course). Other languages could automatically convert keywords in a similar manner.

[pboyer]

@sharwell Thanks for the pointers! Do you do that somewhere in the templates? Not sure if the keyword list is somehow propagated into the stg props.

The other place where we've seen this come up is if the user chooses a rule name that is the same as a method on the RuleContext type - like toString. This could be repaired as well.

[parrt]

@pboyer Take a look at, say, JavaTarget.java and you'll see a list of "bad words" :) Add keywords analogously there in your GoTarget.java.

[curious-attempt-bunny]

Did this work stall out, @pboyer? It's sounding pretty promising!

[dgryski]

My understanding is that the Go target has been merged into antlr4. Any bugs should be opened as new GitHub issues to be properly prioritized.

[jwkohnen]

See also https://github.com/antlr/antlr4/blob/master/doc/go-target.md