Antlr4: maven plugin generates tokens files in wrong directory

[james-q-arnold]

I have a paired lexer/grammar: XyzLexer.g4, XyzParser.g4.
The source files are in a package hierarchy: com.abc.def.
The plugin documentation claims the .tokens files should go in the same directory as the generated java files---and this would make sense. That is not the case. The generated tree:

target/
`-- generated-sources
    `-- antlr4
        |-- com
        |   `-- kestrelei
        |       `-- psl
        |           `-- lib
        |               |-- PslLexer.java
        |               |-- PslParserBaseListener.java
        |               |-- PslParser.java
        |               `-- PslParserListener.java
        |-- PslLexer.tokens
        `-- PslParser.tokens

The compile step is failing at the moment. I'm not sure the .tokens placement is an issue, but it might be.

[rdupz]

Same problem here.
I use a workaround.
I have a src/main/antlr4/Gasy.g4 file which contains

grammar Gasy;

@header {
package fr.polytech.unice.gasys.antlr.annotation;
}

and in the pom.xml i have :

<configuration>
        <outputDirectory>src/main/java/fr/polytech/unice/gasys/antlr/annotation/</outputDirectory>
</configuration>

It's dirty but it does the job :

src/main/java
└── fr
    └── polytech
        └── unice
            └── gasys
                ├── Config.java
                ├── antlr
                │   └── annotation
                │       ├── Gasy.tokens
                │       ├── GasyBaseListener.java
                │       ├── GasyLexer.java
                │       ├── GasyLexer.tokens
                │       ├── GasyListener.java
                │       └── GasyParser.java

[sharwell]

This appears to just be a mistake in the documentation. It's unclear why a workaround of any form would be necessary. Can you provide an example of how this is resulting in incorrect output?

[rdupz]

Here is an archive
You can refer to the README ;)

EDIT : The problem is the documentation says :

If your grammar is intended to be part of a package called org.foo.bar then you would place it in the directory src/main/antlr4/org/foo/bar. The plugin will then *produce .java and .tokens files in the output directory* target/generated-sources/antlr4/org/foo/bar.

and I can't reproduce this.
Best regards

[sharwell]

There are several problems in your pom.xml.

1. In the <dependencies> element, you are depending on antlr4-maven-plugin. This should actually be antlr4-runtime.
2. You included the antlr4-maven-plugin under the path (XPath syntax) /project/build/pluginManagement/plugins/plugin[artifactId/text()='antlr4-maven-plugin']. It should actually be found using the path /project/build/plugins/plugin[artifactId/text()='antlr4-maven-plugin'] (notice the lack of the pluginManagement element).
3. The antlr4-maven-plugin version is listed as 4.0. It should be 4.2.2.
4. An <outputDirectory> element is specified in the antlr4-maven-plugin configuration. This element should be removed.

In addition to the above, you should remove all of the generated files from src/main/java/...

With the pom.xml corrected, the grammars are generated as expected, and the build is successful. The .tokens files are generated in a different folder than the documentation states, but it does not cause any problems.

[rdupz]

Thank you for the corrections.
Indeed the .tokens location does not cause any problems.

[cam-m]

Hi there!

Apologies in advance if bumping an old issue is breaking protocol... but re this:

Can you provide an example of how this is resulting in incorrect output?

I can't report any issue with the output either, however, Intellij's antlr4 plugin (v1.8.1) actually reports a syntax error for the 'tokenVocab' option if the referenced Lexer name has no corresponding .tokens file in the package directory.

The maven plugin generates tokens files in the root of generated-sources, causing this error to appear.

Copying tokens files to the package directory (or configuring the antlr plugin's output directory correctly) resolves this error.

IMO it would be preferable that the mvn antlr plugin and the intellij antlr plugin each generate tokens files in a consistent location. It would also be good to remove the antlr4 plugin error if it is in fact spurious as @sharwell implies above.

Thanks!

Cam