Lark: Incorrect production of _ambig nodes when ambiguity="explicit"

[FrancescoManfredi]

Here what happens (it only happened in this case and I can't imagine why): the parser returns a parsing three containing a different phrase than what was given as input to the .parse method. Moreover there is an ambiguity in the input phrase and this isn't represented. I'm fairly sure the second thing is causing the first one.

I'm attaching a test to reproduce this behavior:
https://pastebin.com/yJW66hu6

Thanks for building this and thanks in advance if you take this issue into consideration.

[MegaIng]

I am not the maintainer, but here a bit of general advice to make it easy for everyone involved:

• Post the difference between what you get and what you want. What do you mean with 'different phrase than what was given as input to the .parse method.'? Please shown an example of this.
• Your grammar is really big, so it would really help if you try to reduce it to an minimal example.
• If you don't manage to do this, it would also help if you point out where the ambiguity you know of is in the grammar.

[FrancescoManfredi]

Thanks for you answer. You are right, I can be more clear. Sorry but this is the first time I submit an issue on GH.
I'm going to add more details:

When I parse the sentence:
"c'è probabilità di nevicate domani"

I get the following parse tree:
start
  weather_question
    weather_question_int_13
      question_snow
        question_snow_int_53
          motion_loc
            motion_loc_int_1    c'è
          chance
            chance_int_0
              probabilità
              di
          query_snow
            query_snow_int_4    nevicate
          motion_loc
            motion_loc_int_1    c'è
          chance
            chance_int_0    probabilità
          query_snow
            query_snow_int_4
              di
              nevicate
          future_point_in_time
            future_point_in_time_int_0  domani

which would be correct if I parsed the sentence: "c'è probabilità di nevicate c'è probabilità di nevicate domani".

About the ambiguity:
chance_int_0 has the ability to match "probabilità di";
query_snow_int_4 has the ability to match "di nevicate";
The expression "chance_int_0 query_snow_int_4" (not directly present in the grammar but achievable as "chance query_snow") can therefore match "probabilità di nevicate" in two different ways depending on which one produces the "di" part.
The parse tree shown above somehow shows that there is an ambiguity, but it does not introduces an _ambig node in the point where the ambiguity arises.

I'm sorry about the size of the grammar, but the original one was even bigger and I managed to extract just the minimum required to run the test.
You can run the test on your own if you just download the pastebin and invoke it with a python interpeter.

[erezsh]

An important thing to note, is that without ambiguity="explicit", the result is a valid (and probably correct) tree:

start
  weather_question
    weather_question_int_13
      question_snow
        question_snow_int_53
          motion_loc
            motion_loc_int_1    c'è
          chance
            chance_int_0
              probabilità
              di
          query_snow
            query_snow_int_4    nevicate
          future_point_in_time
            future_point_in_time_int_0  domani

Which leads me to believe that the bug is probably not in the Earley parser, but the subsequent ForestToAmbiguousTreeVisitor step. @night199uk Perhaps you'd be interested to know.

@FrancescoManfredi If you provide a much shorter example, there are better chances that we would take the time to resolve it.

[FrancescoManfredi]

Thank you for taking into considerations this issue.

I've reduced the complexity of the test and replaced the italian words with uppercase letters so it's easier to keep track of what happens.
Here is the new test: https://pastebin.com/L7BqFU2d

Now the parsed phrase is "A B C D E" and the parse tree that comes out is for the phrase "A B C D A B C D E".

I'm also pasting the grammar here:

start: weather_question
weather_question: question_snow
question_snow: motion_loc chance query_snow future_point_in_time
motion_loc: ML_0
ML_0: "A"
chance: CH_0 [CH_1]
CH_0: "B"
CH_1: "C"
query_snow: [QS_0] QS_1
QS_0: "C"
QS_1: "D"
future_point_in_time: FP_0
FP_0: "E"

%ignore " "

And the resulting parse tree:

start
  weather_question
    question_snow
      motion_loc    A
      chance
        B
        C
      query_snow    D
      motion_loc    A
      chance    B
      query_snow
        C
        D
      future_point_in_time  E

[erezsh]

I narrowed this down to an even simpler version:

from lark import Lark

newG = """
start: ab bc d
!ab: "A" "B"?
!bc: "B"? "C"
!d: "D"
"""

l = Lark(newG, ambiguity='explicit')

p = l.parse("ABCD")
print(p.pretty())

The result is

start
  ab
    A
    B
  bc    C
  ab    A
  bc
    B
    C
  d     D

So the problem here is that the ambiguity node isn't created, and instead the two derivations are just appended to start.

Note that changing the first line to

start: ab bc "D"

Results in a different kind of error, where the ambiguity isn't even detected, and instead only one derivation is chosen (but at least output is a "correct" tree).

@night199uk Maybe you'd like to take a look?

[erezsh]

Can we consider this issue resolved?

[erezsh]

Looks like it's fixed. I consider this closed unless someone shows otherwise.