Lark: Lexer produces incorrect results

[ius]

The following doesn't seem to hold

When using a lexer (standard or contextual), it is the grammar-author's responsibility to make sure the literals don't collide, or that if they do, they are matched in the desired order. Literals are matched in an order according to the following criteria:

#!/usr/bin/env python3
import lark

grammar = """
A.2: "a"
WORD: ("a".."z")+

start: (A | WORD)+
"""

earley = lark.Lark(grammar, parser='earley')
lalr = lark.Lark(grammar, parser='lalr')

text = 'abc'

earley_tree = earley.parse(text)
print('Earley:', earley_tree)

lalr_tree = lalr.parse(text)
print('LALR', lalr_tree)

print(earley_tree == lalr_tree)

Earley: Tree(start, [Token(WORD, 'a'), Token(WORD, 'b'), Token(WORD, 'c')])
LALR Tree(start, [Token(WORD, 'abc')])
False

I'd expect the following:

LALR Tree(start, [Token(A, 'a'), Token(WORD, 'bc')])

The 'A' token seems to go missing here because it's considered to be embedded in 'WORD':
https://github.com/erezsh/lark/blob/255ef0d973a140b5be07cb4d97c5bf8f8e20788e/lark/lexer.py#L153

(Unrelated: I expected the same result for both parsers, not sure if the Earley tokenization is another bug or simply the way an Earley parser is supposed to work)

[erezsh]

Earley by default has its own "lexer", that works very differently from the lexers available for lalr.

And you're right, the documentation should mention embedded strings.

I think the correct behavior should be to not embed strings if their priority is higher than the regexp. Do you agree?

[ius]

Yes, that was the behavior I expected to see.

[erezsh]

Okay, I'll fix it soon, thanks for the bug report.

In the meantime, if you declare A as a regexp, it won't embed.

A.2: /a/

[erezsh]

Fixed.