Lark: IndexError: lark.parsers.earley_forest in visit_token_node

Created on 1 Oct 2019 · 6Comments · Source: lark-parser/lark

Hi!
I'm using earley parser in uwsgi app. And everything is fine, but periodically have the exceptions:

IndexError: deque index out of range
  File "app/helpers/predicates.py", line 16, in check_predicate
    return evaluate_predicate(predicate, attributes=attributes)
  File "predicates/evaluator.py", line 13, in evaluate_predicate
    ast = parser.parse(predicate)
  File "lark/lark.py", line 306, in parse
    return self.parser.parse(text, start=start)
  File "lark/parser_frontends.py", line 185, in parse
    return self._parse(text, start)
  File "lark/parser_frontends.py", line 54, in _parse
    return self.parser.parse(input, start, *args)
  File "lark/parsers/earley.py", line 319, in parse
    return self.forest_tree_visitor.visit(solutions[0])
  File "lark/parsers/earley_forest.py", line 281, in visit
    return super(ForestToTreeVisitor, self).visit(root)
  File "lark/parsers/earley_forest.py", line 204, in visit
    vtn(current)
  File "lark/parsers/earley_forest.py", line 284, in visit_token_node
    self.output_stack[-1].append(node)

I cannot reproduce the error manually.

My grammar:

predicate: _statement

_expr: attribute
    | bool_constant
    | attribute_comparison
    | version_attribute_comparison
    | time_comparison
    | functions
    | _composed_statement
    | not_statement

attribute.1: ATTRIBUTE_NAME

bool_constant.2: BOOLEAN

attribute_comparison.3: ATTRIBUTE_NAME (EQ|N_EQ) BOOLEAN                                      -> bools_comparison
                      | ATTRIBUTE_NAME (EQ|N_EQ) SIGNED_INT                                   -> nums_eq_neq_comparison
                      | ATTRIBUTE_NAME (GE|LE) SIGNED_INT                                     -> nums_ge_le_comparison
                      | ATTRIBUTE_NAME (EQ|N_EQ) _QUOTATION TEXT_WITHOUT_QUOTATION _QUOTATION -> strings_comparison

version_attribute_comparison.4: VERSION_ATTRIBUTE_NAME (EQ|N_EQ|GE|LE) VERSION -> versions_comparison

time_comparison.5: DATE_ATTRIBUTE_NAME (EQ|N_EQ|GE|LE) DATE         -> dates_comparison
                 | DATETIME_ATTRIBUTE_NAME (EQ|N_EQ|GE|LE) DATETIME -> datetimes_comparison

functions.6: _DEFINED _BRACE_OPEN ATTRIBUTE_NAME _BRACE_CLOSE   -> defined
           | _UNDEFINED _BRACE_OPEN ATTRIBUTE_NAME _BRACE_CLOSE -> undefined

_composed_statement.7: _BRACE_OPEN _statement _BRACE_CLOSE

not_statement.8: _NOT _composed_statement

?and_statement.9: _expr [_AND and_statement]

?or_statement.10: and_statement [_OR or_statement]

_statement: or_statement

ATTRIBUTE_NAME: /(?!NOT\_)(\w(\w|[-.])*)([a-zA-Z]\d*)/
VERSION_ATTRIBUTE_NAME: "source_client_version"
DATE_ATTRIBUTE_NAME: /[a-zA-Z0-9_.]+_date/
DATETIME_ATTRIBUTE_NAME: /[a-zA-Z0-9_.]+_time/

_DEFINED: "defined"
_UNDEFINED: "undefined"

BOOLEAN: "true"|"false"
TEXT_WITHOUT_QUOTATION: /[^']+/
_QUOTATION: "'"
VERSION: INT "." INT ("." INT)~0..1
DATE: INT ("-" INT)~2
DATETIME: INT ("-" INT)~2 "T" INT (":" INT)~2

EQ: "="
N_EQ: "!="
GE: ">="
LE: "<="

_BRACE_OPEN: "("
_BRACE_CLOSE: ")"

_NOT: "NOT"

_OR: /\s+OR\s+/
_AND: /\s+AND\s+/

SIGNED_INT: ["+"|"-"] INT

%import common.INT

Parser initialized with the next params:
parser = Lark(grammar, start="predicate", parser="earley")

Source