Pegjs: Different errors returned when cache is enabled/disabled

[dmajda]

Grammar:

Statement
  = "{" __ !Statement Statement __ "}"
__
  = [ \t\r\n]*

Input:

{x}

With results cache disabled, the above produces this error:

Expected "{" or [ \t\r\n] but "x" found.

With results cache enabled, the error changes to:

Expected [ \t\r\n] but "x" found.

The errors should be the same in both cases.

Original report

[nikku]

I'm relying on the quite expressive SyntaxErrors pegjs throws to build syntax completion for users.

However, this issue prevents me from doing that for more complicated grammars:

• with cache enabled it suppresses possible matches
• without cache, parsing slows down by magnitudes, making it unusable

[nikku]

Another grammar for which the error applies:

Start
    = Char+ End

End
    = "e"

Char
    = (!End [a-z])+

Input:

a

With results cache disabled, the above produces this error:

Expected "e", [a-z], or not "e" but end of input found.

With results cache enabled, the error changes to:

Expected [a-z] but end of input found.

The cache disabled error is correct, as ae matches the grammar.

[nikku]

Failing test cases added with #555

[nikku]

The reason for this issue seems to be:

• during initial evaluation of a rule, peg$expect is called to record expected tokens. In certain cases (peg$silentFails > 0) tokens are being ignored though.

• during cached evaluation, the cached result is being restored. At this point the parser must not only restore the new parse state, it must replay relevant calls to peg$expect, too. Otherwise it misses out on expected tokens. This manifests in the incomplete error messages seen above.

In a local prototype based on [email protected] I was able to fix this issue by recording and replaying calls to peg$expect. However with the changes introduced in https://github.com/pegjs/pegjs/commit/669f782a5f3928a2958147992eb07df5b0ecf54a this stuff got a lot more complicated.

I could attempt a fix for pegjs@dev. Just wondering if I'm heading in the right direction.

Any comments @Mingun, @futagoza?

[futagoza]

Can you show me your local prototype based on [email protected]?

[nikku]

You can have a look at a fix here.

[nikku]

It does not work properly yet for nested silenced elements:

Grammar:

Start
  = Char+ End
End "end"
  = "e"
Char
  = !End [a-z]'

Input:

a

Expected Message:

Expected end, or [a-z] but end of input found.

Actual Message:

Expected end, "e", or [a-z] but end of input found.

Cf. failing test case.

[futagoza]

@nikku This should be resolved now (thanks to you 🙇 ), with all 3 of the test cases passing (including the 3rd one for nested silenced elements).

The fix is present in the latest pegjs@dev release is just pushed to NPM, [email protected] (https://github.com/pegjs/pegjs#latest).

[nikku]

Thanks.

[nikku]

I've tested your changes today.

From what I see https://github.com/pegjs/pegjs/commit/f5b323b40124e9ebe1336b509af0716d5a31ce55#diff-cd2c6b13fdcedf68a390c8bb6ea65cafR148 introduces a breaking change, as it effectively removes the peg$silentFails === 0 guard in with --no-cache generated parsers.

[futagoza]

😨 oops, that was a stupid mistake. Gonna fix that soon, thanks for the heads up

[nikku]

Tried to create a test case but failed :cry:. I see that line having an impact on one of my more complicated grammars though.

[futagoza]

Restored the guard in https://github.com/pegjs/pegjs/commit/d06a5b52efcf0fa4b9e5bf21f97607786c1c7db5 and pushed the changes to pegjs@dev, tell me if this fixes the problem

[futagoza]

@nikku Is it fixed, or are you still having this problem? Just want to know whether I should reopen this issue.

[nikku]

It's fixed.

[StoneCypher]

@nikku - It's not fixed. Nothing was released. npm still has 0.10.0, and this was merged to a feature 0.11.0 branch that ryuu said is never actually going to be released

[nikku]

I've stopped hoping for pegjs and looked elsewhere a long time ago for exactly this reason.

No release = no one can actually use it. Pegjs has been historically bad with releases.

[StoneCypher]

I am hoping to change the peg release cycle by:

1. asking the original owner to allow me to re-cut a 0.12.0 from 0.10.0,
2. to cherry pick and consistently rapid-release features from the now dead 0.11.0 branch,
3. to remove the uncommon tools and configuration, in favor of a standard development setup, and
4. to get the library in a place where adding features is easy for the community

If something like that happened, under any set of hands, and then releases actually started, would you give it another chance?

[nikku]

Wish you the best of luck to accomplish that.

[StoneCypher]

thanks 😁

We'll see.