Exist: XQuery parser chokes on legal attribute names

[joewiz]

What is the problem

eXist throws an XPST0003 error if it encounters an direct element constructor with attributes named copy-namespaces, empty-sequence, or schema-element.

I suspect there may be other cases since these are tokens defined in the XQuery parser. Other tokens that also contain hyphens do not raise this error.

What did you expect

I expected these legal attribute names to raise no such error.

Describe how to reproduce or add a test

Open a new query window in eXide or your editor of choice, and paste in any or all of the following elements:

<foo copy-namespaces="bar"/>,
<foo empty-sequence="bar"/>,
<foo schema-element="bar"/>

Submit the query for evaluation, and instead of seeing the literal element returned as expected, you'll see the following error:

Cannot compile xquery: err:XPST0003 unexpected token: copy-namespaces (while expecting closing tag for element constructor: foo)

Needless to say, BaseX and Saxon do not raise this error.

Full error from exist.log:

2019-01-10 12:30:35,757 [qtp1977003571-110] ERROR (XQueryServlet.java [process]:552) - Cannot compile xquery: err:XPST0003 unexpected token: copy-namespaces (while expecting closing tag for element constructor: foo) [at line 3, column 6] 
org.exist.EXistException: Cannot compile xquery: err:XPST0003 unexpected token: copy-namespaces (while expecting closing tag for element constructor: foo) [at line 3, column 6]
    at org.exist.http.servlets.XQueryServlet.process(XQueryServlet.java:445) [exist-optional.jar:4.6.0-SNAPSHOT]
    at org.exist.http.servlets.XQueryServlet.doPost(XQueryServlet.java:192) [exist-optional.jar:4.6.0-SNAPSHOT]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [servlet-api-3.1.jar:3.1.0]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [servlet-api-3.1.jar:3.1.0]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566) [jetty-security-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:168) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:78) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.exist.http.urlrewrite.Forward.doRewrite(Forward.java:51) [exist-optional.jar:4.6.0-SNAPSHOT]
    at org.exist.http.urlrewrite.XQueryURLRewrite.doRewrite(XQueryURLRewrite.java:524) [exist-optional.jar:4.6.0-SNAPSHOT]
    at org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:343) [exist-optional.jar:4.6.0-SNAPSHOT]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [servlet-api-3.1.jar:3.1.0]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at de.betterform.agent.web.filter.XFormsFilter.doFilter(XFormsFilter.java:171) [betterform-exist-5.1-SNAPSHOT-20160615.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) [jetty-security-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.Server.handle(Server.java:502) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.exist.xquery.XPathException: err:XPST0003 unexpected token: copy-namespaces (while expecting closing tag for element constructor: foo) [at line 3, column 6]
    at org.exist.xquery.parser.XQueryParser.elementWithAttributes(XQueryParser.java:13776) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.elementConstructor(XQueryParser.java:12527) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.directConstructor(XQueryParser.java:10920) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.primaryExpr(XQueryParser.java:9187) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.postfixExpr(XQueryParser.java:8479) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.stepExpr(XQueryParser.java:8441) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.relativePathExpr(XQueryParser.java:7973) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.pathExpr(XQueryParser.java:7766) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.valueExpr(XQueryParser.java:7597) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.unaryExpr(XQueryParser.java:7452) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.arrowExpr(XQueryParser.java:6911) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.castExpr(XQueryParser.java:6817) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.castableExpr(XQueryParser.java:6724) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.treatExpr(XQueryParser.java:6632) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.instanceofExpr(XQueryParser.java:6541) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.intersectExceptExpr(XQueryParser.java:7924) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.unionExpr(XQueryParser.java:7178) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.multiplicativeExpr(XQueryParser.java:7113) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.additiveExpr(XQueryParser.java:7064) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.rangeExpr(XQueryParser.java:6985) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.stringConcatExpr(XQueryParser.java:6944) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.comparisonExpr(XQueryParser.java:6273) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.andExpr(XQueryParser.java:6242) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.orExpr(XQueryParser.java:4756) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.exprSingle(XQueryParser.java:4429) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.expr(XQueryParser.java:456) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.queryBody(XQueryParser.java:1067) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.mainModule(XQueryParser.java:812) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.module(XQueryParser.java:593) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.parser.XQueryParser.xpath(XQueryParser.java:500) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.XQuery.compile(XQuery.java:110) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.XQuery.compile(XQuery.java:79) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.xquery.XQuery.compile(XQuery.java:71) ~[exist.jar:4.6.0-SNAPSHOT]
    at org.exist.http.servlets.XQueryServlet.process(XQueryServlet.java:442) ~[exist-optional.jar:4.6.0-SNAPSHOT]
    ... 60 more

Context information

• eXist-db version + Git Revision hash: eXist 4.6.0-SNAPSHOT 9e3461c4e
• Java version: OpenJDK 11.0.1+13
• Operating system: macOS 10.13.6
• 32 or 64 bit: 64 bit
• How is eXist-db installed? clone from GitHub
• Any custom changes in e.g. conf.xml: none

[line-o]

Really nice find, @joewiz .

Weird is the fact that most token strings like boundary-space, base-uri, namespace-node, processing-instruction, descendant-or-self and the like do not raise an error.

[joewiz]

@line-o Thanks! I noticed this when testing https://github.com/eXist-db/eXide/pull/212 with an XSL file that I found in the tei-publisher-docx package; eXide raised an error on the <xsl:copy copy-namespaces="no"> element in https://github.com/transpect/tei2html/blob/master/xsl/tei2html.xsl#L94. I searched the eXist source for copy-namespaces and the only instances were in src/org/exist/xquery/parser. I tried other tokens defined nearby, focusing especially on tokens that contain hyphens; I discovered these other two this way, and while I didn't find any others, I didn't do an exhaustive test of all tokens.

I should add that the problem is not limited to direct element constructors. The error is also raised in computed attribute constructors:

attribute copy-namespaces { "bar" }

The error, again, is:

Cannot compile xquery: exerr:ERROR org.exist.xquery.XPathException: err:XPST0003 unexpected token: copy-namespaces [at line 3, column 11]

However, if you place the attribute name in an enclosed expression, you can avoid the error:

attribute { "copy-namespaces" } { "bar" }

Perhaps this will help identify the source of the problem in the parser.

[adamretter]

So all of the names that you chose have a purpose in XQuery:

1. copy-namespaces is a declaration that can appear in Prolog.
2. empty-sequence is a sequence type declaration.
3. schema-element is a node-kind test.

The parser should only scan for these names in the appropriate place. It looks like we have some ambiguity bug in the parser whereby it is scanning/preventing more tokens that it should in the wrong contexts.