Exist: NPE when running util:serialize with doctype parameters

[lagivan]

I'm trying to serialize DITA topic to string with DOCTYPE's added. So the following code block is used:

let $serialization-params := 
(
    "doctype-public=-//D//DTD P-Topic//EN", 
    "doctype-system=http://domain.com/xmetal/schemas/p-topic.dtd"
)
let $topic-string := util:serialize($topic-document, $serialization-params)

It fails with the following NPE:

Caused by: java.lang.NullPointerException
    at org.exist.xquery.functions.util.Serialize.parseSerializationOptions(Serialize.java:194)
    at org.exist.xquery.functions.util.Serialize.eval(Serialize.java:156)
    at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70)
    at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:55)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:142)

According to the code, it splits the string using space character as a delimiter. It looks like the same code is used to parse the serialization parameters that can be provided in declare option exist:serialize statement. However, in case of util:serialize the parameters format is different and I believe the space should not be considered as a delimiter.

I've tried to replace the space with the non-breaking space code   This technique works well in declare option exist:serialize statement, however, with util:serialize it leads to another error:

exerr:ERROR XMLDB reported an exception while storing documentorg.xmldb.api.base.XMLDBException: The XML parser reported a problem: fatal error at (2,39) : An invalid XML character (Unicode: 0xa0) was found in the public identifier. [at line 57, column 22]

Not sure if it makes more sense to allow using space or   or both but at least one of those ways should be working. Any considerations? It would be nice to fix it before 2.2 final release. I'm willing to help fixing but I'd like to hear developers' opinion on this.

[dizzzz]

Checking the code, it should be possible to specify the serialization options as a XML node as well

• the node must be in the http://www.w3.org/2010/xslt-xquery-serialization namespace
• <a value="b"> and <a>b</a> should be supported (source: SerializerUtils.java)

maybe that helps as a work around. I do not see an other way to parse an 'arbitrary string' yet, but a NPE should be avoided under all circumstances.

At least this route should be documented; it is available since 12 feb 2014.

 new FunctionSignature(
                new QName("serialize", UtilModule.NAMESPACE_URI, UtilModule.PREFIX),
                "Returns the Serialized node set passed in parameter $node-set. $parameters contains a " +
                "sequence of zero or more serialization parameters specified as key=value pairs. The " +
                "serialization options are the same as those recognized by \"declare option exist:serialize\". " +
                "The function does NOT automatically inherit the serialization options of the XQuery it is " +
                "called from.",
                new SequenceType[] { 
                    new FunctionParameterSequenceType("node-set", Type.NODE, Cardinality.ZERO_OR_MORE, "The node set to serialize"),
                    new FunctionParameterSequenceType("parameters", Type.ITEM, Cardinality.ZERO_OR_MORE,
                            "The serialization parameters: either a sequence of key=value pairs or an output:serialization-parameters " +
                            "element as defined by the standard fn:serialize function.")
                },
                new FunctionParameterSequenceType("result", Type.STRING, Cardinality.ZERO_OR_ONE, "the string containing the serialized node set.")

@wolfgangmm I'd like to propose to have a map() option as well, that would be a more natural fit for the 2.x generation

https://en.wikibooks.org/wiki/XQuery/eXist_Crib_sheet#Serialization_Options

[dizzzz]

any update?

[lagivan]

@dizzzz Could you clarify how you think serialization options should look like as an XML node?

I've tried this without success getting the same NPE:

let $serialization-params := 
(
    <xslt:doctype-public value="-//D//DTD P-Topic//EN" xmlns:xslt="http://www.w3.org/2010/xslt-xquery-serialization"/>, 
    <xslt:doctype-public value="http://domain.com/xmetal/schemas/p-topic.dtd" xmlns:xslt="http://www.w3.org/2010/xslt-xquery-serialization"/>
)

Could you say what code did you look at when you came up with this suggestion? Because I'm looking now at Serialize.parseSerializationOptions and it's pretty obvious that XML node won't work. The following is done for each serialization parameter from the sequence above:

String params[] = serializeParams.split(" ");
for(String param : params)
{
    String opt[] = Option.parseKeyValuePair(param);
    outputProperties.setProperty(opt[0], opt[1]);
}

The code above expects common properties without spaces.

[dizzzz]

https://github.com/eXist-db/exist/blob/develop/src/org/exist/xquery/util/SerializerUtils.java

[lagivan]

Thanks! I checked an old version of the class indeed. Now the workaround works:

let $serialization-params := 
    <output:serialization-parameters xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization">
        <output:doctype-public value="-//D//DTD P-Topic//EN"/>
        <output:doctype-system value="http://domain.com/xmetal/schemas/p-topic2.dtd"/>
    </output:serialization-parameters>

Still I agree the NPE should be avoided so the ticket is open.

[dizzzz]

Thnx, I did not figure out the exact syntax/structure ; glad it works
For the NPE......... need some time to study, not sure if it will work at all.....

[duncdrum]

@dizzzz is this still relevant, or good to close?

[dizzzz]

We'd need to check the original query....

[duncdrum]

hmm since the OP the handling of non-breaking space in serialization settings has changed, i can't get a reproducible code sample to run based on the OP.

@dizzzz since you've assigned yourself the issue can you please wrap this up by either creating a testcase, or closing as unreproducible.

[lagivan]

I've tested the issue again and it's still actual for eXist-db 4.6.

Again the reproducible code sample:

let $topic-document := doc('/db/apps/myapp/test-topic.dita')/node()
let $serialization-params := 
(
    "doctype-public=-//OASIS//DTD DITA 1.3 Topic//EN", 
    "doctype-system=http://docs.oasis-open.org/dita/dita/v1.3/os/part3-all-inclusive/all-inclusive-grammars/dtd/technicalContent/dtd/topic.dtd"
)
return util:serialize($topic-document, $serialization-params)

The results are empty (expect to see the DITA content), and there is NPE in log file.

Here is the full stacktrace from the log file:

2019-02-26 14:27:40,413 [qtp1092130260-58] ERROR (XQueryServlet.java [process]:552) - null 
java.lang.NullPointerException: null
    at org.exist.xquery.functions.util.Serialize.parseSerializationOptions(Serialize.java:178) ~[exist.jar:4.6.0]
    at org.exist.xquery.functions.util.Serialize.eval(Serialize.java:148) ~[exist.jar:4.6.0]
    at org.exist.xquery.BasicFunction.eval(BasicFunction.java:74) ~[exist.jar:4.6.0]
    at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:41) ~[exist.jar:4.6.0]
    at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.6.0]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.6.0]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.6.0]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.6.0]
    at org.exist.xquery.XQuery.execute(XQuery.java:261) ~[exist.jar:4.6.0]
    at org.exist.xquery.XQuery.execute(XQuery.java:185) ~[exist.jar:4.6.0]
    at org.exist.http.servlets.XQueryServlet.process(XQueryServlet.java:488) [exist-optional.jar:4.6.0]
    at org.exist.http.servlets.XQueryServlet.doPost(XQueryServlet.java:192) [exist-optional.jar:4.6.0]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [servlet-api-3.1.jar:3.1.0]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [servlet-api-3.1.jar:3.1.0]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566) [jetty-security-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:168) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:78) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.exist.http.urlrewrite.Forward.doRewrite(Forward.java:51) [exist-optional.jar:4.6.0]
    at org.exist.http.urlrewrite.XQueryURLRewrite.doRewrite(XQueryURLRewrite.java:524) [exist-optional.jar:4.6.0]
    at org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:343) [exist-optional.jar:4.6.0]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [servlet-api-3.1.jar:3.1.0]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:859) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) [jetty-security-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.Server.handle(Server.java:502) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

[joewiz]

Please try the following version which uses the fn:serialize function and related serialization options drawn from the XQuery 3.1 spec:

xquery version "3.1";

let $topic-document := doc('/db/apps/myapp/test-topic.dita')/node()
let $serialization-params := 
map {
    "doctype-public": "-//OASIS//DTD DITA 1.3 Topic//EN", 
    "doctype-system": "http://docs.oasis-open.org/dita/dita/v1.3/os/part3-all-inclusive/all-inclusive-grammars/dtd/technicalContent/dtd/topic.dtd"
}
return fn:serialize($topic-document, $serialization-params)

[lagivan]

@joewiz it works! So it solves the issue indeed. However, the original function util:serialize is still not working. So it should be better removed at all and the documentation should be updated not to mislead users:
http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/util&location=java:org.exist.xquery.functions.util.UtilModule

[adamretter]

Looks like a regression. This probably crept in when I added the cdata-section-elements option. I will take a look... seems we had no tests here previously :-(

[dizzzz]

The issue is..... the spaces in "=-//OASIS//DTD DITA 1.3 Topic//EN" - it looks like it never worked...