Exist: util:expand alters node type of attribute and document nodes

[joewiz]

What is the problem

The util:expand() function alters the node types of attribute and document nodes instead of preserving their original node type.

What did you expect

The function documentation describes the function as follows:

Creates an in-memory copy of the passed node set, using the specified serialization options.
$node as node()* The node(s) to create in-memory copies of.
Returns: node()* the results

Thus, this function should return an in-memory copy of the original node set—and the node type of each passed node should be preserved in the output.

Describe how to reproduce or add a test

The following query illustrates the problem:

xquery version "3.1";

(: these correctly return true :)
util:expand( comment { "x" } ) instance of comment(),
util:expand( processing-instruction x { "" } ) instance of processing-instruction(),
util:expand( text { "x" } ) instance of text(),
util:expand( element x {()} ) instance of element(),

(: these incorrectly return false :)
util:expand( attribute x {""} ) instance of attribute(),
util:expand( document { element x {()} } ) instance of document-node(),

(: these incorrectly return true :)
util:expand( attribute x {""} ) instance of text(),
util:expand( document { element x {()} } ) instance of element()

The results shows that attribute nodes are being transformed into text nodes, and document nodes are being transformed into element nodes.

The following XQSuite (attached here: util-expand-test.xql.zip) also demonstrates the problem.

xquery version "3.1";

module namespace ue="http://joewiz.org/ns/test/exist/util/expand";

declare namespace test="http://exist-db.org/xquery/xqsuite";

declare
    %test:assertTrue
function ue:attribute() as xs:boolean {
    util:expand( attribute foo { "" } ) instance of attribute()
};

declare
    %test:assertTrue
function ue:comment() as xs:boolean {
    util:expand( comment { "foo" } ) instance of comment()
};

declare
    %test:assertTrue
function ue:document() as xs:boolean {
    util:expand( document { element foo {()} } ) instance of document-node()
};

declare
    %test:assertTrue
function ue:element() as xs:boolean {
    util:expand( element foo {()} ) instance of element()
};

declare
    %test:assertTrue
function ue:pi() as xs:boolean {
    util:expand( processing-instruction foo { "" } ) instance of processing-instruction()
};

declare
    %test:assertTrue
function ue:text() as xs:boolean {
    util:expand( text { "foo" } ) instance of text()
};

The results, as of now:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite package="http://joewiz.org/ns/test/exist/util/expand"
        timestamp="2017-09-26T10:51:00.805-04:00" failures="2" pending="0" tests="6" time="PT0.012S">
        <testcase name="attribute" class="ue:attribute">
            <failure message="assertExists failed." type="failure-error-code-1"/>
            <output>false</output>
        </testcase>
        <testcase name="comment" class="ue:comment"/>
        <testcase name="document" class="ue:document">
            <failure message="assertExists failed." type="failure-error-code-1"/>
            <output>false</output>
        </testcase>
        <testcase name="element" class="ue:element"/>
        <testcase name="pi" class="ue:pi"/>
        <testcase name="text" class="ue:text"/>
    </testsuite>
</testsuites>

Context information

• eXist-db version + Git Revision hash: 3.5.0-SNAPSHOT+201709261032 bbc8f8d08
• Java version: 1.8.0_144
• Operating system: macOS 10.13
• 32 or 64 bit: 64 bit
• Any custom changes in e.g. conf.xml: none

[joewiz]

Another way to demonstrate the failing behavior is to open eXide, and with the default serialization method (Adaptive), submit a query that returns an attribute (e.g., attribute foo { "bar" }. Instead of foo="bar", you'll get "bar". Uncheck Highlight matches and submit your query again, and you'll get the expected foo="bar". This happens because the highlight matches feature is implemented with util:expand. Unchecking it prevents the use of this function.

(This is how I discovered the problem.)

[joewiz]

This is an expanded version of #1486, which only reported the problem with attributes, so I'll close that one.

[joewiz]

Here's another weird variation on this report. Take the following query:

xquery version "3.1";

util:expand(
    document { 
        <?foo?>,
        <bar/>
    }
)

It returns a single item:

<?foo?>

Somehow util:expand() causes the <bar/> element to go poof!

[joewiz]

Similarly:

util:expand(
    document { 
        <?foo?>,
        <bar/>
    }
) instance of processing-instruction()

returns true()!

[joewiz]

On the other hand:

util:expand(
    (
        <?foo?>,
        <bar/>
    )
)

correctly returns the sequence, (<?foo?>, <bar/>). Something about the util:expand() function's handling of the outer document-node() is causing this query to go haywire. I think the relevant code is here: https://github.com/eXist-db/exist/blob/develop/src/org/exist/xquery/functions/util/Expand.java#L106-L116.

[joewiz]

And the other failing case involves attributes:

util:expand( attribute foo { "bar" } )

... returns bar

And:

util:expand( attribute foo { "bar" } ) instance of text()

returns true()

[adamretter]

This is likely related to https://github.com/eXist-db/exist/issues/1463

[rvdb]

This reminds me of other issues with documents containing processing instructions or comments outside the document element, as reported in https://github.com/eXist-db/exist/issues/623, where there's a difference when the document is retrieved directly from the database, or first stored in an in-memory copy. Could that be related to the behaviour reported in this issue?

[adamretter]

@rvdb sounds very likely.

[joewiz]

The bugs in the xqsuite test suite here still affect eXist 5.0.0.