Exist: NPE for nesting for with group and order

[ahenket]

eXist-db 4.3.1 macOS Mojave.

I'm trying to get to the core of the problem but so far I cannot get closer than this query fragment:

declare variable $colDecorCache     := collection('/db/apps/decor/cache');
declare variable $colDecorData      := collection('/db/apps/decor/data');

declare function local:getTemplateById($id as xs:string, $flexibility as xs:string?, $prefix as xs:string, $version as xs:string?) {

let $internalrepositories       := $colDecorData/decor[project/@prefix = $prefix][1]
let $buildingBlockRepositories  := $internalrepositories/project/buildingBlockRepository
let $x                          := local:getTemplateById($prefix, $id, $prefix, $buildingBlockRepositories, ())

return ()
};

declare %private function local:getTemplateById($basePrefix as xs:string, $id as xs:string, $prefix as xs:string, $externalrepositorylist as element()*, $bbrList as element()*) as element()* {
    for $repository in $externalrepositorylist
    let $repourl                := $repository/@url
    let $repoident              := $repository/@ident
    return
        <repositoryTemplateList>
        {
                let $cachedProject  := $colDecorCache//cacheme[@bbrident = $repoident][@bbrurl = $repourl]/decor
                return ()
        }
        </repositoryTemplateList>
};

<x>
{
    (: Don't show links for templates in versioned projects :)
    for $projectTemplateSet in $colDecorData/decor[@repository = 'true']//template[@ref]
    let $prefix                 := $projectTemplateSet/ancestor::decor/project/@prefix
    let $tid                    := $projectTemplateSet/@id | $projectTemplateSet/@ref
    group by $prefix, $tid
    order by $prefix, $tid
    return (
        for $template in $projectTemplateSet
        return (
            if ($template[@ref]) then (
                local:getTemplateById($template/@ref, 'dynamic', $prefix, ())//template[@id]
            ) else ()
        )
    )
}
</x>

This code yields a NullPointException from the exist.log no stack trace:

2018-08-03 17:30:41,336 [qtp1226557879-3531] INFO  (XQueryServlet.java [init]:158) - encoding = UTF-8 
2018-08-03 17:30:41,371 [qtp1226557879-3531] ERROR (XQueryServlet.java [process]:552) - null 
java.lang.NullPointerException: null

When I remove the line "order by $prefix, $tid" the xquery finishes normally. When I remove the line "let $cachedProject := $colDecorCache//cacheme[@bbrident = $repoident][@bbrurl = $repourl]/decor" the xquery finishes normally.

The collection that $colDecorCache operates on has combined range indexes configured.

This xquery fragment is hard to replay anywhere else because of data requirements. I still hope we can get to the bottom of it.

[duncdrum]

hmm feels like a blast from the past #845, however i can run the code of the old issue without problems.

@ahenket to reproduce this we would need a minimal sample data, and the xconf, any chance you could expand your report?

[dizzzz]

I remembered the original issue, and the fix. A new NPE would be bad :(

[ahenket]

repro-github-2099.zip

I created a repro package. Setup is described in the repro.xquery, but for convenience I'll copy that here:

• Copy cache-collection.xconf as /db/system/config/db/apps/decor/cache/collection.xconf

• Copy data-collection.xconf as /db/system/config/db/apps/decor/data/collection.xonf

• Copy demo1-decor.xml into collection /db/apps/decor/data

• Copy cache.xml into collection /db/apps/decor/cache

• Run repro.query

[ahenket]

Ok, so I've created a simpler reproduction based on a single file, stripped to the max. Rename to xquery and run it against your server. It's going create /db/apps/tmp with one small resource, and index that using a single combined index of 2 fields.

You may solve the NullPointer by:

• Removing the index
• Removing the order by clause all the way to the bottom
• Replace the function call with the function contents
• Remove the <project/> element in the data

And probably a couple more other ways. None of those is a satisfactory workaround.

reproductionForNullPointerCombinedIndex.txt

declare variable $strDecorData  := xmldb:create-collection(repo:get-root(), 'tmp');

declare %private function local:scenarioBasics($scenario as element(), $ds-id as xs:string?, $ds-ed as xs:string?) as element(transaction)* {
    if (empty($ds-id)) then ($scenario/transaction) else (
        $scenario/transaction[.//representingTemplate[@sourceDataset = $ds-id][@sourceDatasetFlexibility = $ds-ed]]
    )
};

let $data           :=
<decor>
    <project id="2.16.840.1.113883.3.1937.99.62.3" prefix="demo1-" defaultLanguage="en-US"/>
    <datasets>
        <dataset id="2.16.840.1.113883.3.1937.99.62.3.1.1" statusCode="draft" effectiveDate="2012-05-30T11:32:36"/>
    </datasets>
    <scenarios>
        <scenario id="2.16.840.1.113883.3.1937.99.62.3.3.1" statusCode="draft" effectiveDate="2012-09-05T16:59:35">
            <transaction>
                <transaction>
                    <representingTemplate/>
                </transaction>
            </transaction>
        </scenario>
    </scenarios>
</decor>
let $indexconf      :=
<collection xmlns="http://exist-db.org/collection-config/1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <index>
        <fulltext default="none" attributes="false"/>
        <range>
            <create qname="representingTemplate">
                <field name="representingTemplate-sourceDataset" match="@sourceDataset" type="xs:string"/>
                <field name="representingTemplate-sourceDatasetFlexibility" match="@sourceDatasetFlexibility" type="xs:string"/>
            </create>
        </range>
    </index>
</collection>

let $mkindexdir     := xmldb:create-collection('/db/system/config', $strDecorData)

let $storeindex     := xmldb:store($mkindexdir, 'collection.xconf', $indexconf)
let $storedata      := xmldb:store($strDecorData, 'demo1-decor.xml', $data)
let $reindex        := xmldb:reindex($strDecorData)

let $decor          := collection($strDecorData)/decor[project/@prefix = 'demo1-']
let $ds-id          := ($decor/datasets/dataset/@id)[1]
let $ds-ed          := ($decor/datasets/dataset/@effectiveDate)[1]

return
<scenarios>
{
    for $scenario in $decor/scenarios/scenario
    order by $scenario/name[1]/lower-case(text()[1])
    return
        local:scenarioBasics($scenario, $ds-id, $ds-ed)
}
</scenarios>

[joewiz]

I can reproduce the NPE on 4.3.1 (release) on macOS 10.13.6 with Java 10.0.2+13. The beginning of exist.log:

2018-09-04 17:17:13,673 [qtp323437281-13217] ERROR (XQueryServlet.java [process]:552) - null 
java.lang.NullPointerException: null
    at org.exist.xquery.InternalFunctionCall.analyze(InternalFunctionCall.java:80) ~[exist.jar:4.3.1]
    at org.exist.xquery.modules.range.OptimizeFieldPragma.eval(OptimizeFieldPragma.java:64) ~[exist-index-range.jar:4.3.1]
    at org.exist.xquery.ExtensionExpression.eval(ExtensionExpression.java:70) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:450) ~[exist.jar:4.3.1]
...

[dizzzz]

Thnx! A stacktrace helps....

[ahenket]

Just in case, from the xmlrpc.log (I ran it through Oxygen)

2018-09-05 10:24:47,586 [qtp1639290814-38] ERROR (XmlRpcErrorLogger.java [log]:36) - Failed to invoke method queryP in class org.exist.xmlrpc.RpcConnection: null 
org.apache.xmlrpc.common.XmlRpcInvocationException: Failed to invoke method queryP in class org.exist.xmlrpc.RpcConnection: null
    at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.invoke(ReflectiveXmlRpcHandler.java:129) ~[xmlrpc-server-3.1.3.jar:3.1.3]
...
Caused by: java.lang.NullPointerException
    at org.exist.xquery.InternalFunctionCall.analyze(InternalFunctionCall.java:80) ~[exist.jar:4.3.1]
    at org.exist.xquery.modules.range.OptimizeFieldPragma.eval(OptimizeFieldPragma.java:64) ~[exist-index-range.jar:4.3.1]
    at org.exist.xquery.ExtensionExpression.eval(ExtensionExpression.java:70) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:450) ~[exist.jar:4.3.1]
    at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326) ~[exist.jar:4.3.1]
    at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:256) ~[exist.jar:4.3.1]
    at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:243) ~[exist.jar:4.3.1]
    at org.exist.xquery.LocationStep.eval(LocationStep.java:474) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) ~[exist.jar:4.3.1]
    at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:102) ~[exist.jar:4.3.1]
    at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:162) ~[exist.jar:4.3.1]
    at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) ~[exist.jar:4.3.1]
    at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:302) ~[exist.jar:4.3.1]
    at org.exist.xquery.FunctionCall.eval(FunctionCall.java:223) ~[exist.jar:4.3.1]
    at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) ~[exist.jar:4.3.1]
    at org.exist.xquery.OrderByClause.eval(OrderByClause.java:60) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.ForExpr.processItem(ForExpr.java:251) ~[exist.jar:4.3.1]
    at org.exist.xquery.ForExpr.eval(ForExpr.java:189) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.EnclosedExpr.eval(EnclosedExpr.java:85) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.ElementConstructor.eval(ElementConstructor.java:329) ~[exist.jar:4.3.1]
    at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276) ~[exist.jar:4.3.1]
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) ~[exist.jar:4.3.1]
    at org.exist.xquery.XQuery.execute(XQuery.java:261) ~[exist.jar:4.3.1]
    at org.exist.xquery.XQuery.execute(XQuery.java:185) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.doQuery(RpcConnection.java:252) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.lambda$null$52(RpcConnection.java:1739) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.function.XmlRpcCompiledXQueryFunction.apply(XmlRpcCompiledXQueryFunction.java:41) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.lambda$compileQuery$123(RpcConnection.java:3672) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.lambda$queryP$53(RpcConnection.java:1739) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.function.XmlRpcFunction.apply(XmlRpcFunction.java:45) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.withDb(RpcConnection.java:3928) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.withDb(RpcConnection.java:3912) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.queryP(RpcConnection.java:1711) ~[exist.jar:4.3.1]
    at org.exist.xmlrpc.RpcConnection.queryP(RpcConnection.java:3548) ~[exist.jar:4.3.1]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
    at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.invoke(ReflectiveXmlRpcHandler.java:115) ~[xmlrpc-server-3.1.3.jar:3.1.3]
    ... 61 more

[ahenket]

I'm encountering more places in our code where I need a rewrite to something without order or a revert to a different index. This makes the issue blocking for migration to 4.3.1, and we would preferably need 4.3.2. Anything that can be said about this?

[xatapult]

I've looked into this bug as well, and its getting weirder and weirder. It looks like a total memory/pointer fuck-up.

The situation can be summarized as follows:

• There is some data with a combined index on some attributes
• You access this data with a predicate on the indexed attributes inside a function that is called from a for-loop with an order-by clause (I have tried various order-by clauses and it doesn't seem to be what is ordered-by on, even ordering by on a constant (as useless as it is) has the same effect)
• Result: Null pointer exception

Some other observations:

• If you put the code from the function inside the for loop with the order-by clause, all is good, no probs.
• If you disable the indexes the problem stays until you restart eXist.
• If you subsequently enable the indexes again the problem seems to be gone until you restart eXist again

[xatapult]

Here is some more minimal code that also gives this error:

declare variable $col    := collection('/db/apps/decor-test/cache');

declare function local:try-it()
{
  $col//cacheme[@bbrident = 'ad1bbr-'][@bbrurl = 'http://art-decor.org/decor/services/']
};

<test>
{ 
  for $x in (1,2,3)
  order by $x
  return
    local:try-it()
}
</test>

The collection.xconf for /db/apps/decor-test/cache reads:

<collection xmlns="http://exist-db.org/collection-config/1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <index>
    <fulltext default="none" attributes="false"/>
    <range>
      <create qname="cacheme">
        <field name="cacheme-bbrurl" match="@bbrurl" type="xs:string"/>
        <field name="cacheme-bbrident" match="@bbrident" type="xs:string"/>
      </create>
    </range>
  </index>
</collection>

Commenting-out the order by clause or putting the code from the function into the main for loop makes the bug disappear. As we say in The Netherlands: Can anybody make chocolate from that?

[wolfgangmm]

Likely identical to the issue fixed by #2221, but need to confirm when github allows me again ...

[joewiz]

@wolfgangmm says this is most likely the same as https://github.com/eXist-db/exist/pull/2221.

[wolfgangmm]

Confirm: PR #2221 will fix this issue as well.

[wolfgangmm]

Should be fixed by #2221. Closing for now.