Exist: WeakLazyStripes.drainClearedReferences kills performance

[wolfgangmm]

What is the problem

I was testing latest develop by creating a collection with roughly 30000 documents. Running some queries unrelated to the documents stored, I found query performance to break down on many expressions rooted in a fn:collection call. Sometimes performance is good (< 100ms), but if you repeat the same expression, it will break down completely (> 3s). This renders the version largely unusable for mid-sized collections.

YourKit profiler shows that nearly 80% of the query time are spent in WeakLazyStripes.drainClearedReferences:

Describe how to reproduce or add a test

Running a count(collection("/db")) in the Java admin client should be sufficient to see the effect, but you need a sufficiently large number of documents stored in a subcollection. The data size does not seem to matter, just the number of docs.

My original query was:

xquery version "3.1";

declare namespace tei="http://www.tei-c.org/ns/1.0";
declare namespace expath="http://expath.org/ns/pkg";

collection(repo:get-root())//expath:package[@name = "http://exist-db.org/apps/eXide"]

Context information

Please always add the following information

• eXist version 5.0.0-SNAPSHOT (8b4e94b75)
• Java version "1.8.0_162"
• MacOS 10.14.3

[wolfgangmm]

Note: commenting out the call to drainClearedReferences results in performance to improve to 100 to 200ms for each execution.

[joewiz]

I can also confirm that commenting out https://github.com/eXist-db/exist/blob/develop/exist-core/src/main/java/org/exist/util/WeakLazyStripes.java#L110-L113 speeds up queries on collections with 20,000 documents that seemed to drag.

[adamretter]

For reference, running this simple query a couple times is enough to reproduce the issue for me:

for $i in (1 to 30000) return
    doc("/db/doc" || $i || ".xml")