Jooq: Multithread deadlock: org.jooq.impl.AbstractField references subclass org.jooq.impl.Val in static initializer

[daniel-huss]

Expected behavior

org.jooq.impl.Val can always be initialized when another thread is concurrently initializing org.jooq.impl.AbstractField.

Actual behavior

Sporadic class initialization deadlock, see thread dump:

"ForkJoinPool-1-worker-3" #26 daemon prio=5 os_prio=0 cpu=46.88ms elapsed=188.28s tid=0x000001d9f0b30870 nid=0x2b54 in Object.wait()  [0x0000007b404fc000]
   java.lang.Thread.State: RUNNABLE
        at org.jooq.impl.DSL.val(DSL.java:24410)
        - waiting on the Class initialization monitor for org.jooq.impl.AbstractField
        at org.jooq.impl.DSL.val(DSL.java:23960)
        at org.jooq.impl.DSL.any(DSL.java:9602)
        at company.app.infrastructure.util.jobprocessing.SqlJobRepository.<init>(SqlJobRepository.kt:21)
        at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:183)

   Locked ownable synchronizers:
        - None

"ForkJoinPool-1-worker-1" #27 daemon prio=5 os_prio=0 cpu=1296.88ms elapsed=188.27s tid=0x000001d9f0b99570 nid=0x5f68 in Object.wait()  [0x0000007b405fb000]
   java.lang.Thread.State: RUNNABLE
        at org.jooq.impl.DSL.val(DSL.java:24410)
        - waiting on the Class initialization monitor for org.jooq.impl.Val
        at org.jooq.impl.DSL.val(DSL.java:23960)
        at org.jooq.impl.DSL.inline(DSL.java:23036)
        at org.jooq.impl.Tools.inline(Tools.java:1973)
        at org.jooq.impl.AbstractField.<clinit>(AbstractField.java:117)
        at org.jooq.impl.AbstractTable.createField(AbstractTable.java:790)
        at org.jooq.impl.AbstractTable.createField(AbstractTable.java:749)
        at company.app.infrastructure.util.persistence.jooq.tables.FlowTable.<init>(FlowTable.java:60)
        at company.app.infrastructure.util.persistence.jooq.tables.FlowTable.<init>(FlowTable.java:142)
        at company.app.infrastructure.util.persistence.jooq.tables.FlowTable.<init>(FlowTable.java:167)
        at company.app.infrastructure.util.persistence.jooq.tables.FlowTable.<clinit>(FlowTable.java:47)
        at company.app.infrastructure.flow.SqlFlowRepository.acquireAnyExecutableFlow(SqlFlowRepository.kt:108)
        at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:183)

   Locked ownable synchronizers:
        - None

What seems to be happening:

• Thread A wants to init class AbstractField via the class initializer of FlowTable.
• Thread A enters static initializer, then Tools.inline, then DSL.inline, then DSL.val.
• Thread B wants to init class Val.
• Thread B wants to init Val superclass AbstractField, which is locked, so it waits for thread A.
• Thread A executes statement new Val() within DSL.val, which blocks because B has the lock
• Thread B blocks because initialization of superclass AbstractField blocks in Thread A -> deadlock

(Where Thread A is ForkJoinPool-1-worker-1 in my example)

So essentially the situation boils down to:

    class AbstractField {
        private static Val val = new Val();
    }

    class Val extends AbstractField { }

Steps to reproduce the problem

I don't know how to reliably reproduce this concurrency issue. It happens in about 1 of 20 builds if I randomize test case order and use only two threads. With your expertise I'm sure the above explanation is sufficient, though 😬

Versions

• jOOQ: 3.14.4
• Java:

openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

• Database (include vendor): PostgreSQL 11 (probably doesn't matter)
• OS: Windows
• JDBC Driver (include name if inofficial driver): none

[lukaseder]

Thanks a lot for your report.

These things can be difficult to reproduce, but in this particular case, I think there's no need. The stack traces and IntelliJ inspection message is sufficient. I'll try to fix this right away.

Without reliable reproduction, I won't be able to write a test, so I will rely on your feedback, possibly prior to the actual release of the fix. Would this be possible? If you're licensed, snapshot builds are available from here (once the fix is included): https://www.jooq.org/download/versions. If you're not licensed, then you can build a snapshot version yourself, from this branch: https://github.com/jOOQ/jOOQ/tree/version-3.14.0-branch

[lukaseder]

The change / regression was introduced here: https://github.com/jOOQ/jOOQ/issues/8495, https://github.com/jOOQ/jOOQ/commit/4b14717dde753ce2a43483aef2a35a04a73aa44a#diff-7e2b9030c96499c6642c0acea805116a847e7980544e7c4ffdaece92a8b2e0b9R117

[lukaseder]

We don't have to eagerly initialise those values, so a nested class could be sufficient to delay initialisation and thus work around this problem.

[lukaseder]

Fixed for jOOQ 3.15.0 and 3.14.5 (#11212)

[Stephan202]

Some of our teams hit this issue as well. We're attempting to work around it by calling this method very early after application startup:

    /**
     * Attempts to load a jOOQ class if available, thereby avoiding a potential class initialization
     * deadlock.
     *
     * @see <a href="https://github.com/jOOQ/jOOQ/issues/11200">jOOQ/jOOQ#11200</a>
     */
    private static void preventJooqClassInitializationBug() {
        try {
            Class.forName("org.jooq.impl.AbstractField");
        } catch (ClassNotFoundException ignored) {
            // This is fine; not all applications use jOOQ.
        }
    }

(We _think_ this works, but time will tell. Such is the nature of multi-threading issues :smile:.)

Thought I'd share it here in case other people observe this issue. (Though a 3.14.5 release would be nice :crossed_fingers:)

[lukaseder]

(Though a 3.14.5 release would be nice 🤞)

So, I was going to release this version, but Sonatype wouldn't let me: https://twitter.com/sonatype_ops/status/1352274029047021568

The commercial distributions and the OSS Edition ZIP file will be released today, available from https://www.jooq.org/download/versions in a bit.

The Maven Central distribution will have to wait for some more time, until those issues are fixed.

[Stephan202]

The Maven Central distribution will have to wait for some more time, until those issues are fixed.

Tnx for the update! Let's hope they manage to resolve the issue. (I don't often use it, but this emoji seems applicable once more: :crossed_fingers:.)