Realm-java: Accelarate Realm transformer

[beeender]

We got some complains (http://stackoverflow.com/questions/37928953/realm-gradle-tasks) about the speed of transformers and there is always space to improve it, like:

• Replace slow groovy with kotlin
• ...

[cmelchior]

Is Groovy really slower than Kotlin? It is all byte code under the hood, which in turn is dex. I think we should switch for other reasons though, and there are probably room for improvement in any case.

[emanuelez]

I don't think switching to Groovy would improve performance a lot, simply because most of the time is spent in Javassist. A more promising change would probably be to switch to ASM from Javassist.

[cmelchior]

There is also some things we need to investigate from this: https://twitter.com/ebtokyo/status/808816830118653952

• When/how is the transformer triggered. Is it possible to be more clever about which files are processed.
• How does incremental builds effect this.
• How/when is the annotation processor triggered. Especially, why is it triggered if there is no code changes. I would assume the gradle build system could easily detect this case.

[emanuelez]

The Transform API has support for incremental builds which we're not using right now so there's space for improvements.
We don't have much control over when the AP is triggered. In general, I think it needs to run on every build because the generated classes might use or be indirectly used by the classes being changed.

[eboudrant]

@cmelchior thanks for the followup, I just wanted to update you, the slowness I reported on Twitter about Gradle was not because of Realm, it was another problem (our BuildConfig modified for each build). If we could have incremental builds support for Realm code-gen via Transform API that would be great.

[beeender]

Complain from SO http://stackoverflow.com/questions/42208219/android-build-time-is-too-long-due-to-more-tables-in-realm-database

[bmeike]

Another request: https://stackoverflow.com/questions/45318711/dont-try-to-rebuild-realm-model-every-time-run-android-build/45335850#45335850

[nateridderman]

Incremental builds would be huge. When I change a line of Java code in a class unrelated to Realm, then Realm shouldn't have to transform everything again. Based on profiling my Gradle build, this would save me at least 20 seconds out of a 60 second build for the transformClassesWithRealmTransformer task, and it seems like Realm is also contributing towards the compileJavaWithJavac task.

[meierjan]

Any reasons why this task can take up to 38 secs?

:app:transformClassesWithRealmTransformerForXDebug | 37.690s

using "io.realm:realm-gradle-plugin:3.3.2"

[thorbenprimke]

@beeender @cmelchior Any update on this? It's one of the longest gradle tasks when building our app (@pinterest). Even just improving it for incremental builds would be huge.

[whataa]

Several modules of our project used realm, the compile time has been steadily increasing linearly with the number of modules. But your team did not make any new progress on the issue in more than a year, any specific schedule on this?

[AamirAbro]

have been hit by realm slow compilations issues too. transformClassesWithRealm some times take more than 30 secs.
:app:transformClassesWithRealmTransformerForDebug | 38.287s

[muthuraj57]

After changing a line in layout xml or anything not even remotely connected to realm still triggers this task which takes more than half of my total build time.

:app:transformClassesWithRealmTransformerForDebug | 1m 18s

[cmelchior]

We have added support for incremental builds in the Realm transformer and would really appreciate if you can try this out in real projects since it changes the logic about which files are being touched and also in order to get input about the improvement on real-world projects.

We have released a special SNAPSHOT build that can be used to test this:

buildscript {
    repositories {
        jcenter()
        google()
        maven {
            url 'http://oss.jfrog.org/artifactory/oss-snapshot-local'
        }
    }
    dependencies {
        classpath "io.realm:realm-gradle-plugin:5.2.0-transformer-SNAPSHOT"
    }
}

allprojects {
    repositories {
        jcenter()
        google()
        maven {
            url 'http://oss.jfrog.org/artifactory/oss-snapshot-local'
        }
    }
}

The debug log will output the amount of time used inside the transformer

// Old logger
./gradlew assemble --debug | grep "Realm Transform time:"

// New logger
./gradlew assemble --debug | grep "realm-stopwatch"

Note, there are edge cases when ignored fields are being changed (marked as either @Ignore, static or transient ). Hitting these cases will cause the incremental build to not do the right thing. These cases are described in the issue: https://github.com/realm/realm-java/pull/5925.

[thorbenprimke]

@cmelchior - thanks for the update. I'll get that patched in on Monday and will report back.

[thorbenprimke]

@cmelchior - looks like it nicely improved things. 💯 🎆 👋 👏

Before:
[initial build] 09:36:26.565 [DEBUG] [realm-logger] Realm Transform time: 16696 milliseconds
[java change] 09:39:19.969 [DEBUG] [realm-logger] Realm Transform time: 21049 milliseconds
[java revert] 09:42:22.689 [DEBUG] [realm-logger] Realm Transform time: 22807 milliseconds

After:
[initial build]
09:57:49.286 [DEBUG] [realm-stopwatch] Prepare output classes: 15 ms.
09:57:49.380 [DEBUG] [realm-stopwatch] Prepare referenced classes: 94 ms.
09:57:49.438 [DEBUG] [realm-stopwatch] Mark mediators as transformed: 57 ms.
09:57:49.460 [DEBUG] [realm-stopwatch] Transform model classes: 21 ms.
09:57:50.384 [DEBUG] [realm-stopwatch] Transform references to model fields: 923 ms.
09:57:50.397 [DEBUG] [realm-stopwatch] Copy resource files: 13 ms.
09:57:50.397 [DEBUG] [realm-stopwatch] Realm Transform time: 1126 ms.
10:00:23.837 [DEBUG] [realm-stopwatch] Prepare output classes: 499 ms.
10:00:24.473 [DEBUG] [realm-stopwatch] Prepare referenced classes: 636 ms.
10:00:24.477 [DEBUG] [realm-stopwatch] Mark mediators as transformed: 3 ms.
10:00:24.477 [DEBUG] [realm-stopwatch] Transform model classes: 0 ms.
10:00:40.175 [DEBUG] [realm-stopwatch] Transform references to model fields: 15697 ms.
10:00:40.666 [DEBUG] [realm-stopwatch] Copy resource files: 491 ms.
10:00:40.666 [DEBUG] [realm-stopwatch] Realm Transform time: 17328 ms.

[java change]
11:23:58.204 [DEBUG] [realm-stopwatch] Prepare output classes: 5 ms.
11:23:58.390 [DEBUG] [realm-stopwatch] Prepare referenced classes: 185 ms.
11:23:58.390 [DEBUG] [realm-stopwatch] Mark mediators as transformed: 0 ms.
11:23:58.390 [DEBUG] [realm-stopwatch] Transform model classes: 0 ms.
11:23:58.477 [DEBUG] [realm-stopwatch] Transform references to model fields: 86 ms.
11:23:59.046 [DEBUG] [realm-stopwatch] Copy resource files: 569 ms.
11:23:59.046 [DEBUG] [realm-stopwatch] Realm Transform time: 846 ms.

[java revert]
11:25:50.004 [DEBUG] [realm-stopwatch] Prepare output classes: 0 ms.
11:25:50.181 [DEBUG] [realm-stopwatch] Prepare referenced classes: 177 ms.
11:25:50.182 [DEBUG] [realm-stopwatch] Mark mediators as transformed: 0 ms.
11:25:50.182 [DEBUG] [realm-stopwatch] Transform model classes: 0 ms.
11:25:50.214 [DEBUG] [realm-stopwatch] Transform references to model fields: 31 ms.
11:25:50.706 [DEBUG] [realm-stopwatch] Copy resource files: 492 ms.
11:25:50.706 [DEBUG] [realm-stopwatch] Realm Transform time: 702 ms.

Test steps:
Do a gradle clean, trigger a build (initial build), then make a java change in the main module (second build) and then revert that change (third build).

Our realm model classes are in a separate gradle module. I think that's why in the _java change_ and _java revert_ steps, only show the output once because the other module didn't change.

[muthuraj57]

@cmelchior This snapshot improved my build time considerably. Previously it took more than 3 minutes to incremental builds. Now it is more or less one minute. Thanks a lot.

[cmelchior]

@muthuraj57 Thank you for reporting back. Would it be possible to post the log numbers for your build as well? A minute still sounds like a lot for incremental builds, so would be interesting to see where the time is actually being spent

[thorbenprimke]

@cmelchior - do you know when you will release the next version (5.1.1 or 5.2) that will include the new transformer improvements?

[cmelchior]

5.2.0 should be available in the next 24 hours.