There is an approach for Gradle 4.6 to allow incremental compilation when annotation processors are used: https://github.com/gradle/gradle/issues/1320
Is this approach sufficient for Lombok? Does it require a lot of work on Lombok side?
The current state prohibits the use of lombok in big gradle projects.
Lombok is a bit of a special case, as it uses internal APIs to change existing types. This is not officially allowed by the Processor API. Given how incremental annotation processing in Gradle currently works (to be released in 4.7), Lombok would work fine. But there is no guarantee that this would keep working in the future if we decide to take more control of the round environment for instance.
I guess this is the trade-off that Lombok has always taken - Rely on compiler internals in order to provide the most concise code.
I've asked gradle team about this topic. They said lombok need changes.
Lombok should be changed to properly separate it’s annotations from its processors to avoid this annoyance for users. Dagger (shown in our docs) does this correctly, they have a library jar and a processor jar.
I copy message here, but it's better to go and read all thread
@nsiniakevich That change would also be great but is unrelated to incremental compilation. Can you please open a separate issue?
I've taken a closer look at Lombok. Its usage of Trees.instance(procEnv) would fail if used with Gradle's incremental compiler, because Gradle decorates the processing environment. Of course Lombok could extract the original environment if the devs are ready to take on that additional workaround.
@rzwitserloot what do you think about this workaround?
@oehme I've just wanted to open issue, but it's already here https://github.com/rzwitserloot/lombok/issues/1611 ;)
@oehme I'm gonna try to interpret this one:
Okay, so, Lombok should call Trees.instance(procEnv) as normal, but, before proceeding, call .getClass().toString() to check if it so happens to be gradle's wrapper around the Trees class. If it is, we use reflection to fetch the javac Trees impl, and we carry on with whatever we were doing, using that reflection-obtained original environment.
Is that all it's gonna take? Because – that sounds rather simple to do. Can you point me at the wrapper code, or, better yet, tell me the fully qualified name of the wrapper + the method/field we can use to extract the original environment?
@rzwitserloot You'll also need to register Lombok in one of Gradle's incremental categories. I did take a look at the annotations and it seems like they all look at each annotated element in isolation, making Lombok an isolating processor (the most efficient category). Also as far as I can see you never generated code, but always mutate the original AST. Is that correct?
The delegating wrapper is here. If you decide to go forward with this, we can add a smoke test to Gradle to ensure we don't break you without advance notice.
Since Gradle team says Lombok should be changed to properly separate it’s annotations from its processors to avoid this annoyance for users. but https://github.com/rzwitserloot/lombok/issues/1611 won't be fixed (which is really bad in my opinion), is supporting Incremental annotation processing still possible for Lombok?
@eximius313 as mentioned above already, this is unrelated to incremental compilation.
@oehme or anyone else familiar with this new Gradle feature: In order to take advantage of this, do we need to do anything special? For example, is doing compile <lombok:1.16.22> with Gradle 4.6+ sufficient for incremental annotation processing now?
It's Gradle 4.7+ and you'll need a lombok version that includes the change. Not sure if that already was released @rzwitserloot?
Lombok 1.16.22 [1] says: PLATFORM: lombok now counts as an incremental annotation processor for gradle. and the jar contains the correspondent META-INF/gradle/incremental.annotation.processors file.
Awesome, thanks!
@oehme Got it! Just to confirm with more clarity from my end:
In order for incremental compilation with Lombok to be used, all that is needed is:
1.16.22+4.7+No other special configuration needed. Properties such as annotationProcessorPath are recommended for optimality.
Is this correct?
Well, you also need to enable incremental compilation if you haven't already.
I had not. Looks like that's what I was missing. Thanks!
@oehme I now see the following error. It goes away if I remove the incremental compilation option:
warning: Can't get the delegate of the gradle IncrementalProcessingEnvironment. Lombok won't work.
/Users/jrlin/Projects/test-api/api-it-test-fwk/src/main/java/api/it/test/internal/Test.java:7: warning: lombok.javac.apt.LombokProcessor could not be initialized. Lombok will not run during this compilation: java.lang.ClassCastException: org.gradle.api.internal.tasks.compile.processing.IncrementalProcessingEnvironment cannot be cast to com.sun.tools.javac.processing.JavacProcessingEnvironment
public class Test {
^
at lombok.javac.apt.LombokProcessor.getJavacProcessingEnvironment(LombokProcessor.java:426)
at lombok.javac.apt.LombokProcessor.init(LombokProcessor.java:89)
at lombok.core.AnnotationProcessor$JavacDescriptor.want(AnnotationProcessor.java:112)
at lombok.core.AnnotationProcessor.init(AnnotationProcessor.java:165)
at lombok.launch.AnnotationProcessorHider$AnnotationProcessor.init(AnnotationProcessor.java:73)
at org.gradle.api.internal.tasks.compile.processing.DelegatingProcessor.init(DelegatingProcessor.java:57)
at org.gradle.api.internal.tasks.compile.processing.IsolatingProcessor.init(IsolatingProcessor.java:41)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$ProcessorState.<init>(JavacProcessingEnvironment.java:500)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:597)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:690)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1170)
at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:856)
at com.sun.tools.javac.main.Main.compile(Main.java:523)
at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
at org.gradle.api.internal.tasks.compile.IncrementalAnnotationProcessingCompileTask.call(IncrementalAnnotationProcessingCompileTask.java:76)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:50)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:36)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerCallable.call(AbstractDaemonCompiler.java:88)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerCallable.call(AbstractDaemonCompiler.java:76)
at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:42)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:46)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.process.internal.worker.request.WorkerAction.run(WorkerAction.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:745)
@pbi-qfs can you take a look at this? The field name didn't change so I don't know why this would fail.
@oehme thanks for the pointer, I didn't account for a delegate having a delegate having ...
I fixed this in https://github.com/pbi-qfs/lombok/tree/fixGradle5Incremental
@linjer can you try this quick build: lombok.jar.zip ?
@pbi-qfs I just tried the jar you provided above and I get the same result. To make more clear how I set up the module that is failing, here's part of the build.gradle:
repositories {
flatDir {
dirs 'lib' // jar was unzipped into <module>/lib/lombok.jar
}
}
dependencies {
apt name: 'lombok'
compileOnly name: 'lombok'
// ...
}
// See https://blog.gradle.org/incremental-compiler-avoidance
compileJava {
options.annotationProcessorPath = configurations.apt
}
Are you using apt plugin?
@linjer There should be a stacktrace output on stderr before the "warning: Can't get the delegate of the gradle IncrementalProcessingEnvironment. Lombok won't work." message (maybe you have to run gradle with -i or -d to see it), pointing to an error in some line between 416 and 419. Can you paste this as well?
@eximius313 You are correct I was using the apt plugin. In order to help isolate the issue, I've made a run without the plugin and only declaring compile 'org.projectlombok:lombok:1.16.22'.
I get a similar error still:
java.lang.ClassCastException: com.sun.tools.javac.processing.JavacProcessingEnvironment cannot be cast to com.sun.tools.javac.processing.JavacProcessingEnvironment
at lombok.javac.apt.LombokProcessor.getJavacProcessingEnvironment(LombokProcessor.java:419)
at lombok.javac.apt.LombokProcessor.init(LombokProcessor.java:89)
at lombok.core.AnnotationProcessor$JavacDescriptor.want(AnnotationProcessor.java:112)
at lombok.core.AnnotationProcessor.init(AnnotationProcessor.java:165)
at lombok.launch.AnnotationProcessorHider$AnnotationProcessor.init(AnnotationProcessor.java:73)
at org.gradle.api.internal.tasks.compile.processing.DelegatingProcessor.init(DelegatingProcessor.java:57)
at org.gradle.api.internal.tasks.compile.processing.IsolatingProcessor.init(IsolatingProcessor.java:41)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$ProcessorState.<init>(JavacProcessingEnvironment.java:500)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:597)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:690)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1170)
at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:856)
at com.sun.tools.javac.main.Main.compile(Main.java:523)
at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
at org.gradle.api.internal.tasks.compile.IncrementalAnnotationProcessingCompileTask.call(IncrementalAnnotationProcessingCompileTask.java:76)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:50)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:36)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerCallable.call(AbstractDaemonCompiler.java:88)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerCallable.call(AbstractDaemonCompiler.java:76)
at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:42)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:46)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.process.internal.worker.request.WorkerAction.run(WorkerAction.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:745)
warning: Can't get the delegate of the gradle IncrementalProcessingEnvironment. Lombok won't work.
/Users/jrlin/Projects/talent-pemberly-api/api-server/src/main/java/com/linkedin/talent/factories/api/server/internal/clients/hireidentity/HireIdentitiesClientFactory.java:16: warning: lombok.javac.apt.LombokProcessor could not be initialized. Lombok will not run during this compilation: java.lang.ClassCastException: org.gradle.api.internal.tasks.compile.processing.IncrementalProcessingEnvironment cannot be cast to com.sun.tools.javac.processing.JavacProcessingEnvironment
public class HireIdentitiesClientFactory extends SimpleSingletonFactory<HireIdentitiesClient> {
^
at lombok.javac.apt.LombokProcessor.getJavacProcessingEnvironment(LombokProcessor.java:426)
at lombok.javac.apt.LombokProcessor.init(LombokProcessor.java:89)
at lombok.core.AnnotationProcessor$JavacDescriptor.want(AnnotationProcessor.java:112)
at lombok.core.AnnotationProcessor.init(AnnotationProcessor.java:165)
at lombok.launch.AnnotationProcessorHider$AnnotationProcessor.init(AnnotationProcessor.java:73)
at org.gradle.api.internal.tasks.compile.processing.DelegatingProcessor.init(DelegatingProcessor.java:57)
at org.gradle.api.internal.tasks.compile.processing.IsolatingProcessor.init(IsolatingProcessor.java:41)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$ProcessorState.<init>(JavacProcessingEnvironment.java:500)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:597)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:690)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1170)
at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:856)
at com.sun.tools.javac.main.Main.compile(Main.java:523)
at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
at org.gradle.api.internal.tasks.compile.IncrementalAnnotationProcessingCompileTask.call(IncrementalAnnotationProcessingCompileTask.java:76)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:50)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:36)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerCallable.call(AbstractDaemonCompiler.java:88)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerCallable.call(AbstractDaemonCompiler.java:76)
at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:42)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:46)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.process.internal.worker.request.WorkerAction.run(WorkerAction.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:745)
you have to have both:
dependencies {
annotationProcessor("org.projectlombok:lombok:$lombokVersion")
compileOnly("org.projectlombok:lombok:$lombokVersion")
}
via https://github.com/rzwitserloot/lombok/issues/1602#issuecomment-389179459
"java.lang.ClassCastException: com.sun.tools.javac.processing.JavacProcessingEnvironment cannot be cast to com.sun.tools.javac.processing.JavacProcessingEnvironment" sounds as if you have two classloaders in place, and the ProcessingEnvironment is coming from another one than the classloader of the LombokProcessor class.
In addition you're not executing against the version of my comment above, since there is no cast to JavacProcessingEnvironment anymore in line 419, but to ProcessingEnvironment (which would probably fail in the next step again due to the classloader issues)
@eximius313 I can't get annotationProcessor "org.projectlombok:1.16.22" to work. It gives me the following error:
Could not resolve all files for configuration ':annotationProcessor'.
> Could not find org.projectlombok:1.16.22:.
Required by:
project :
@pbi-qfs This makes sense. I am using some custom tooling/infra for my company. I'm trying to reproduce this using a vanilla gradle project and it looks like it works in that context, so this issue may be due to our usage of multiple classloaders or some related issue.
@linjer it's org.projectlombok:lombok, you forgot the latter.
@pbi-qfs Gradle doesn't wrap the environment multiple times, so I don't think the change you linked will fix this.
@oehme thanks for this info - it was just a wild guess (since the relevant stacktrace was not included in the first comment), so I'll bury that "fix".
@oehme I believe you're correct. I see a similar error with the patched jar above from @pbi-qfs . I had tried to reproduce this in a new, plain gradle project to try to help isolate the issue, but it's been difficult to reproduce.
It's not a high priority for me so I think I'll move on for now. Let me know if there's anything either of you would like me to do to help document this in the meanwhile.
@oehme Regarding the dependency spec string org.projectlombok:lombok. I intentionally omitted the latter lombok component because in all the posts (above and in the linked issue), @eximius313 specifies the following:
annotationProcessor("org.projectlombok:$lombokVersion")
To me it looks intentionally suggested to use annotationProcessor("org.projectlombok:1.16.22") but it doesn't work.
@linjer That's a typo and can't work. A Maven dependency needs group. artifact and version.
As for the error: If you can create a reproducible example that would be great. I can't think of anything that could break classloading in this way off the top of my hat.
For reference, @oehme I also tried setting the following properties:
org.gradle.parallel=true
org.gradle.daemon=true
But with the very simple usage I could not reproduce the issue.
Daemon is the default and parallel has no influence on classloading. If anything, the problem would have to be some very deep hack into our compiler infrastructure. The only plugin I know that does such a thing is the errorprone plugin.
@linjer - sorry, that was copy-paste error. I corrected that
Why still get this warning using 1.18.22
> Task :app:compileErmengDebugJavaWithJavac
The following annotation processors are not incremental: jetified-compiler-3.8.5.jar (com.arialyy.aria:compiler:3.8.5).
Make sure all annotation processors are incremental to improve your build speed.