Spotless: Cannot add a configuration with name 'classpath' as a configuration with that name already exists (intermittent plugin-gradle concurrency bug)

Created on 12 Mar 2019 · 33Comments · Source: diffplug/spotless

Hi, we just had a spurious error on our jenkins after upgrading our spotless gradle plugin to 3.18 due to "org.gradle.api.InvalidUserDataException: Cannot add a configuration with name 'classpath' as a configuration with that name already exists."
This is for a multiproject build with parallel execution and it seems that gradle tries to create the classpath configuration multiple times in DefaultScriptHandler.
Seeing as there are no locks in this code, I'm guessing that defineConfiguration() is not meant to be called in parallel but somehow it is being called that way.

Also the error message printed to the log about missing repos is a bit confusing due to GradleProvisioner catching Exception and printing a error message that's better suited for catching ResolveException. I first troubleshooting thought was to go to our artifactory and check our logs (it sent the files correctly).

Gradle 5.2.1, Spotless gradle plugin 3.18, Ubuntu 16.04 docker container.

buildSrc/build.gradle
=====================

repositories {
  maven {
    url 'https://ourartifactory.stuff/mvn'
    credentials {
      username = artifactory_user
      password = artifactory_password
    }
  }
}
dependencies {
  compile "com.diffplug.spotless:spotless-plugin-gradle:3.18.0" // Use of buildSrc to infer repositories/dependencies to all build scripts
}


build.gradle
============
allprojects {
  pluginManager.withPlugin('java') {
    apply plugin: "com.diffplug.gradle.spotless"
    spotless {
      // See https://github.com/diffplug/spotless/tree/master/plugin-gradle for configuration
      java {
        target 'src/**/*.java' // Only format checked in code, not generated sources        
        removeUnusedImports() // removes any unused imports
        eclipse('4.9.0').configFile "$rootDir/utils/.settings/org.eclipse.jdt.core.prefs"
      }
    }
    tasks.withType(JavaCompile)*.dependsOn 'spotlessApply'    
  }
}

STACKTRACE from jenkins: 07:42:47 > Task :SubProj1:spotlessJava 07:42:47 You probably need to add a repository containing the '[com.google.googlejavaformat:google-java-format:1.7]' artifact in the 'build.gradle' of your root project. 07:42:47 E.g.: 'buildscript { repositories { mavenCentral() }}' 07:42:47 Note that included buildscripts (using 'apply from') do not share their buildscript repositories with the underlying project. 07:42:47 You have to specify the missing repository explicitly in the buildscript of the root project. 07:42:47 07:42:47 org.gradle.api.InvalidUserDataException: Cannot add a configuration with name 'classpath' as a configuration with that name already exists. 07:42:47 at org.gradle.api.internal.DefaultNamedDomainObjectCollection.assertCanAdd(DefaultNamedDomainObjectCollection.java:212) 07:42:47 at org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:93) 07:42:47 at org.gradle.api.internal.AbstractValidatingNamedDomainObjectContainer.create(AbstractValidatingNamedDomainObjectContainer.java:46) 07:42:47 at org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:75) 07:42:47 at org.gradle.api.internal.initialization.DefaultScriptHandler.defineConfiguration(DefaultScriptHandler.java:111) 07:42:47 at org.gradle.api.internal.initialization.DefaultScriptHandler.getConfigurations(DefaultScriptHandler.java:101) 07:42:47 at com.diffplug.gradle.spotless.GradleProvisioner.lambda$fromProject$1(GradleProvisioner.java:40) 07:42:47 at com.diffplug.spotless.JarState.provisionWithTransitives(JarState.java:87) 07:42:47 at com.diffplug.spotless.JarState.from(JarState.java:76) 07:42:47 at com.diffplug.spotless.JarState.from(JarState.java:71) 07:42:47 at com.diffplug.spotless.java.GoogleJavaFormatStep$State.<init>(GoogleJavaFormatStep.java:99) 07:42:47 at com.diffplug.spotless.java.GoogleJavaFormatStep$State.<init>(GoogleJavaFormatStep.java:95) 07:42:47 at com.diffplug.spotless.java.RemoveUnusedImportsStep.lambda$create$0(RemoveUnusedImportsStep.java:33) 07:42:47 at com.diffplug.spotless.FormatterStepImpl.calculateState(FormatterStepImpl.java:56) 07:42:47 at com.diffplug.spotless.LazyForwardingEquality.state(LazyForwardingEquality.java:56) 07:42:47 at com.diffplug.spotless.LazyForwardingEquality.writeObject(LazyForwardingEquality.java:68) 07:42:47 at sun.reflect.GeneratedMethodAccessor535.invoke(Unknown Source) 07:42:47 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 07:42:47 at java.lang.reflect.Method.invoke(Method.java:498) 07:42:47 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128) 07:42:47 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) 07:42:47 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 07:42:47 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 07:42:47 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 07:42:47 at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.serialize(DefaultValueSnapshotter.java:180) 07:42:47 at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.processValue(DefaultValueSnapshotter.java:172) 07:42:47 at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.snapshot(DefaultValueSnapshotter.java:59) 07:42:47 at org.gradle.internal.snapshot.ValueSnapshotStrategy.snapshot(ValueSnapshotStrategy.java:30) 07:42:47 at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.processValue(DefaultValueSnapshotter.java:99) 07:42:47 at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.snapshot(DefaultValueSnapshotter.java:59) 07:42:47 at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.snapshotTaskInputProperties(ResolveBeforeExecutionStateTaskExecuter.java:124) 07:42:47 at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.createExecutionState(ResolveBeforeExecutionStateTaskExecuter.java:90) 07:42:47 at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.execute(ResolveBeforeExecutionStateTaskExecuter.java:72) 07:42:47 at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58) 07:42:47 at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:109) 07:42:47 at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67) 07:42:47 at org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46) 07:42:47 at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93) 07:42:47 at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45) 07:42:47 at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:94) 07:42:47 at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57) 07:42:47 at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56) 07:42:47 at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36) 07:42:47 at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:63) 07:42:47 at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49) 07:42:47 at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:46) 07:42:47 at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416) 07:42:47 at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406) 07:42:47 at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165) 07:42:47 at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250) 07:42:47 at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158) 07:42:47 at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102) 07:42:47 at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36) 07:42:47 at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46) 07:42:47 at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43) 07:42:47 at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355) 07:42:47 at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343) 07:42:47 at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336) 07:42:47 at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322) 07:42:47 at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134) 07:42:47 at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129) 07:42:47 at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202) 07:42:47 at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193) 07:42:47 at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129) 07:42:47 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63) 07:42:47 at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46) 07:42:47 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 07:42:47 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 07:42:47 at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55) 07:42:47 at java.lang.Thread.run(Thread.java:748) 07:42:47

bug

Source

staffanf

All 33 comments

after upgrading our spotless gradle plugin to 3.18

What did you upgrade from? I suspect it's a very old version because...

You probably need to add a repository containing the '[com.google.googlejavaformat:google-java-format:1.7]' artifact in the 'build.gradle' of your root project.

is something you probably have to do. If you just got this error message for the first time, I'm guessing you upgraded from a pretty old one.

nedtwigg on 12 Mar 2019

3.6.0. But like I wrote, I think that error message is incorrect in this case since I get a InvalidUserDataException not a ResolveException and our artifactory logs shows that it resolved correctly.

staffanf on 12 Mar 2019

Have you tried adding buildscript { repositories { mavenCentral() }} ?

nedtwigg on 12 Mar 2019

Well, I have these entries in our artifactory logs....

[12/Mar/2019:07:42:46 +0100] "GET /artifactory/mvn/com/google/googlejavaformat/google-java-format/1.7/google-java-format-1.7.jar HTTP/1.1" 200 225839 "-" "Gradle/5.2.1 (Linux;4.9.0-6-amd64;amd64) (Oracle Corporation;1.8.0_171;25.171-b11)" "https"

So it's not a problem with resolving the dependency, but the error message is constructed with the assumption that if the code fails here, then that must be the error.
I believe Configuration config = project.getRootProject().getBuildscript().getConfigurations().detachedConfiguration(deps); gets called in parallel and since this getConfigurations() is not safe for concurrent access it fails sometimes. This might actually be a gradle issue or just a usage issue.
I'm not entirely sure what triggers this code to be executed. If you want we could discuss more on the community-slack.

staffanf on 12 Mar 2019

I'd be happy to discuss at https://gitter.im/diffplug/spotless However, I'd like to first try the error message's suggestion first. This code has a lot of people using it exactly like you, except that many seem to have added repositories { jcenter() } in their root build.grade. Also this code has not changed in a while, so I'd be surprised if there is a new bug here.

nedtwigg on 13 Mar 2019

👍1

Ahh, ok I kinda see how that could help... Our spotless dependency comes from buildSrc/build.gradle since we want it available for all .gradle scripts. That probably means that a "classpath" configuration linked to the rootProject's buildScript hasn't been created yet. Adding it to the buildscript should force create the "classpath" configuration in the rootProjects build.gradle in the configure phase, making it safe to use during task execution.

So I added your suggestion and tested with a loop similar to this (with more debugging added):
while [ 1 ] ; do ./gradlew -stop ; git clean -fdx ; ./gradlew spotlessApply ; sleep 1 ; done

It failed on iteration 42 after 1234 s. Ran again and it failed on iteration 30 after 852 s.

Then I removed the workaround and ran the same script again.
It failed on iteration 14 after 400 s. Another run with no fix resulted in a fail after 19 iterations and 541 s.

So basically the same issues with or without repositories { jcenter() }. It's intermittent and not very fast to reproduce.

I still unsure if it's ok to call project.getRootProject().getBuildscript().getConfigurations() during task execution? If that the case then the creation of the creation of "classpath" configuration should be thread-safe in the gradle internals.

staffanf on 13 Mar 2019

Interesting! And when you say it failed, you mean that it failed with org.gradle.api.InvalidUserDataException: Cannot add a configuration with name 'classpath' as a configuration with that name already exists.?

I still unsure if it's ok to call project.getRootProject().getBuildscript().getConfigurations() during task execution?

It's an odd error, because I would expect that calling getters, even if they aren't threadsafe, would at worst return stale data. And spotless doesn't use use those except to create a detachedConfiguration, whose whole point is to be separate from the other configurations.

It's super valuable that you've found this issue and found a way to reproduce it. However, it's rare and "fails safe" (no silent incorrect output), so it's a fairly benign issue. Happy to merge any PR's which fix the issue, but it's not going to make the top of my todo list. Creating these detached configurations is used throughout the codebase to enable lazy evaluation, so it's likely that there's no easy workaround without a major performance regression.

nedtwigg on 13 Mar 2019

Copied from gradle forums

Disclaimer: I haven’t taken much time to understand the codebase.

It seems there’s a race condition initializing the root project’s buildscript classpath. What’s the reason for wanting to create the detached configurations in the root project’s buildscript? Why not create them in the current project?

If you did

Configuration config = project.getConfigurations().detachedConfiguration(deps)

Instead of

Configuration config = project.getRootProject().getBuildscript().getConfigurations().detachedConfiguration(deps)

I’d assume any race conditions could be avoided since each thread would be mutating a separate project instance

uklance on 27 Mar 2019

Well https://github.com/diffplug/spotless/commit/c188743526d76ad66729fe485a0e41ab8139ff41 changed this to use the rootProject's buildscript block instead since it makes configuration of the buildscript repos easier...
And yes, reverting to that would probably solve the threading issue...

I guess reverting that would require a good explanation of how to configure buildscript repositories for all projects. That change simplified configuration.
I built the change locally to test it, but I'm having troubles using it in my project. --include-build doesn't wanna play.

staffanf on 27 Mar 2019

I'm having troubles using it in my project. --include-build doesn't wanna play.

See here for an example of using composite builds to build an external plugin then use it

Or you could simply copy the spotless gradle sources to your project's buildSrc folder for a hack test

uklance on 27 Mar 2019

Well I get the setttings.gradle/includeBuild to work...it just doesn't replace the the spotless dependency for me. Any idea why? "./gradlew buildEnvironment" shows the non-included version.

staffanf on 27 Mar 2019

Hack the version so it's different to what's in the plugin repository.

See here for how to include the plugin

uklance on 27 Mar 2019

I believe Configuration config = project.getRootProject().getBuildscript().getConfigurations().detachedConfiguration(deps); gets called in parallel and since this getConfigurations() is not safe for concurrent access it fails sometimes.

Aha! This seems like a great insight.

I wonder if this would solve the problem while still allowing the buildscript repositories simplification:

synchronized (GradleProvisioner.class) {
   project.getRootProject().getBuildscript().getConfigurations().detachedConfiguration(deps)
}

Btw, here is how I test a plugin locally.

nedtwigg on 27 Mar 2019

There's absolutely no need to use the root project to create the Configuration. You could create it in the current project without needing to add synchronisation logic. As you can see from the code below, the only reason the configuration is created is to get the Set<File> which is a result of Configuration.resolve().
```
Configuration config = project.getRootProject().getBuildscript().getConfigurations().detachedConfiguration(deps);
config.setDescription(mavenCoords.toString());
config.setTransitive(withTransitives);
return config.resolve();

uklance on 27 Mar 2019

Any idea why composite builds does not work for spotless? Have you tried that for local development?
I got the publishToMavenLocal thing working and have added the synchronized block. I'm running my loop and if does not fail over night I'll consider this a work-around and submit a pull-request

staffanf on 27 Mar 2019

have added the synchronized block

The wrong solution IMHO

uklance on 27 Mar 2019

Any idea why composite builds does not work for spotless? Have you tried that for local development?

No and no :) Possibly because spotless uses a pretty old gradle wrapper. We're still compatible with 2.x.

There's absolutely no need to use the root project to create the Configuration. You could create it in the current project without needing to add synchronisation logic.

My memory might be wrong, but I think this is the story:

we used to just do project.getConfigurations().detachedConfiguration()
but that meant that we were using the project repositories instead of the buildscript repositories. That was a problem because Spotless is a build tool, and it was surprising that it was using project repositories to do build things.
so then we changed to project.getBuildscript().getConfigurations().detachedConfiguration()
but then the problem was that subprojects usually didn't have any repositories defined at all, and so they weren't able to resolve their deps. Spotless is unusual in that it defers until build time to resolve deps, which allowed us to do lazy up-to-date-checking back in 2.x, not as important now that gradle has lazy evaluation built-in.
so we changed to our current project.getRootProject().getBuildscript().getConfigurations().detachedConfiguration

It would be possible to confirm or refute this with a little archaeology, but...

I'm running my loop and if does not fail over night I'll consider this a work-around and submit a pull-request

if this works out then it's probably not worth the archaeology, because we'll have a simple and proven fix to merge :)

nedtwigg on 27 Mar 2019

Thanks for the detailed information... I see why you're doing it in the root project's buildscript.

It's worth raising a bug against gradle. It seems that two threads are trying to add the "classpath" configuration to the root project's buildscript. I guess both threads think they're the first to load the buildscript classpath

uklance on 27 Mar 2019

If anyone wants to raise the bug and shepherd it through Gradle, that's fine with me. If I were a gradle dev, I would probably close it and say "don't do this!", or if I were really diligent I would lock down the API more carefully so that our proposed workaround is impossible. "Don't modify one project from another" is a good restriction that makes parallel project evaluation possible, and if they really enforce the limit then spotless' multiproject support is totally broken, and users are going to have to declare buildscript { repo { blocks in all their subprojects.

https://gizmodo.com/astronauts-have-done-so-so-much-with-duct-tape-and-ele-1711503831

nedtwigg on 27 Mar 2019

Well, if you're mutating the configurations in the root project, one could argue that the tasks should live there too

You could even create "dummy" tasks in the subprojects which dependsOn the "real" tasks in the root project

uklance on 27 Mar 2019

Well, if you're mutating the configurations in the root project, one could argue that the tasks should live there too

Agreed! That's why I don't think raising a bug with gradle is likely to be useful.

But we're doing a special mutation, we're creating a detached configuration, which doesn't change the other named configurations. That's why I think we can get away with abusing Gradle's threading spec, so long as we agree to at least coordinate with ourselves by only abusing the spec one-at-a-time. This will probably break if another plugin out there uses the same hack that we are using, but I think our situation is unique because we implemented lazy-task-configuration before they had APIs for it.

You could even create "dummy" tasks in the subprojects which dependsOn the "real" tasks in the root project.

That's true. It's a much bigger change than duct-taping in a synchronized. Happy to merge such a change, but only if it works better for our users than a synchronized hack.

At best, such a change will necessarily prevent parallel configuration of spotless tasks, and it will require spotless to always be applied to the root project, which is not required currently. And unless we're careful, we might end up preventing parallel execution of spotless tasks as well.

nedtwigg on 27 Mar 2019

I'll not be contributing such a fix so I think your duct tape wins. I'd argue

synchronized (project.getRootProject()) {...}

Is slightly better than

synchronized (GradleProvisioner.class) {...}

uklance on 27 Mar 2019

👍1

For what it's worth, I think there's still a bug in Gradle. I get the feeling this race condition exists even if you don't mutate the root project. It seems the race condition is before that, it occurs when creating the root project's buildscript classloader.

uklance on 27 Mar 2019

Here's the pull-request
I used the suggested changes and did not get any error during testing.

staffanf on 28 Mar 2019

Published in x.21.0. Thanks again @staffanf and @uklance for sticking with it and surfacing/fixing a subtle bug.

nedtwigg on 28 Mar 2019

I might be a bit late to the party, however, is there a true reason to create a detached configuration and resolve it?

Could Spotless create its own configuration if it requires extra tools on the classpath?

For instance, we have the following tool-specific configurations:

checkstyle - The Checkstyle libraries to be used for this project.
\--- com.puppycrawl.tools:checkstyle:8.22
     +--- info.picocli:picocli:3.9.6
     +--- antlr:antlr:2.7.7
     +--- org.antlr:antlr4-runtime:4.7.2
     +--- commons-beanutils:commons-beanutils:1.9.3
     |    \--- commons-collections:commons-collections:3.2.2
     +--- com.google.guava:guava:27.1-jre
     |    +--- com.google.guava:failureaccess:1.0.1
     |    +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     |    +--- com.google.code.findbugs:jsr305:3.0.2
     |    +--- org.checkerframework:checker-qual:2.5.2
     |    +--- com.google.errorprone:error_prone_annotations:2.2.0
     |    +--- com.google.j2objc:j2objc-annotations:1.1
     |    \--- org.codehaus.mojo:animal-sniffer-annotations:1.17
     \--- net.sf.saxon:Saxon-HE:9.9.1-3

jacocoAgent - The Jacoco agent to use to get coverage data.
\--- org.jacoco:org.jacoco.agent:0.8.2

jacocoAnt - The Jacoco ant tasks to use to get execute Gradle tasks.
\--- org.jacoco:org.jacoco.ant:0.8.2
     +--- org.jacoco:org.jacoco.core:0.8.2
     |    +--- org.ow2.asm:asm:6.2.1
     |    +--- org.ow2.asm:asm-commons:6.2.1
     |    |    +--- org.ow2.asm:asm:6.2.1
     |    |    +--- org.ow2.asm:asm-tree:6.2.1
     |    |    |    \--- org.ow2.asm:asm:6.2.1
     |    |    \--- org.ow2.asm:asm-analysis:6.2.1
     |    |         \--- org.ow2.asm:asm-tree:6.2.1 (*)
     |    \--- org.ow2.asm:asm-tree:6.2.1 (*)
     +--- org.jacoco:org.jacoco.report:0.8.2
     |    \--- org.jacoco:org.jacoco.core:0.8.2 (*)
     \--- org.jacoco:org.jacoco.agent:0.8.2

spotbugsPlugins - The SpotBugs plugins to be used for this project.
No dependencies

It looks like Spotless could create regular configuration rather than trying to clone one from rootproject in an unsafe way.
@nedtwigg , what do you think?

I've got a CI failure around GradleProvisioner, however, the next build (with no changes at all) succeeded.

> Task :src:release:spotlessJava FAILED
You probably need to add a repository containing the '[com.google.googlejavaformat:google-java-format:1.7]' artifact in the 'build.gradle' of your root project.
E.g.: 'buildscript { repositories { mavenCentral() }}'
Note that included buildscripts (using 'apply from') do not share their buildscript repositories with the underlying project.
You have to specify the missing repository explicitly in the buildscript of the root project.

org.gradle.api.InvalidUserDataException: You must specify a base url or at least one artifact pattern for an Ivy repository.
    at org.gradle.api.internal.artifacts.repositories.DefaultIvyArtifactRepository.validate(DefaultIvyArtifactRepository.java:212)
    at org.gradle.api.internal.artifacts.repositories.DefaultIvyArtifactRepository.createDescriptor(DefaultIvyArtifactRepository.java:162)
    at org.gradle.api.internal.artifacts.repositories.AbstractResolutionAwareArtifactRepository.getDescriptor(AbstractResolutionAwareArtifactRepository.java:32)
    at org.gradle.api.internal.artifacts.configurations.ResolveConfigurationResolutionBuildOperationDetails$RepositoryImpl$1.transform(ResolveConfigurationResolutionBuildOperationDetails.java:168)
    at org.gradle.api.internal.artifacts.configurations.ResolveConfigurationResolutionBuildOperationDetails$RepositoryImpl$1.transform(ResolveConfigurationResolutionBuildOperationDetails.java:165)
    at org.gradle.util.CollectionUtils.collect(CollectionUtils.java:207)
    at org.gradle.util.CollectionUtils.collect(CollectionUtils.java:199)
    at org.gradle.api.internal.artifacts.configurations.ResolveConfigurationResolutionBuildOperationDetails$RepositoryImpl.transform(ResolveConfigurationResolutionBuildOperationDetails.java:165)
    at org.gradle.api.internal.artifacts.configurations.ResolveConfigurationResolutionBuildOperationDetails$RepositoryImpl.access$000(ResolveConfigurationResolutionBuildOperationDetails.java:160)
    at org.gradle.api.internal.artifacts.configurations.ResolveConfigurationResolutionBuildOperationDetails.<init>(ResolveConfigurationResolutionBuildOperationDetails.java:64)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$7.description(DefaultConfiguration.java:656)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
    at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolveGraphIfRequired(DefaultConfiguration.java:603)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.access$600(DefaultConfiguration.java:139)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$6.run(DefaultConfiguration.java:583)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$SafeExclusiveLockImpl.withLock(DefaultProjectStateRegistry.java:245)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolveExclusively(DefaultConfiguration.java:579)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.access$500(DefaultConfiguration.java:139)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$5.run(DefaultConfiguration.java:570)
    at org.gradle.internal.Factories$1.create(Factories.java:26)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry.withLenientState(DefaultProjectStateRegistry.java:133)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry.withLenientState(DefaultProjectStateRegistry.java:125)
    at org.gradle.api.internal.artifacts.transform.DomainObjectProjectStateHandler.withLenientState(DomainObjectProjectStateHandler.java:59)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolveToStateOrLater(DefaultConfiguration.java:567)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.getResolvedConfiguration(DefaultConfiguration.java:556)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration_Decorated.getResolvedConfiguration(Unknown Source)
    at com.github.vlsi.gradle.checksum.ChecksumDependency.verifyDependencies(ChecksumDependency.kt:206)
    at com.github.vlsi.gradle.checksum.ChecksumDependency.access$verifyDependencies(ChecksumDependency.kt:58)
    at com.github.vlsi.gradle.checksum.ChecksumDependency$resolutionListener$1$beforeResolve$4$1.invoke(ChecksumDependency.kt:98)
    at com.github.vlsi.gradle.checksum.ChecksumDependency$resolutionListener$1$beforeResolve$4$1.invoke(ChecksumDependency.kt:88)
    at com.github.vlsi.gradle.checksum.Stopwatch.invoke(Stopwatch.kt:36)
    at com.github.vlsi.gradle.checksum.Stopwatch.invoke$default(Stopwatch.kt:31)
    at com.github.vlsi.gradle.checksum.ChecksumDependency$resolutionListener$1$beforeResolve$4.execute(ChecksumDependency.kt:97)
    at com.github.vlsi.gradle.checksum.ChecksumDependency$resolutionListener$1$beforeResolve$4.execute(ChecksumDependency.kt:88)
    at org.gradle.internal.event.BroadcastDispatch$ActionInvocationHandler.dispatch(BroadcastDispatch.java:92)
    at org.gradle.internal.event.BroadcastDispatch$ActionInvocationHandler.dispatch(BroadcastDispatch.java:80)
    at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:42)
    at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:231)
    at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:150)
    at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
    at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:325)
    at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:235)
    at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:141)
    at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:37)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at com.sun.proxy.$Proxy41.afterResolve(Unknown Source)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$7.run(DefaultConfiguration.java:621)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
    at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolveGraphIfRequired(DefaultConfiguration.java:603)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.access$600(DefaultConfiguration.java:139)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$6.run(DefaultConfiguration.java:583)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$SafeExclusiveLockImpl.withLock(DefaultProjectStateRegistry.java:245)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolveExclusively(DefaultConfiguration.java:579)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolveToStateOrLater(DefaultConfiguration.java:574)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.access$2300(DefaultConfiguration.java:139)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$ConfigurationFileCollection.getSelectedArtifacts(DefaultConfiguration.java:1241)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration$ConfigurationFileCollection.getFiles(DefaultConfiguration.java:1230)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.getFiles(DefaultConfiguration.java:491)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration_Decorated.getFiles(Unknown Source)
    at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.resolve(DefaultConfiguration.java:481)
    at com.diffplug.gradle.spotless.GradleProvisioner.lambda$fromProject$1(GradleProvisioner.java:50)
    at com.diffplug.spotless.JarState.provisionWithTransitives(JarState.java:87)
    at com.diffplug.spotless.JarState.from(JarState.java:76)
    at com.diffplug.spotless.JarState.from(JarState.java:71)
    at com.diffplug.spotless.java.GoogleJavaFormatStep$State.<init>(GoogleJavaFormatStep.java:99)
    at com.diffplug.spotless.java.GoogleJavaFormatStep$State.<init>(GoogleJavaFormatStep.java:95)
    at com.diffplug.spotless.java.RemoveUnusedImportsStep.lambda$create$0(RemoveUnusedImportsStep.java:33)
    at com.diffplug.spotless.FormatterStepImpl.calculateState(FormatterStepImpl.java:56)
    at com.diffplug.spotless.LazyForwardingEquality.state(LazyForwardingEquality.java:56)
    at com.diffplug.spotless.LazyForwardingEquality.writeObject(LazyForwardingEquality.java:68)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.serialize(DefaultValueSnapshotter.java:171)
    at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.processValue(DefaultValueSnapshotter.java:163)
    at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.processValue(DefaultValueSnapshotter.java:88)
    at org.gradle.internal.snapshot.impl.DefaultValueSnapshotter.snapshot(DefaultValueSnapshotter.java:53)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.lambda$fingerprintInputProperties$2(CaptureStateBeforeExecutionStep.java:135)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.visitInputProperties(ExecuteActionsTaskExecuter.java:266)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.fingerprintInputProperties(CaptureStateBeforeExecutionStep.java:131)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.createExecutionState(CaptureStateBeforeExecutionStep.java:113)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:67)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:47)
    at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:140)
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:62)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:108)
    at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67)
    at org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:94)
    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:95)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
    at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
    at java.lang.Thread.run(Thread.java:748)

vlsi on 30 Sep 2019

Spotless could create regular configuration rather than trying to clone one from rootproject in an unsafe way

What is unsafe about it? The failure you describe seems important, if you can get it to happen even just 1 time out 10, it would be worth opening an issue, as it would reveal a gradle bug worth fixing.

Currently, a step doesn't do any work at all until it runs for the first time. This basically allowed us to do configuration avoidance before Gradle had APIs for that. Now that Gradle has that built in, we could dump ours, but it's baked deeply into the code. Also, our maven plugin benefits from the lazy configuration, and maven does not have the configuration avoidance APIs that gradle does.

Summary: it is very difficult to change this aspect of Spotless, and even if it were easy there are real downsides to our non-gradle users. If you have a bug, it would be very helpful to find a way to reproduce it and post it as an issue.

nedtwigg on 1 Oct 2019

What is unsafe about it?

The above "You probably need to add a repository containing" was produced out of a project that has repositories defined. I don't know if it is Spotless bug or Gradle bug, however, the code that synchronizes on configurations looks fishy, so I ask here first.

vlsi on 1 Oct 2019

It is fishy, and there is a long story (see here). The constraints are:

some people want their build tools to suck deps from a different place than their project's deps (e.g. enable a snapshot repo for their project, but not for the buildtools)
most subprojects don't have any buildscript repositories defined
that means that we have to grab them from the root buildscript

As described above, our synchronized hack worked for somebody's overnight test. I think the options to really fix this are:

isolate and fix the gradle issue (not really a bug, more like adding a threadsafe feature to gradle)
figure out how to create detached configuration in a subproject that has the same repos as the root project
break everyone's buildscripts by making them add buildscript { repo to every subproject

nedtwigg on 1 Oct 2019

e.g. enable a snapshot repo for their project, but not for the buildtools

This could probably be handled with https://docs.gradle.org/current/userguide/declaring_repositories.html#declaring_a_repository_filter (and releasesOnly() / snapshotOnly())

I agree it was not available before Gradle 5.0. However, it looks like for now users have a way to confine artifacts to repositories, so is this "build script repository" really needed now?
Isn't a regular configuration enough?

As a bonus, it would have a proper name in the build logs, and users would be able to use Gradle's dependency approach to specify versions (e.g. platform aka BOM).

isolate and fix the Gradle issue (not really a bug, more like adding a threadsafe feature to gradle)

They seem to go in the quite opposite direction: they forbid resolution from user threads, they forbid to resolve configurations across projects. I won't be too much surprised if they would watch carefully the code Spotless uses, think much, and invent a clever way to forbid that style of use :(

For instance, you might have seen https://github.com/gradle/gradle/pull/10844 which is pretty much the same issue. And you see how they close the issue.

vlsi on 1 Oct 2019

They seem to go in the quite opposite direction

I agree this is not likely to be successful.

This could probably be handled with

The snapshot was just an example. As in the issue that you linked, people expect their project and buildscript repos to be separate. Another case might be buildscript { repo 'approvedBuildTools' } repo {'approvedShippingDeps' }.

I believe the most likely paths to nirvana here are:

figure out how to create detached configuration in a subproject that has the same repos as the root project
break everyone's buildscripts by making them add buildscript { repo to every subproject

nedtwigg on 1 Oct 2019

What is unsafe about it?

It seems to defeat Gradle Build Scan as well (however they seem to agree to fix that):

https://discuss.gradle.org/t/your-build-scan-could-not-be-displayed-what-does-this-mean/33302/10

vlsi on 2 Oct 2019

👍1

Starting with 3.27.0, we no longer abuse the root buildscript with a hacky synchronized block (see PR above).

nedtwigg on 2 Jan 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

NPE in EclipseXmlFormatterStepImpl.java / PREFERENCE_FACTORY is null

epkanol · 5Comments

Don't fail after first codestyle violation

ubuntudroid · 6Comments

Exceptions when running clean & spotlessCheck during same build with buildcache

JLLeitschuh · 7Comments

Bug: Directory 'build/spotless/spotlessKotlin' specified for property 'spotlessOutDirectory' does not exist

erikhuizinga · 4Comments

java.lang.NoClassDefFoundError: javax/script/ScriptException with freshmark on Java9

wreulicke · 4Comments