Gradle: Definitions from buildSrc/ not found in settings.gradle.kts using gradle 6.0-rc-1

[wolfseifert]

Definitions from buildSrc/ not found in settings.gradle.kts using gradle 6.0-rc-1.
This still worked with gradle-6.0-20190827220029+0000 and earlier versions.

[ldaley]

Hi @wolfseifert,

This has changed in 6.0, and was deprecated in 5.6. Please see: https://docs.gradle.org/current/userguide/upgrading_version_5.html#buildsrc_usage_in_gradle_settings

Can you please share some information about what kind of logic in buildSrc you are trying to use in the settings script?

[wolfseifert]

Hello  Luke,

here are some sample usages:

buildSrc/:

fun RepositoryHandler.nexus(): MavenArtifactRepository =
  maven {
    url = URI("http://nexushost:8081/repository/maven-public")
  }

fun Project.repositories() {
  repositories { nexus() }
}

fun PluginManagementSpec.repositories() {
  repositories { nexus() }
}

 
val osName: String =
  System.getProperty("os.name")!!

fun unsupportedOS(): Nothing =
  throw GradleException("""Operating system "$osName" is not supported.""")

enum class OS {
  LINUX, MACOSX, WINDOWS
}

val os: OS = when {
  osName.contains("linux", true) -> OS.LINUX
  osName.contains("mac", true) -> OS.MACOSX
  osName.contains("windows", true) -> OS.WINDOWS
  else -> unsupportedOS()
}

settings.gradle.kts:

pluginManagement {
  repositories()
}

if (os == OS.LINUX) include("gnome")

My intention was to configure this once (in buildSrc/) and then reuse it in different build.gradle.kts files.

The settings.gradle.kts is (in this sense) just another gradle build file.

Regards,

Wolfgang Seifert

[rschmitt]

I have a use case involving a large composite build, where code in buildSrc defines a list of packages that may be included in the build, settings.gradle.kts detects which of these packages are actually present and adds them to the build, and build.gradle.kts defines tasks for all included packages. What am I supposed to do on Gradle 6.0?

[ldaley]

Thanks for the info @wolfseifert.

Unfortunately, there are pros and cons to both arrangements (settings-then-buildSrc and buildSrc-then-settings), and we opted for the former after considering.

The con of this is exactly what you have hit. It's now less convenient to use complex logic in your settings script. Now, you have to either:

1. Inline the logic into the settings file
2. Move the logic to a shared script that can be used where it needs to
3. Move the logic to a pre-built binary that you load in the settings file (i.e. a settings plugin)

The pros that compelled us to make the change:

1. Settings plugins can influence buildSrc and main build (i.e. apply a build plugin to both)
2. Build cache configuration is applied to buildSrc
3. buildSrc behaves more like a regular included build

We won't be changing the behaviour back to the pre Gradle 6 arrangement. Please let us know if you would like more detail on how to use one of the alternative mechanisms for using complex logic in a settings script.

[JavierSegoviaCordoba]

@ldaley do you have some examples of points 2 and 3?

I was declaring all the modules in buildSrc so now I have to declare two times, in settings and in buildSrc, can I avoid this?

[ldaley]

@JavierSegoviaCordoba can you please detail what you mean by “declaring all modules”. Exactly what that is will dictate the solution.

[JavierSegoviaCordoba]

I had a more elaborated version of val modules = listOf(app, core, ...) in buildSrc. So I can easily implement them in every build.gradle.kts which need them.

To easily include all in my project, in settings.gradle.kts I was doing a modules.forEach { module -> include(module) }

Now I have to declare them in the settings Gradle and in the buildSrc Kotlin object files.

[ldaley]

So I can easily implement them in every build.gradle.kts which need them.

What do you mean by this?

[JavierSegoviaCordoba]

If a module depends of other module, for example I need the core module, I can put inside of dependencies: implementation(project(Modules.core))

[ldaley]

@JavierSegoviaCordoba I suggest moving your list of modules to a data file, and reading it in both places. For example, adding a modules.properties to your project root and enumerating it one way in settings and one in buildSrc.

[malachid]

Probably an important change to be calling out since it breaks anyone using resolutionStrategy with buildSrc for the versions.

[ldaley]

@malachid can you elaborate please.

[malachid]

Sure. In previous versions, we would use something like Versions.kt inside buildSrc to ensure a consistent version across dependencies, plugins, etc. We could have a resolutionStrategy that relied on that. Now, we have to have two copies of the versions - one inside settings.gradle.kts for any plugin that doesn't auto-resolve, and one in buildSrc to use for classpath/plugins/dependencies.

[ldaley]

@malachid what did your Versions.kt look like? Can you provide an indicative example?

[malachid]

If you look around you'll see that the most common recommendation for buildSrc over the last year was to migrate your Kotlin version information over rather than using strings everywhere.

So, for example, let's say you are writing a multi-module project. You might define const val junitVersion to point to Jupiter, then have an object or setOf for your test libs. In your various build.gradle.kts, you might do something like TestLibs.forEach { testImplementation(it) }

Now let's say a few of those modules are Kotlin multiplatform and rely on https://github.com/yshrsmz/BuildKonfig

When you apply plugin, it can't be found; so you add a resolutionStrategy to settings.gradle.kts that simply calls useModule for any that need it (using mapOf in the versions)...

say, something like this:

    resolutionStrategy {
        eachPlugin {
            Plugins.resolve
                    .filterKeys { it == requested.id.id }
                    .values
                    .forEach { useModule(it) }
        }
    }

With the change, you have to manually hard code the name and version of that plugin in settings.gradle.kts _as well as_ in your versions logic for the rest of the build; whereas before you only had one copy.

[ldaley]

Thanks @malachid. Your additional information is useful.

[bejibx]

I have pretty much the same configuration as @malachid in my work project. I spent a lot of time in past to refactor it to this architecture and it works pretty good for us. It is very convenient to have all your dependencies and versions in one place. This makes updating them super easy - you just changes version there and that's all. Also we are using Gradle Kotlin DSL and declare our dependencies/version as properties inside Kotlin object, which also have many benefits.

Now I'm trying to update Gradle to latest version and suddenly my whole architecture is gone. Now I have to redeclare some dependencies (plugins, repositories...) in settings.gradle.kts. Now every time you want to update some plugin, you have to remember to change it version in two places.

Does Gradle 6.+ have other mechanism to achieve similar thing?

[nkiesel]

same here: I use the "buildSrcverions" plugin which generates a buildSrc/src/main/kotlin/Versions.kt file containing const val org_jlleitschuh_ktlint_gradle_plugin: String = "9.1.1". I currently then use that in settings.gradle like so: id "org.jlleitschuh.gradle.ktlint" version Versions.org_jlleitschuh_ktlint_gradle_plugin. This no longer works. Iguess I have to look at newer version of "buildSrcVersions" which might remedy this, but it will for sure mean that I have to re-tran my team to understand the new version mgmt.

[nanjingdaqi]

Thanks for the info @wolfseifert.

Unfortunately, there are pros and cons to both arrangements (settings-then-buildSrc and buildSrc-then-settings), and we opted for the former after considering.

The con of this is exactly what you have hit. It's now less convenient to use complex logic in your settings script. Now, you have to either:

1. Inline the logic into the settings file
2. Move the logic to a shared script that can be used where it needs to
3. Move the logic to a pre-built binary that you load in the settings file (i.e. a settings plugin)

The pros that compelled us to make the change:

1. Settings plugins can influence buildSrc and main build (i.e. apply a build plugin to both)
2. Build cache configuration is applied to buildSrc
3. buildSrc behaves more like a regular included build

We won't be changing the behaviour back to the pre Gradle 6 arrangement. Please let us know if you would like more detail on how to use one of the alternative mechanisms for using complex logic in a settings script.

I am using gradle kts. Could anyone provide the demo for "shared project" and "setting plugin"? I failed to search relevant information.

[dybarsky]

Hi here.
We also faced this problem. In our multimodule project we have different kotlin files in buildSrc dir. For example:

object AppModule {
  const val APPLICATION = ":application"
  const val BASE_API = ":base-api"
  const val BASE_EXTENSION = ":base-extension"
  const val BASE_GEOLOCATION = ":base-geolocation"
  ...
}

object AppLibrary {
    const val OKHTTP = "com.squareup.okhttp3:okhttp:${AppVersion.OKHTTP}"
}

object AppVersion {
  const val OKHTTP = "4.4.1"
}

Our settings.gradle file in root project uses AppModule class like this

include(
  AppModule.APPLICATION,
  AppModule.BASE_API,
  AppModule.BASE_EXTENSIONS,
   ...
)

With new gradle version we can't use AppModule class anymore and should use module names as strings. Any ideas how we can solve this without duplication of modules definitions or reading from properties file?

[JavierSegoviaCordoba]

FileWalkTree to find all the modules or use the auto-module plugin

[eygraber]

@ldaley any suggestions for how to handle the issue that @malachid presented? I run into this issue a lot (and it appears that others do as well).

[android10]

Same case here. In order to have dependencies in one place defined inside buildSrc.
For examples Dependencies.kt where we define modules:

object Modules {
    const val app = ":app"
}

From settings.gradle.kts:

include (Modules.app)  -> ERROR: Unresolved reference: Modules
//include (":app")     -> Works but I do not want string literals here

I'm not expecting another API change if the decision has been made but at least some tips/tricks/pattern on how to handle this scenario.

Thank you very much :heart:

[JavierSegoviaCordoba]

@android10 take a look to auto-module.

Or you can create an algorithm to include every module automatically using FileWalkTree and searching every build.gradle.kts file in the project (excluding the root build.gradle.kts file).

[dybarsky]

@JavierSegoviaCordoba I tried both options. automodule is over-complicated solution.

[JavierSegoviaCordoba]

@dybarsky auto-module option is a one-line solution that even creates for you the Modules.kt with all the modules of the project automatically. It is easier than doing manually, pre-Gradle 6 even.

[dybarsky]

@JavierSegoviaCordoba It's onle line solution, it's true. But it adds task, generates/modifies gradle files, handles dependencies. That's not what we need.

[dybarsky]

rootDir
  .walk()
  .maxDepth(1)
  .filter {
    it.name != "buildSrc"
      && it.isDirectory 
      && file("${it.absolutePath}/build.gradle.kts").exists()
  }
  .forEach {
    include(":${it.name}")
  }

@android10 That's solution which works for us.
@JavierSegoviaCordoba Thanks for pointing at FileWalkTree class

[android10]

@dybarsky @JavierSegoviaCordoba thanks both of you for the contribution :heart:

[Dreyar]

rootDir
  .walk()
  .maxDepth(1)
  .filter {
    it.name != "buildSrc"
      && it.isDirectory 
      && file("${it.absolutePath}/build.gradle.kts").exists()
  }
  .forEach {
    include(":${it.name}")
  }

@android10 That's solution which works for us.
@JavierSegoviaCordoba Thanks for pointing at FileWalkTree class

How did you used this to fix your issue above? (https://github.com/gradle/gradle/issues/11090#issuecomment-632019679)
I changed gradle from 6.5 to 5.6.4 but i do not like that approach.

[dybarsky]

@Dreyar I have replaced include(AppModule.APPLICATION, ...) with logic above. Now settings.gradle.kts file has no references to classes from buildSrc scope.

[sellmair]

I also would like to add that this is causing inconveniences for me in certain projects:
E.g. in https://github.com/Kotlin/dokka, we would like to have some custom build logic regarding the repositories where we get our dependencies from. Now I have to duplicate this in settings.gradle.kts, because the same logic should be used in the pluginManagement block (which can't access our buildSrc folder) 😢

I think this really example really contradicts the purpose of having a buildSrc folder.

[Blacklands]

I've read all of this, and I'm still confused.

I'm fairly new to Gradle. I started with the Kotlin DSL from the beginning already.
First I read that it's preferred to use the new plugins {} block to apply plugins.

Then I read that it's preferred to apply plugins in settings.gradle[.kts], because the plugins {} block doesn't allow (most) custom logic, by design. In the settings file, the pluginManagement {} block is there specifically to allow custom logic.

Up to this point, that makes sense to me. But if settings.gradle[.kts] is basically the very first thing that is being evaluated/built in a project, then where should that custom logic come from, if I don't want to put it all right into the file itself...?

This comment said:

2. Move the logic to a shared script that can be used where it needs to
3. Move the logic to a pre-built binary that you load in the settings file (i.e. a settings plugin)

Does 3 mean that one would need to create a plugin, compile it, and then include it in settings.gradle[.kts] somehow (so we would need to publish it first, or define a folder of the project as a custom plugin repository?)? That sounds very overkill if one just has some simple build logic (that may also be project-specific, so the plugin couldn't even be reused somewhere else).

For 2, what is a "shared script"? I couldn't find anything about that, anywhere. Is it a Gradle script? If it is, wouldn't it have the same problem as buildSrc, being built only _after_ settings.gradle[.kts] is already processed?

Overally, it feels like there is a bit of a gap here now. Maybe something else is needed, something that is specifically made to include custom logic into settings.gradle[.kts], and is built before it? But I suppose that it's somewhat of a chicken-and-egg problem... something needs to be the _first_ thing to be built, after all, so that would just move the problem?

My specific use-case right now

I wanted to have a file defined that contains all version numbers, metadata, and the like, for the project.
I know about gradle.properties, but it's too limited. I can't easily define lists or nested properties, and there's also no type-safety. I also wanted a bit of custom logic (mostly just concatenating some properties into a list), which is also not possible there of course.
Everything pointed me to buildSrc, so I used that and defined some Kotlin files there. I got everything I wanted. Except now I can't access any of that in settings.gradle.kts...
I would be fine with putting it somewhere else, as long as it's still a Kotlin file and not gradle.properties again... but it seems there is no simple way to do that at the moment.

[malachid]

Up to this point, that makes sense to me. But if settings.gradle[.kts] is basically the very first thing that is being evaluated/built in a project, then where should that custom logic come from, if I don't want to put it all right into the file itself...?
...
Overally, it feels like there is a bit of a gap here now. Maybe something else is needed, something that is specifically made to include custom logic into settings.gradle[.kts], and is built before it? But I suppose that it's somewhat of a chicken-and-egg problem... something needs to be the _first_ thing to be built, after all, so that would just move the problem?

I think that's the crux. Pre-6, the _first_ was buildSrc. 6 made a breaking change to the order of operations so that it was no longer first.

I'm a bit resigned to the fact I will have to copy/pasta everything to do with plugins from the buildSrc into the settings.gradle.kts and maintain two copies. This has, unfortunately, prevented us from migrating to 6 as it is a large tech debt (multiple repos to refactor) with extremely low priority. We know we eventually have to do it - but it is hard to justify at this point.

[floscher]

Note: This is only a solution for the plugin management part, not the part about including subprojects.

This is how I worked around this limitation:

I create a file ${rootProject.projectDir}/buildSrc/buildSrc/build.gradle.kts:

plugins {
  `kotlin-dsl`
}
repositories.jcenter()
sourceSets.main {
  java {
    setSrcDirs(setOf(projectDir.parentFile.resolve("src/main/kotlin")))
    include("VersionNumber.kt")
  }
}

This makes that the VersionNumber class is compiled one additional time before the buildSrc project is evaluated. I can then use that class in ${rootProject.projectDir}/buildSrc/build.gradle.kts.

I can then use the plugin versions in my ${rootProject.projectDir}/buildSrc/build.gradle.kts like this:

dependencies {
  implementation("plugin.group:plugin.module:${VersionNumber.PLUGIN}")
}

And in subprojects I can write the plugin definitions without version numbers:

plugins {
  id("plugin.id")
}

In fact Gradle will then even complain when I add .version("1.2.3").

See also Kotlin/dokka#1321 for an example of this approach.

[sellmair]

😮 Impressive workaround!

[jonecx]

Same case here. In order to have dependencies in one place defined inside buildSrc.
For examples Dependencies.kt where we define modules:

object Modules {
    const val app = ":app"
}

From settings.gradle.kts:

include (Modules.app)  -> ERROR: Unresolved reference: Modules
//include (":app")     -> Works but I do not want string literals here

I'm not expecting another API change if the decision has been made but at least some tips/tricks/pattern on how to handle this scenario.

Thank you very much ❤️

@android10 were you able get around this one 🤔?

[roths]

I am confuse.
This way can work:
settings.gradle code like this:

gradle.beforeProject { project ->
    if (project == project.rootProject) {
        project.with {
            apply from: "plugin.gradle"
        }
    }
}

plugin.gradle content like this:

project.with {
            buildscript.dependencies.with {
                classpath Deps.plugin
            }
}

Deps.kt file is in buildSrc dir.
But, when i put the logic from plugin.gradle into gradle.beforeProject
settings.gradle code like this:

gradle.beforeProject { project ->
    if (project == project.rootProject) {
        project.with {
            buildscript.dependencies.with {
                classpath Deps.plugin
            }
        }
    }
}

i will get an error: Could not get unknown property 'Deps'.
Could anyone explain it?

[floscher]

@roths You can't write Deps.plugin anywhere in your settings.gradlefile, because the Deps class is compiled and put on the classpath only after the settings.gradle script is evaluated.

The reason it works when you put it in a separate file and apply it inside a gradle.beforeProject{} block is, that the content of the plugin.gradle script is only evaluated as soon as the build.gradle scripts are ready to be evaluated. At that point the classes from your buildSrc/ directory are on the classpath and can be used in the plugin.gradle script.

But essentially I think your solution is equivalent to just writing

buildscript.dependencies.with {
  classpath Deps.plugin
}

at the beginning of your root build.gradle file.

---

Above in https://github.com/gradle/gradle/issues/11090#issuecomment-674798423 I describe a similar way of managing plugin versions. My approach defines the plugin versions in buildSrc/build.gradle.kts.

For your project it would look something like this:

(click this text to expand :arrow_up_down: )

Add the file ${rootProject.projectDir}/buildSrc/buildSrc/build.gradle.kts:

plugins {
  `kotlin-dsl`
}
repositories.jcenter()
sourceSets.main {
  java {
    setSrcDirs(setOf(projectDir.parentFile.resolve("src/main/kotlin")))
    include("Deps.kt")
  }
}

Add the following to ${rootProject.projectDir}/buildSrc/build.gradle.kts:

dependencies {
  implementation(Deps.plugin)
}

Then you can add plugins without version number:

plugins {
  id("plugin.id")
}