Gradle: Upgrade embedded Kotlin to 1.4

[eskatos]

The first preview version 1.4-M1 is out
https://blog.jetbrains.com/kotlin/2020/03/kotlin-1-4-m1-released/
https://github.com/JetBrains/kotlin/releases/tag/v1.4-M1

Gradle 7.0 could be a good target for Kotlin 1.4.

A good first step would be to set up a branch integrating early 1.4 previews in order to discover problems, breakages or regressions.

Things to take into considerations

• https://youtrack.jetbrains.com/issue/KT-38070

[bamboo]

It seems we might be able to use a more regular syntax for collection elements in the Kotlin DSL when we upgrade:

tasks {
    val foo: FooTask by existing // take Task type from val

    val bar: BarTask by registering { // take Task type from val
        gazonk = 42
    }
}

[turansky]

@bamboo Is it possible only in Kotlin 1.4?

[vlsi]

val foo: FooTask by existing // take Task type from val

Does that defeat configuration avoidance?

[bamboo]

@bamboo Is it possible only in Kotlin 1.4?

My comment was triggered by the _Better inference for delegated properties_ section from the 1.4-M1 announcement.

Maybe that improvement is already available in Kotlin 1.3.x but I haven't checked.

[bamboo]

val foo: FooTask by existing // take Task type from val

Does that defeat configuration avoidance?

Not necessarily. The underlying delegate could still hold a TaskProvider<FooTask> as the currently implementation does.

[vlsi]

Well, I just thought val foo: FooTask means foo has to be FooTask.
At least, from the use perspective, user could be able to write something like foo.enabled = false.

Do you mean it would return a proxy that delegates configuration actions to TaskProvider?

[bamboo]

@vlsi, for delegated properties the contract is a little more complex:

val delegatedFoo: FooType by FooDelegate()
val foo: FooType = delegatedFoo

// roughly translates to:
val delegatedFoo = FooDelegate().provideDelegate(...)
val foo: FooType = delegatedFoo.getValue(...)

So within that framework, FooDelegate().provideDelegate(...).getValue(...) must be assignable to FooType but the compiler doesn't care about the types of the intermediate expressions as long as the combined whole type checks.

See https://kotlinlang.org/docs/reference/delegated-properties.html for more details.

[vlsi]

So within that framework, FooDelegate().provideDelegate(...).getValue(...)

AFAIK every read of delegatedFoo translates to getValue(..).
That getValue(..) would have to translate to TaskProvider<..>#get()

I agree

    val bar: BarTask by registering { // take Task type from val
        gazonk = 42
    }

Can work and support configuration avoidance. However, I guess users would want to write code as follows:

    val bar: BarTask by registering { // take Task type from val
        gazonk = 42
    }

    tasks.jar {
        dependsOn(bar) // <-- bar would compile here to an expression of BarTask type
        // for instance, .getValue(...) which would have to return BarTask
    }

That is why I think dependsOn(bar) would break avoidance.
Am I missing something?

PS. I have written delegated properties for Gradle plugins, so I know a bit how it works.

[bamboo]

tasks.jar {
        dependsOn(bar) // <-- bar would compile here to an expression of BarTask type
        // for instance, .getValue(...) which would have to return BarTask
}

That's correct, the bar task would be realized at that point (dependsOn(bar)) but notice that only happens if jar itself is realized.

[turansky]

Is there any simple possibility to replace embedded-kotlin in existed Gradle dist (6.3 for example)?

[turansky]

Related ticket - KT-38010

[vRallev]

What's the target release of this issue? Kotlin 1.4.0 was released today (or the next days) and it's not possible to use the newer version for Gradle plugins even though the Kotlin DSL isn't used.

Edit: Can we assume 6.7? #14147

[SerVB]

Trailing commas available in 1.4 will be great to use in build scripts. Waiting!

[eskatos]

https://github.com/gradle/gradle/pull/14147 is just about smoke testing the usage of the Kotlin Gradle Plugin v1.4 to build production/test sources without updating the Gradle embedded Kotlin to 1.4. A little step forward.

[cesar1000]

Not entirely proud of this hack but I guess it's not too bad since the smoke test of 1.4 was successful :)

fun DependencyHandler.removeKotlin(configuration: Configuration, name: String) {
    val dependency = configuration.dependencies
        .filterIsInstance<FileCollectionDependency>()
        .find { it.files.toString() == name }
    if (dependency != null) {
        configuration.dependencies.remove(dependency)
        val files = dependency.resolve().filter { !it.name.startsWith("kotlin-") }
        add(configuration.name, serviceOf<FileCollectionFactory>().fixed(name, files))
    }
}

dependencies {
    implementation(kotlin("stdlib-jdk8"))
    implementation(kotlin("reflect"))

    configurations.all {
        removeKotlin(this, "gradleKotlinDsl")
        removeKotlin(this, "Gradle API files")
        removeKotlin(this, "Gradle TestKit files")
    }
}

This will remove all warnings and enable the use of the Kotlin libraries. The build dependencies still use 1.3.72, since there's some constraint pinning them to the embedded compiler.

[martinbonnin]

Any chance we can get a 7-dev build with Kotlin 1.4? That could unblock the situation for user wanting to upgrade their builds to 1.4 and also give some opportunity to test the change before it is released?

[eskatos]

For those interested, here's a snapshot build of Gradle with the embedded Kotlin upgraded to 1.4.10:
6.8-branch-eskatos_kotlin_embedded_1_4-20200922125722+0000.

The .gradle.kts scripts still use Kotlin language and API 1.3.

You can upgrade your wrapper with the following:

./gradlew wrapper --gradle-version=6.8-branch-eskatos_kotlin_embedded_1_4-20200922125722+0000

Or download the distribution at:
https://services.gradle.org/distributions-snapshots/gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200922125722+0000-bin.zip

:warning: Note that the version number is 6.8-something but that's not a commitment to ship it with Gradle 6.8. This distribution is unsupported at this stage.

With this build we would like to gather feedback on how it works for you.
Please share both successes and failures.

[SerVB]

The .gradle.kts scripts still use Kotlin language and API 1.3.

How the change can be noticed then?

[eskatos]

@SerVB no more classpath conflict when building Gradle plugins / buildSrc with the Kotlin Gradle Plugin v1.4.

We can't move scripts to language/API 1.4 in a minor Gradle version as it could break builds.
The snapshot distro above is a first step, we'll get there.

[ZacSweers]

Why not require users to upgrade their scripts? I think Gradle is actively (even if unintentionally) limiting community adoption of a newer kotlin version with this model. What happens if the next minor release of kotlin happens right after a major gradle version? Would we need to wait a year?

Plenty of other significant tools using kotlin updated readily without the ecosystem collapsing, people with projects that are for some reason not ready to update can simply just hold off on updating.

[Vampire]

Unfortunately that's not how backwards compatibility guarantees work.

Gradle guarantees that you can use a newer minor version without the build breaking and that you can use a newer major version without the build breaking if you didn't have deprecation warnings in the latest minor version of the previous major version. (Except for edge cases or usage of internals)

[eskatos]

@ZacSweers, Gradle is used to run builds of projects that started a decade ago (and more), or in very conservative environments. Our users expect no breakage from the core in minor versions.

That question is a very good one:

What happens if the next minor release of kotlin happens right after a major gradle version?

We currently do ~1 major a year but we have some latitude and can adapt. We work with Kotlin folks and are aware of their schedule.

Kotlin 1.4.0 was released the 17th of August. that's just one month ago.
Gradle 7.0 should happen relatively soon. Just a little patience :)

FWIW, Groovy which is embedded the same way as Kotlin has released 3.0 almost 10 months ago, we'll probably upgrade the embedded one for 7.0 too.

[ZacSweers]

I can understand not forcing a newer version for everyone, but what I cannot understand is why Gradle actively prevents developers that _are_ prepared to upgrade from doing so. The only explanation I've seen is "because Gradle touches compiler internals that might break", which suggests it's conservative to your own toolchain rather than protecting external projects. This limitation isn't imposed on, say, Java versions. Gradle 6.7 adds the ability to run on Java 15.

I would be totally fine with Gradle saying "we don't officially support kotlin 1.4 but please report any issues you do see".

[Vampire]

The difference probably is, that Groovy and Kotlin are embedded where you provide the Java version externally. I guess the integration would need to be changed to be able to provide the Kotlin and Groovy versions externally too.

I'm confident that a PR that properly adds this possibility would be accepted. Challenge accepted? ;-)

[eskatos]

I would be totally fine with Gradle saying "we don't officially support kotlin 1.4 but please report any issues you do see".

What happens currently if you mix Kotlin versions in your build logic is that you get a warning about just that. Until it fails when something incompatible gets used. Did you expect something else?

[ZacSweers]

Currently it forces 1.3.x (whatever version Gradle bundles) _and_ I get a ton of warnings about mismatched file versions

[gildor]

Did you expect something else?

I would at least expect flag to suppress warnings about incompatible versions

What even worse in the current situation is that it affects how Gradle build logic is used. If you follow Gradle best practices, have buildSrc/included build, and have, let's say, convention plugin for Kotlin modules, you got a bunch of warnings because you need Kotlin plugin in dependencies, but if you just put all your build logic to build.gradle and applied new version of Kotlin plugin, you have no warnings, just because Kotlin plugin is not in build classpath.

I'm not sure that it's a healthy situation. I can understand backward compatibility concerns or lack of official support, but I think we need a way to use newer versions of Kotlin (maybe just allow suppressing version check warnings)

[eskatos]

Agreed @gildor.

FWIW, the pinning to 1.3 is done by the kotlin-dsl plugin and the warning is emitted by the Kotlin Gradle Plugin. If the warning is the most annoying thing we could consider adding a way to disable the pinning in the kotlin-dsl plugin. It would remove the warning but the embedded version of stdlib would still be used at runtime.

Fixing this situation properly would require deep classloading changes in Gradle, not saying it's not possible. buildSrc being special in this regard is getting in the way somehow. Using included builds for build logic will provide an easier path to proper fixes.

---

Back to upgrading to 1.4. I didn't read any success/failure report on the snapshot distribution shared above yet. Got a success report on Slack though. Anyone?

[Mahoney]

Back to upgrading to 1.4. I didn't read any success/failure report on the snapshot distribution shared above yet. Got a success report on Slack though. Anyone?

Just tried it - worked perfectly.

(Though the deprecation to mainClassName has stuffed up my custom application plugin... ApplicationPluginConvention doesn't provide access to the new mainClass property)

[xsveda]

Success report: Kotlin Multiplatform Project (Android & iOS), build logic in buildSrc using Kotlin DSL, all upgraded to Kotlin 1.4 and we don't see any warning anymore with the snapshot build -> ALL GOOD!

[minyushov]

Back to upgrading to 1.4. I didn't read any success/failure report on the snapshot distribution shared above yet. Got a success report on Slack though. Anyone?

I can't build an Android project with included build with this snapshot. It's probably not related to Kotlin 1.4 but still

Could not create task ':core:api:generateDebugDependenciesForLint'.
> The root project is not yet available for build.

Caused by: java.lang.IllegalStateException: The root project is not yet available for build.
        at org.gradle.invocation.DefaultGradle.getRootProject(DefaultGradle.java:220)
        at org.gradle.invocation.DefaultGradle_Decorated.getRootProject(Unknown Source)
        at org.gradle.invocation.DefaultGradle.getRootProject(DefaultGradle.java:73)
        at com.android.build.gradle.internal.ide.dependencies.BuildMappingUtils.computeBuildMapping(BuildMapping.kt:63)
        at com.android.build.gradle.internal.lint.LintModelDependenciesWriterTask$CreationAction.configure(LintModelDependenciesWriterTask.kt:191)
        at com.android.build.gradle.internal.lint.LintModelDependenciesWriterTask$CreationAction.configure(LintModelDependenciesWriterTask.kt:159)
        at com.android.build.gradle.internal.tasks.factory.TaskConfigurationActions.execute(TaskFactoryUtils.kt:94)
        at com.android.build.gradle.internal.tasks.factory.TaskConfigurationActions.execute(TaskFactoryUtils.kt:80)

All is working with Gradle 6.6.1.
I'm using Android Gradle Plugin 4.2.0-alpha12.

[martinbonnin]

Warning's gone here too with 6.8-branch-eskatos_kotlin_embedded_1_4-20200922125722+0000 👍 .

About this:

We can't move scripts to language/API 1.4 in a minor Gradle version as it could break builds.

I'm curious why this is the case. Shouldn't code compiled with Kotlin 1.3 and using Kotlin 1.3 compile and behave the same with Kotlin 1.4? If it weren't the case, upgrading any project to 1.4 would be super painful.
Would you have an example of where moving scripts to language/API 1.4 would break existing builds?

[SerVB]

Would you have an example of where moving scripts to language/API 1.4 would break existing builds?

I guess some constructions deprecated in Kotlin 1.3 can be removed in 1.4. So if anyone used those deprecated symbols, after updating to 1.4 they would get compilation errors.

For example, old coroutines package is removed from stdlib in 1.4, so technically if somebody used it, updating to 1.4 would break the build...

So the possibility of breaking is real.

[martinbonnin]

For example, old coroutines package is removed from stdlib in 1.4, so technically if somebody used it, updating to 1.4 would break the build...

Got it 👍 . Thanks for the pointers!

[xsveda]

Success report: Kotlin Multiplatform Project (Android & iOS), build logic in buildSrc using Kotlin DSL, all upgraded to Kotlin 1.4 and we don't see any warning anymore with the snapshot build -> ALL GOOD!

@eskatos I need to update my report:
It looks that gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200922125722+0000 version has a huge impact on Android Gradle Plugin's Lint task.

• With AGP 4.1.0-rc03 and Gradle 6.6.1 the Lint task finished in 1m45s.
• With AGP 4.1.0-rc03 and Gradle 6.7-rc-1 the Lint task finished in 1m43s.
• With AGP 4.1.0-rc03 and Gradle gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200922125722+0000 it did not finished in 20m after which I've killed the process. It got "stuck" on:

> :android:app:lintVitalRelease
> :android:app:lintRelease > Resolve files of :android:app:debugRuntimeClasspath

[eskatos]

Thank you all for the feedback so far!

@xsveda there are changes that can cause locking contention and possibly a dead lock with lint. This is a known issue that is being fixed, see https://github.com/gradle/gradle/issues/14516

@minyushov, I believe your issue with lint is also related to the above and not to the Kotlin upgrade

[eskatos]

@xsveda, @minyushov, could you please give it a try with 6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000?

[minyushov]

@eskatos, it's working now 👍 .

[xsveda]

@eskatos Android Lint also seems to be working fine with 6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000 👍

[ZacSweers]

Were the artifacts deleted? I'm unable to help test this

Exception in thread "main" java.io.FileNotFoundException: https://downloads.gradle-dn.com/distributions/gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000-all.zip
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
    at org.gradle.wrapper.Download.downloadInternal(Download.java:87)
    at org.gradle.wrapper.Download.download(Download.java:67)
    at org.gradle.wrapper.Install$1.call(Install.java:68)
    at org.gradle.wrapper.Install$1.call(Install.java:48)
    at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
    at org.gradle.wrapper.Install.createDist(Install.java:48)
    at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
    at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)

[xsveda]

@ZacSweers: This one works for me:
https://services.gradle.org/distributions-snapshots/gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000-bin.zip

[ZacSweers]

How are you updating? The wrapper task doesn't like this

'gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000' is not a valid Gradle version string (examples: '1.0', '1.0-rc-1')

[soberich]

@ZacSweers

./gradlew wrapper \
  --gradle-version=6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000 \
  --distribution-type=all \
  --warning-mode=all \
  --gradle-distribution-sha256-sum="$(curl https://services.gradle.org/distributions-snapshots/gradle-6.8-branch-eskatos_kotlin_embedded_1_4-20200929110025+0000-all.zip.sha256 -Ls | sed '/^[[:space:]]*$/d')" \
  -S \
  --info \
  --no-daemon

[ZacSweers]

Thanks. Our project unfortunately doesn't appear to compile with this branch

public final fun android(action: Action<AndroidHandler>): Unit defined in slack.gradle.SlackExtension'
e: /Users/zsweers/dev/slack/slack-android-ng/app/build.gradle.kts:294:5: Unresolved reference: app
e: /Users/zsweers/dev/slack/slack-android-ng/app/build.gradle.kts:295:7: Unresolved reference: permissionAllowlist
e: /Users/zsweers/dev/slack/slack-android-ng/app/build.gradle.kts:297:11: Unresolved reference: setAllowlistFile

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/zsweers/dev/slack/slack-android-ng/app/build.gradle.kts' line: 294

* What went wrong:
Script compilation errors:

  Line 294:     app {
                ^ Unresolved reference: app

  Line 295:       permissionAllowlist {
                  ^ Unresolved reference: permissionAllowlist

  Line 297:           setAllowlistFile(file("permissionsAllowlist.txt"))
                      ^ Unresolved reference: setAllowlistFile

3 errors

I don't really see anything wrong with it though. We define this extension in buildSrc

@DslMarker
annotation class SlackExtensionMarker

@SlackExtensionMarker
abstract class SlackExtension @Inject constructor(objects: ObjectFactory) {

  internal val androidHandler = objects.newInstance<AndroidHandler>()

  fun android(action: Action<AndroidHandler>) {
    action.execute(androidHandler)
  }
}

@SlackExtensionMarker
@Suppress("UnnecessaryAbstractClass")
abstract class AndroidHandler @Inject constructor(objects: ObjectFactory) {
  internal val appHandler = objects.newInstance<SlackAndroidAppExtension>()

  fun app(action: Action<SlackAndroidAppExtension>) {
    action.execute(appHandler)
  }
}

@SlackExtensionMarker
abstract class SlackAndroidAppExtension {
  internal var allowlistAction: Action<PermissionAllowlistConfigurer>? = null

  /**
   * Configures a permissions allowlist on a per-variant basis with a VariantFilter-esque API.
   *
   * Example:
   * ```
   * slack {
   *   permissionAllowlist {
   *     if (buildType.name == "release") {
   *       setAllowlistFile(file('path/to/allowlist.txt'))
   *     }
   *   }
   * }
   * ```
   */
  fun permissionAllowlist(factory: Action<PermissionAllowlistConfigurer>) {
    allowlistAction = factory
  }

  interface PermissionAllowlistConfigurer : VariantConfiguration {
    /**
     * Sets a file containing a newline-delimited allowlist of permissions. If set, merged manifest permissions
     * for this variant will have their permissions checked against the allowlist defined in [file].
     */
    fun setAllowlistFile(file: File)
  }

  internal class DefaultPermissionAllowlistConfigurer(
    variant: ApplicationVariant
  ) : PermissionAllowlistConfigurer, VariantConfiguration by DefaultVariantConfiguration(variant) {
    internal var file: File? = null

    override fun setAllowlistFile(file: File) {
      this.file = file
    }
  }
}

/** Base interface for interfaces that configure on a per-variant basis and need to expose variant info. */
interface VariantConfiguration {
  /**
   * Returns the Build Type.
   */
  val buildType: BuildType

  /**
   * Returns the list of flavors, or an empty list.
   */
  val flavors: List<ProductFlavor>

  /**
   * Returns the unique variant name.
   */
  val name: String
}

private class DefaultVariantConfiguration(private val variant: ApplicationVariant) : VariantConfiguration {
  override val buildType: BuildType get() = variant.buildType
  override val flavors: List<ProductFlavor> get() = variant.productFlavors
  override val name: String get() = variant.name
}

app/build.gradle.kts

slack {
  android {
    app {
      permissionAllowlist {
        if (name == "externalRelease") {
          setAllowlistFile(file("permissionsAllowlist.txt"))
        }
      }
    }
  }
}

[eskatos]

@ZacSweers this looks weird indeed. Given the snippets you pasted I don't see what would cause this at first sight.

Could you please try again with 6.8-20201005220043+0000?

If it fails the same way, please reach out to me on the community slack, we could do a pairing session to understand what's happening as I understand you can't share your full project here.

[eskatos]

@ZacSweers I tried to reproduce by creating a build. You can find it at https://github.com/eskatos/kt-1.4-zac-issue

It works for me with both Gradle 6.6 that embeds Kotlin 1.3 and Gradle 6.8 nightly that embeds Kotlin 1.4.
I added you as a collaborator of the repo in case you find out what the difference is with your build and want to enhance the reproducer project.

I used https://services.gradle.org/distributions-snapshots/gradle-6.8-20201007220043+0000-bin.zip

[gildor]

Kotlin Team announced new release cycle for Kotlin with regular 6 month releases:
https://blog.jetbrains.com/kotlin/2020/10/new-release-cadence-for-kotlin-and-the-intellij-kotlin-plugin/#commento-login-box-container

What does it mean for Gradle and Kotlin DSL? Would be great to hear some official plans from Gradle team.

Should we expect that Gradle releases would always include latest stable version of Kotlin or maybe it would be possible to update it manually (with no warranty of work of course)

[eskatos]

The embedded Kotlin has been upgraded to 1.4.10 for Gradle 6.8.

The problem reported by Zac mentioned above has been fixed.
Note that 6.8-milestone-1 doesn't have the fix for that problem.

You can test using 6.8-20201015220041+0000.

Note that .gradle.kts scripts still use API and language 1.3. They will be updated to 1.4 in Gradle 7.0, see https://github.com/gradle/gradle/issues/14888

[eskatos]

@gildor it doesn't change much. Upgrading the embedded Kotlin will still be a case by case decision following our backwards compatibility policy depending on the content of the Kotlin rolling releases.