Kotlinx.coroutines: Add Support for Kotlin/Native

[ScottPierce]

Now that Kotlin-native supports multi-platform projects, this library can add support for Kotlin-native.

[LouisCAD]

@ScottPierce It's written Kotlin/Native ;-)

[elizarov]

That is something we'll be working on pretty soon. Stay tuned.

[ScottPierce]

Awesome! I assume that https://github.com/Kotlin/kotlinx.coroutines/issues/201 would also be resolved with that work as well? I found the lack of channels in the javascript side of this library fairly crippling, and it would be the same for kotlin-native.

[elizarov]

For the time being, brining basic coroutines (launch/async) to Kotlin/Native has more priority than bringing the channels everywhere. But it will be the next step. I expect that we'll be able to bring channels to both JS and Native in one swoop as soon as native basics are established.

[ScottPierce]

Is there any update on this? I've been working on an Android app, and splitting up code into common modules with the hopes that I can share code with ios via Kotlin/native. This is currently the only issue that I know of that's really stopping me from using app code and compiling our codebase for iOS.

[ScottPierce]

Work was progressing on this in this branch, but seems to have stagnated: https://github.com/Kotlin/kotlinx.coroutines/tree/mpp

[santoshastagi]

We are interested in this too. Any updates?

[kengura]

@ScottPierce If you are really starving to use coroutines in multi-platform kotlin project for Android and iOS you could take a look at demo project from mobius conference talk Multiplatform architecture with Kotlin for iOS and Android. Here is video in russian and presentation file in english

Code in common module looks like:

override suspend fun getTickersForAllMarkets(): List<Pair<Market, Ticker>> {
    return async(context) { repository.getAllMarkets() }
            .await()
            .subList(0, 20)
            .map { market ->
                market to async(context) {
                    repository.getTicker(market)
                }
            }
            .map { (market, job) ->
                market to job.await()
            }
}

To implement coroutines for iOS they declared the expect coroutines interfaces in common module which mapped to kotlinx.coroutines implementation in Android platform module and have custom actual implementation in iOS platform module.

I had EXC_BAD_ACCESS issues with AsyncDispatcher implementation on iOS. After replacement of dispatch_async with konan.worker.schedule problem was solved. Here is my version of AsyncDispatcher.

[ScottPierce]

@kengura Thanks for the recommendation. I'll give that a shot with my team.

[kpgalligan]

@kengura @ScottPierce I took a look at the demo implementation. It's very interesting. I'd love to have coroutines on iOS, but it's tricky. I'm guessing the reason kotlinx.coroutines is taking a while to materialize on native is due to state and threading rules.

I wouldn't put the worker-based solution into anything critical. Unless I really misunderstand the domain and how coroutines are routing, the worker implementation will eventually fail (or have memory leaks).

override fun <T> dispatchResume(value: T, continuation: Continuation<T>): Boolean {
        worker.schedule(TransferMode.UNCHECKED, {DataObject(value, continuation)}) {
            it.continuation.resume(it.value)
        }
        return true
    }

'value' and 'continuation' aren't frozen. If you used 'TransferMode.CHECKED' I assume it would blow up and tell you that. The issue you're going to have is reference counting in K/N is not atomic on regular data, but is atomic on frozen data. You'll eventually wind up with the same crash that you saw with the dispatch async example. I got that error, and in my case, it was trying to free memory that's already been freed. That's probably because its trying to move non-frozen data around to different threads.

In the worker example, at some point, dispatchResume will try to return while schedule block is starting, or some combination thereof, and you'll either have leaks or crashes for the same reason.

I think, anyway. I'd try hammering those calls hard in a test to see if it'll crash before putting the code in anything you'd ship.

As a general rule, if TransferMode.CHECKED is failing, you could be in trouble.

[kengura]

@kpgalligan Yes you are right. This problem stops me from using such solution anywhere except test project. But someone could find better approach.

[ScottPierce]

I've noticed that Kotlin/native's threading model makes quite a few things more difficult. I wonder if raw shared memory could be used as a solution.

[qwwdfsad]

Late update: we're working on this.
Current roadmap is:
1) Release single-threaded K/N coroutines (available via runBlocking) soon. mpp branch already has a working prototype, we're waiting for toolchain upgrade to release it
2) Provide analogue of UI dispatcher for iOS. Still single-threaded, but asynchronous
3) Start working on multithreaded coroutines. This is where K/N threading model kicks. Raw shared memory cannot be a solution, because it will make kotlinx.coroutines public API very error-prone and counterintuitive.

[ScottPierce]

@qwwdfsad The latest kotlin/native blog post says:

Other important JetBrains-provided libraries, like kotlinx.coroutines, kotlinx.serialization, and Ktor HTTP client are getting experimental Kotlin/Native support, which means it is possible to write an iOS app and Android application sharing the same REST API-related logic.

I'm not seeing anything like this for kotlinx.couroutines yet. Can you point me in the right direction?

[elizarov]

We are still working on proper publishing. There is a test version 0.23.4-native-1 that was deployed to Bintray with kotlinx.coroutines for all the Mac OS and iPhone targets, but no Linux and Windows yet.

[brettwillis]

There is a test version 0.23.4-native-1 that was deployed to Bintray with kotlinx.coroutines for all the Mac OS and iPhone targets, but no Linux and Windows yet.

I am trying to use org.jetbrains.kotlinx:kotlinx-coroutines-core-native:0.23.4-native-1 in a Kotlin/Native project targeting iOS. It seems that I have to manually add the kotlinx-coroutines-core-native_release_ios_x64 etc. dependencies, but I can't get around Konan compiler error IllegalStateException: Could not find "atomicfu-native" (even after also manually adding atomicfu-native dependencies).

Are you able to point me to to an example showing how to correctly set up these coroutines dependencies in a Konan project?

[elizarov]

You'll need the recent version of Gradle and you should use new DSL for dependencies declarations like here: https://github.com/Kotlin/kotlinx.coroutines/blob/native/gradle/compile-native.gradle and your Kotlin/Native version shall exactly match the one that was used to build that library: https://github.com/Kotlin/kotlinx.coroutines/blob/native/gradle.properties

[AregevDev]

Will it be available for Windows/Linux/Android/Wasm soon?

[brettwillis]

Hi @elizarov, I followed your instructions, but unfortunately it doesn't seem to change anything. As I don't wish to pollute this thread, perhaps you could help me over here:
https://stackoverflow.com/questions/51494681/how-to-build-kotlinx-coroutines-in-kotlin-native-test-version-0-23-4-native-1

Thanks :)

[elizarov]

@brettwillis It is coming. You can check the docs on using in Kotlin/Native project here (the compatible build is not released yet, but you can already build it yourself from develop branch): https://github.com/Kotlin/kotlinx.coroutines/blob/develop/native/README.md

[elizarov]

@AregevDev Windows/Linux/iOS/MacOS/Android is coming, but no Wasm support is planned in the short term.

[luca992]

@elizarov I am able to use coroutines in a kotlin native program using the new DSL. But I see that the new plugin does not support setting the output kind to generate a dynamic library (.so/.dylib/*.dll).

Since the new plugin isn't an option a the moment, is there anyway to use coroutines with the 'konan' plugin so I can generate a dynamic library?

adding:

dependencies { implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-core-native:0.24.0' }

with apply plugin: 'konan' causes

Could not find method implementation() for arguments [org.jetbrains.kotlinx:kotlinx-coroutines-core-native:0.24.0] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

[r4zzz4k]

@luca992 .so artifacts produced by dynamic library output are mostly useful for Kotlin/Native libraries which export API consumable from C language (or others via libffi, for example). kotlinx.coroutines is expected to be used by Kotlin code, so the natural way to go is with .klib artifacts, which this project actually generates for usage by Kotlin/Native apps.

[luca992]

@r4zzz4k Yes I understand, but I am not planning on interacting with the coroutines in C. I plan on calling a kotlin function from C. Have kotlin perform an http request using curl. Then once it gets a result, pass the data to C through an interface.

I'm not sure if I'm being clear.... but basically, coroutines will only be used internally and I just need to be able to call some kotlin functions from C code

[r4zzz4k]

@luca992 okay, now I re-read your message and found JetBrains/kotlin-native#1877, I see that you are asking for klib artifacts produced for kotlinx.coroutines by the old plugin so that you can consume it by your code which outputs so artifacts via the old plugin. Are klib artifacts produced by both plugins incompatible with each other?
Sorry for misunderstanding!

[luca992]

@r4zzz4k I do not have access to a .klib for kotlix.coroutines. Are you saying I could clone this project and build a klib for my to project use via konanArtifacts?

[r4zzz4k]

@luca992 I didn't really try to use native version of kotlinx.coroutines yet, but there are native binaries published via new plugin, which can be found at project BinTray page, klibs can be found among platform-specific artifacts. I have no idea if just putting several klib files as a dependencies would be enough or some additional work is needed, but at least you can try.

[elizarov]

The kotlinx.coroutines artifacts are not compatible with the old konan plugin. You shall use kotlin-platform-native. See readme here: https://github.com/Kotlin/kotlinx.coroutines/blob/master/native/README.md

You should be able to produce a .so library from your code by setting outputKinds = [KLIBRARY] and then you'll be able to use the result from your C code.

[luca992]

@elizarov outputKinds = [KLIBRARY] produces a *.klib with no header file

[luca992]

@r4zzz4k your suggestion to use the klib kotlinx.coroutines allowed me to output a dynamic library using the konan plugin. So until there is support for output of a dynamic library with the 'kotlin-platform-native' plugin. This will have to do.

[elizarov]

You are right. You'll need additional compiler options with kotlin-platform-native plugin:

sourceSets.main.component {
    outputKinds = [ KLIBRARY ]
    extraOpts '-p', 'dynamic', '-module_name', baseName.get()
}

[luca992]

@elizarov That works too. Thanks for the help

[steve-the-edwards]

Hi @elizarov , I used kotlinx.coroutines for my shared library project, with the include as
api "org.jetbrains.kotlinx:kotlinx-coroutines-core-native:$coroutines_version"

Where version was 1.2.1

The common code co-routines work fine from Android.

This produced a .framework successfully for use in iOS. However, all the suspend functions are not put in as attributes in the header, and thus not available in Swift or ObjC. Does this still require an actual implementation on the iOS side of things?

How can this be used in iOS? Or do we need to have an async wrapper in the common code as well strictly for iOS usage?

[LouisCAD]

Hi @steve-the-edwards, Kotlin is the only programming language that supports suspending functions for now, so you need to write callback wrappers if you want to call those from other programming languages like Swift or Objective-C. This code can be in an iOS specific source set, no need to pollute the common code with this.

[steve-the-edwards]

Thanks @LouisCAD yes that makes sense. I wondered if these wrappers had been done in a generic way as part of the kotlinx.coroutines-native library.