Ktor: Ktor Client - Downloading large files with ByteReadChannel

[flixlo]

Hello guys,

I'm developing a Kotlin Multiplatform app for Android and iOS. The app should be able to download large files via HTTP by writing the data directly into a local file. The following code works perfectly for small files (I tried it with files smaller than 1 MB).

fileWriter.initialize()

val httpRequest = HttpRequestBuilder().apply {
    url("…")
    header("Authorization", "Bearer …")
}

httpClient.get(httpRequest).execute { response: HttpResponse ->
    val bytes = response.receive<ByteReadChannel>()

    val byteBufferSize = 1024 * 100
    val byteBuffer = ByteArray(byteBufferSize)

    do {
        val currentRead = bytes.readAvailable(byteBuffer, 0, byteBufferSize)

        if (currentRead > 0) {
            println("Write ${currentRead} bytes...")

            // write the data into the file
            fileWriter.writeToFile(if (currentRead < byteBufferSize) {
                byteBuffer.slice(0 until currentRead)
            } else {
                byteBuffer
            })
        }
    } while (currentRead >= 0)

    fileWriter.finish()
}

I tried to download a 2 GB file, but the app is stuck inhttpClient.get(httpRequest) until the RAM usage is too high and the OS decides to kill my app. I would expect that response.receive<ByteReadChannel>() streams the content of the request and I can read it step by step with bytes.readAvailable(…), but the code actually never reaches bytes.readAvailable(…).

Why is httpClient.get(httpRequest) downloading the 2 GB intead of streaming the data? How would you approach this and how can you download a large file without filling the RAM with the Ktor client?

Thank you! :)

Edit:
I tested the described scenario only on iOS so far.
Ktor version: 1.3.1
Kotlin version: 1.3.61

[gvdhoven]

Isn't this related to #1636 ?

[flixlo]

Im not sure. In the linked issue, the user can't download files larger than 4096 bytes. In my case, I tried downloading files with 1 MB without issues. Only very large files cause issues (e.g. 2 GB), because the RAM overflows as described above.

Edit: I double checked and tried it with a 30 MB file. The code ist stuck at httpClient.get(httpRequest) for about 30 seconds before it continues to read the bytes with bytes.readAvailable(…).

[dmitrievanthony]

Hi @flixlo, thanks for reporting.

I prepared the following code to reproduce the problem, but it works fine as far as I see. Could you please also try it?

suspend fun main() {
    embeddedServer(Netty, port = 8763) {
        routing {
            get("/get") {
                val response = ByteArray(1024 * 1024) // 1MB
                call.respond(object : OutgoingContent.WriteChannelContent() {
                    override val contentType = ContentType.Application.OctetStream
                    override suspend fun writeTo(channel: ByteWriteChannel) {
                        var written = 0
                        repeat(1024) { // 1MB * 1024 = 1GB
                            channel.writeFully(response, 0, response.size)
                            channel.flush()
                            written += response.size
                        }
                        println("Total written: $written bytes")
                    }
                })
            }
        }
    }.also { it.start(false) }

    HttpClient(CIO).also { client ->
        val bytes = client.get<ByteReadChannel>("http://localhost:8763/get")

        val byteBufferSize = 1024 * 100
        val byteBuffer = ByteArray(byteBufferSize)

        var read = 0

        do {
            val currentRead = bytes.readAvailable(byteBuffer, 0, byteBufferSize)
            if (currentRead > 0) {
                read += currentRead
            }
        } while (currentRead >= 0)

        println("Total read: $read")
    }
}

[flixlo]

Hi @dmitrievanthony,

thanks for your response. The code you suggested works properly.

I did some further investigations: The code I posted above works properly on Android aswell. If I try to run my code on iOS, the described problem still occurs. Can you confirm that your solution works on a iOS client?

My config:

    sourceSets {
        commonMain.dependencies {
            // [...]

            api "io.ktor:ktor-client-core:$ktorVersion"
            api "io.ktor:ktor-client-serialization:$ktorVersion"
            api "io.ktor:ktor-client-json:$ktorVersion"
        }

        androidMain.dependencies {
            // [...]

            api "io.ktor:ktor-client-okhttp:$ktorVersion"
            api "io.ktor:ktor-client-core-jvm:$ktorVersion"
            api "io.ktor:ktor-client-serialization-jvm:$ktorVersion"
            api "io.ktor:ktor-client-json-jvm:$ktorVersion"
        }

        iosMain.dependencies {
            // [...]

            api "io.ktor:ktor-client-ios:$ktorVersion"
            api "io.ktor:ktor-client-core-native:$ktorVersion"
            api "io.ktor:ktor-client-serialization-native:$ktorVersion"
            api "io.ktor:ktor-client-json-native:$ktorVersion"
        }
    }

[dmitrievanthony]

As far as I remember, native clients don't implement streaming so far, they download the whole response at first and only after that return it to you. As a result, if your response is big it will be fully loaded into the memory and may cause OOM.

Hi @cy6erGn0m, @e5l, please correct me if I'm wrong.

[e5l]

Streaming support and download is fixed in master. The fix will be available with the next minor release.