Ktor: [Question] CPU 100% Apache Ktor Http Client
Ktor Version
1.1.3
Ktor Engine Used(client or server and name)
Apache
JVM Version, Operating System and Relevant Context
Feedback
I utilize ktor http client to communicating from service to other service, but after 10 minutes deployment my machine experience CPU 100%, any idea why ?
here's the building block that I use,
override fun sendAsync(request: HttpRequest, responseCallback: Callback<HttpResponse>) {
GlobalScope.launch (Dispatchers.IO + CoroutineName(key ?: "")) {
try {
decorateRequest(request)
val requestUrl = URL(request.url)
val requestBody = String(request.body.asArray(), StandardCharsets.UTF_8)
val requestMethod = HttpMethod(request.method)
val requestContentType = createContentType(request)
val clientResponse = httpClient!!.request<io.ktor.client.response.HttpResponse>{
method = requestMethod
url(requestUrl)
body = TextContent(requestBody, requestContentType)
request.headers.forEach {
if (!unsafeHeaderSet.contains(it.name)) {
header(it.name, it.value)
}
}
}
val response = HttpResponse()
val responseBodyString = String(clientResponse.readBytes())
response.statusCode = clientResponse.status.value
response.bodyEncoding = "UTF-8"
response.body = IndirectNIOBuffer(ByteBuffer.wrap(responseBodyString.toByteArray(charset("UTF-8"))), true)
clientResponse.headers.forEach { name, value ->
val headerValue = value.joinToString(",")
response.addHeader(name, headerValue)
}
clientResponse.close()
responseCallback.onComplete(response)
} catch (error: Exception) {
responseCallback.onException(error)
}
}
}
perf problem
davidasync
All 23 comments
Hi @davidasync, could you reproduce the problem with 1.2.0?
e5l
on 27 May 2019
Experience the same problem in https://github.com/Heapy/kotbot/commit/bb85b688871e4ab641c668dbeaf800dfeea74255
Kotlin: 1.3.61
Ktor: 1.2.6
Coroutines: 1.3.2
cpu shanpshot from visualvm (sorry, not yourkit on this machine, but I can try with yourkit later if you wish)
kotbot.zip
Ubuntu 16.04 (Server), Docker, Amazon OpenJDK 11.0.5
Fedora 30 (Desktop), Amazon OpenJDK 11.0.5
IRus
on 13 Dec 2019
Looks like some infinite loops running, because of this jump in CPU
IRus
on 13 Dec 2019
First request to server at 2:49PM, and later one core is occupied without any reason (12%, I have 8 hyper threads)
IRus
on 13 Dec 2019
Same application on OpenJDK 1.8.0_212:
IRus
on 13 Dec 2019
Upgrade to kotlinx.coroutines 1.3.3 does not help
IRus
on 13 Dec 2019
👀1
@e5l any thought on that? I think issue related to SSL/TLS stack on JDK11
IRus
on 13 Dec 2019
Yes, it's probably. We're facing the same issue internally. Could you post the stack trace to verify this?
e5l
on 13 Dec 2019
In particular moment I got blocked thread:
DefaultDispatcher-worker-8 Blocked CPU usage on sample: 0ms
org.apache.http.nio.reactor.ssl.SSLIOSession.setEvent(int) SSLIOSession.java:672
org.apache.http.impl.nio.NHttpConnectionBase.requestOutput() NHttpConnectionBase.java:358
org.apache.http.impl.nio.conn.CPoolProxy.requestOutput() CPoolProxy.java:120
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionAllocated(NHttpClientConnection) AbstractClientExchangeHandler.java:334
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$000(AbstractClientExchangeHandler, NHttpClientConnection) AbstractClientExchangeHandler.java:62
org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.completed(NHttpClientConnection) AbstractClientExchangeHandler.java:387
org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.completed(Object) AbstractClientExchangeHandler.java:383
org.apache.http.concurrent.BasicFuture.completed(Object) BasicFuture.java:123
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.completed(CPoolEntry) PoolingNHttpClientConnectionManager.java:306
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.completed(Object) PoolingNHttpClientConnectionManager.java:297
org.apache.http.concurrent.BasicFuture.completed(Object) BasicFuture.java:123
org.apache.http.nio.pool.AbstractNIOConnPool.fireCallbacks() AbstractNIOConnPool.java:506
org.apache.http.nio.pool.AbstractNIOConnPool.lease(Object, Object, long, long, TimeUnit, FutureCallback) AbstractNIOConnPool.java:290
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(HttpRoute, Object, long, long, TimeUnit, FutureCallback) PoolingNHttpClientConnectionManager.java:295
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection() AbstractClientExchangeHandler.java:377
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start() DefaultClientExchangeHandlerImpl.java:129
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(HttpAsyncRequestProducer, HttpAsyncResponseConsumer, HttpContext, FutureCallback) InternalHttpAsyncClient.java:141
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(HttpAsyncRequestProducer, HttpAsyncResponseConsumer, FutureCallback) CloseableHttpAsyncClient.java:68
io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(CloseableHttpAsyncClient, ApacheRequestProducer, CoroutineContext, Continuation) ApacheHttpRequest.kt:50
io.ktor.client.engine.apache.ApacheEngine.execute(HttpRequestData, Continuation) ApacheEngine.kt:23
io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(Object) HttpClientEngine.kt:49
io.ktor.client.engine.HttpClientEngine$install$1.invoke(Object, Object, Object) HttpClientEngine.kt
io.ktor.util.pipeline.SuspendFunctionGun.loop(boolean) PipelineContext.kt:268
io.ktor.util.pipeline.SuspendFunctionGun.proceed(Continuation) PipelineContext.kt:141
io.ktor.util.pipeline.SuspendFunctionGun.execute(Object, Continuation) PipelineContext.kt:161
io.ktor.util.pipeline.Pipeline.execute(Object, Object, Continuation) Pipeline.kt:27
io.ktor.client.features.HttpSend$DefaultSender.execute(HttpRequestBuilder, Continuation) HttpSend.kt:90
io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(Object) HttpSend.kt:62
io.ktor.client.features.HttpSend$Feature$install$1.invoke(Object, Object, Object) HttpSend.kt
io.ktor.util.pipeline.SuspendFunctionGun.loop(boolean) PipelineContext.kt:268
io.ktor.util.pipeline.SuspendFunctionGun.proceed(Continuation) PipelineContext.kt:141
io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(Object, Continuation) PipelineContext.kt:151
io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(Object) HttpCallValidator.kt:87
io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(Object, Object, Object) HttpCallValidator.kt
io.ktor.util.pipeline.SuspendFunctionGun.loop(boolean) PipelineContext.kt:268
io.ktor.util.pipeline.SuspendFunctionGun.proceed(Continuation) PipelineContext.kt:141
io.ktor.util.pipeline.SuspendFunctionGun.execute(Object, Continuation) PipelineContext.kt:161
io.ktor.util.pipeline.Pipeline.execute(Object, Object, Continuation) Pipeline.kt:27
io.ktor.client.HttpClient.execute(HttpRequestBuilder, Continuation) HttpClient.kt:141
io.ktor.client.call.HttpClientCallKt.call(HttpClient, Function2, Continuation) HttpClientCall.kt:140
io.ktor.client.call.UtilsKt.call(HttpClient, HttpRequestBuilder, Continuation) utils.kt:30
io.heapy.kotbot.bot.rule.CombotCasRule$validate$1.invokeSuspend(Object) Rules.kt:171
io.heapy.kotbot.bot.rule.CombotCasRule$validate$1.invoke(Object, Object) Rules.kt
kotlinx.coroutines.flow.SafeFlow.collect(FlowCollector, Continuation) Builders.kt:53
kotlinx.coroutines.flow.FlowKt__CollectionKt.toCollection(Flow, Collection, Continuation) Collection.kt:32
kotlinx.coroutines.flow.FlowKt.toCollection(Flow, Collection, Continuation)
kotlinx.coroutines.flow.FlowKt__CollectionKt.toList(Flow, List, Continuation) Collection.kt:15
kotlinx.coroutines.flow.FlowKt.toList(Flow, List, Continuation)
kotlinx.coroutines.flow.FlowKt__CollectionKt.toList$default(Flow, List, Continuation, int, Object) Collection.kt:15
kotlinx.coroutines.flow.FlowKt.toList$default(Flow, List, Continuation, int, Object)
io.heapy.kotbot.bot.KotBot$onUpdateReceived$1.invokeSuspend(Object) KotBot.kt:38
kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(Object) ContinuationImpl.kt:33
kotlinx.coroutines.DispatchedTask.run() Dispatched.kt:241
kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(Task) CoroutineScheduler.kt:594
kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler, Task) CoroutineScheduler.kt:60
kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run() CoroutineScheduler.kt:740
On "hot" thread I got following stacktraces:
Ktor-client-apache Runnable CPU usage on sample: 987ms
--
sun.security.ssl.SSLEngineImpl.checkParams(ByteBuffer[], int, int, ByteBuffer[], int, int) SSLEngineImpl.java:395
sun.security.ssl.SSLEngineImpl.wrap(ByteBuffer[], int, int, ByteBuffer[], int, int) SSLEngineImpl.java:133
sun.security.ssl.SSLEngineImpl.wrap(ByteBuffer[], int, int, ByteBuffer) SSLEngineImpl.java:116
javax.net.ssl.SSLEngine.wrap(ByteBuffer, ByteBuffer) SSLEngine.java:479
org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(ByteBuffer, ByteBuffer) SSLIOSession.java:263
org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake() SSLIOSession.java:301
org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady() SSLIOSession.java:503
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(IOSession) AbstractIODispatch.java:120
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(SelectionKey) BaseIOReactor.java:162
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(SelectionKey) AbstractIOReactor.java:337
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(Set) AbstractIOReactor.java:315
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute() AbstractIOReactor.java:276
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(IOEventDispatch) BaseIOReactor.java:104
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run() AbstractMultiworkerIOReactor.java:588
java.lang.Thread.run() Thread.java:834
Ktor-client-apache Runnable CPU usage on sample: 977ms
--
sun.security.ssl.SSLEngineImpl.getHandshakeStatus() SSLEngineImpl.java:801
sun.security.ssl.SSLEngineImpl.writeRecord(ByteBuffer[], int, int, ByteBuffer[], int, int) SSLEngineImpl.java:185
sun.security.ssl.SSLEngineImpl.wrap(ByteBuffer[], int, int, ByteBuffer[], int, int) SSLEngineImpl.java:136
sun.security.ssl.SSLEngineImpl.wrap(ByteBuffer[], int, int, ByteBuffer) SSLEngineImpl.java:116
javax.net.ssl.SSLEngine.wrap(ByteBuffer, ByteBuffer) SSLEngine.java:479
org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(ByteBuffer, ByteBuffer) SSLIOSession.java:263
org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake() SSLIOSession.java:301
org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady() SSLIOSession.java:503
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(IOSession) AbstractIODispatch.java:120
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(SelectionKey) BaseIOReactor.java:162
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(SelectionKey) AbstractIOReactor.java:337
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(Set) AbstractIOReactor.java:315
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute() AbstractIOReactor.java:276
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(IOEventDispatch) BaseIOReactor.java:104
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run() AbstractMultiworkerIOReactor.java:588
java.lang.Thread.run() Thread.java:834
Yourkit deadlock detection:
Potential deadlock: frozen threads found
It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Name |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| +---DefaultDispatcher-worker-8 Frozen for at least 7m 31s |
| | | |
| | +---org.apache.http.nio.reactor.ssl.SSLIOSession.setEvent(int) SSLIOSession.java:672 |
| | | |
| | +---org.apache.http.impl.nio.NHttpConnectionBase.requestOutput() NHttpConnectionBase.java:358 |
| | | |
| | +---org.apache.http.impl.nio.conn.CPoolProxy.requestOutput() CPoolProxy.java:120 |
| | | |
| | +---org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionAllocated(NHttpClientConnection) AbstractClientExchangeHandler.java:334 |
| | | |
| | +---org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$000(AbstractClientExchangeHandler, NHttpClientConnection) AbstractClientExchangeHandler.java:62 |
| | | |
| | +---org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.completed(NHttpClientConnection) AbstractClientExchangeHandler.java:387 |
| | | |
| | +---org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.completed(Object) AbstractClientExchangeHandler.java:383 |
| | | |
| | +---org.apache.http.concurrent.BasicFuture.completed(Object) BasicFuture.java:123 |
| | | |
| | +---org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.completed(CPoolEntry) PoolingNHttpClientConnectionManager.java:306 |
| | | |
| | +---org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.completed(Object) PoolingNHttpClientConnectionManager.java:297 |
| | | |
| | +---org.apache.http.concurrent.BasicFuture.completed(Object) BasicFuture.java:123 |
| | | |
| | +---org.apache.http.nio.pool.AbstractNIOConnPool.fireCallbacks() AbstractNIOConnPool.java:506 |
| | | |
| | +---org.apache.http.nio.pool.AbstractNIOConnPool.lease(Object, Object, long, long, TimeUnit, FutureCallback) AbstractNIOConnPool.java:290 |
| | | |
| | +---org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(HttpRoute, Object, long, long, TimeUnit, FutureCallback) PoolingNHttpClientConnectionManager.java:295 |
| | | |
| | +---org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection() AbstractClientExchangeHandler.java:377 |
| | | |
| | +---org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start() DefaultClientExchangeHandlerImpl.java:129 |
| | | |
| | +---org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(HttpAsyncRequestProducer, HttpAsyncResponseConsumer, HttpContext, FutureCallback) InternalHttpAsyncClient.java:141 |
| | | |
| | +---org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(HttpAsyncRequestProducer, HttpAsyncResponseConsumer, FutureCallback) CloseableHttpAsyncClient.java:68 |
| | | |
| | +---io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(CloseableHttpAsyncClient, ApacheRequestProducer, CoroutineContext, Continuation) ApacheHttpRequest.kt:50 |
| | | |
| | +---io.ktor.client.engine.apache.ApacheEngine.execute(HttpRequestData, Continuation) ApacheEngine.kt:23 |
| | | |
| | +---io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(Object) HttpClientEngine.kt:49 |
| | | |
| | +---io.ktor.client.engine.HttpClientEngine$install$1.invoke(Object, Object, Object) HttpClientEngine.kt |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.loop(boolean) PipelineContext.kt:268 |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.proceed(Continuation) PipelineContext.kt:141 |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.execute(Object, Continuation) PipelineContext.kt:161 |
| | | |
| | +---io.ktor.util.pipeline.Pipeline.execute(Object, Object, Continuation) Pipeline.kt:27 |
| | | |
| | +---io.ktor.client.features.HttpSend$DefaultSender.execute(HttpRequestBuilder, Continuation) HttpSend.kt:90 |
| | | |
| | +---io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(Object) HttpSend.kt:62 |
| | | |
| | +---io.ktor.client.features.HttpSend$Feature$install$1.invoke(Object, Object, Object) HttpSend.kt |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.loop(boolean) PipelineContext.kt:268 |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.proceed(Continuation) PipelineContext.kt:141 |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(Object, Continuation) PipelineContext.kt:151 |
| | | |
| | +---io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(Object) HttpCallValidator.kt:87 |
| | | |
| | +---io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(Object, Object, Object) HttpCallValidator.kt |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.loop(boolean) PipelineContext.kt:268 |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.proceed(Continuation) PipelineContext.kt:141 |
| | | |
| | +---io.ktor.util.pipeline.SuspendFunctionGun.execute(Object, Continuation) PipelineContext.kt:161 |
| | | |
| | +---io.ktor.util.pipeline.Pipeline.execute(Object, Object, Continuation) Pipeline.kt:27 |
| | | |
| | +---io.ktor.client.HttpClient.execute(HttpRequestBuilder, Continuation) HttpClient.kt:141 |
| | | |
| | +---io.ktor.client.call.HttpClientCallKt.call(HttpClient, Function2, Continuation) HttpClientCall.kt:140 |
| | | |
| | +---io.ktor.client.call.UtilsKt.call(HttpClient, HttpRequestBuilder, Continuation) utils.kt:30 |
| | | |
| | +---io.heapy.kotbot.bot.rule.CombotCasRule$validate$1.invokeSuspend(Object) Rules.kt:171 |
| | | |
| | +---io.heapy.kotbot.bot.rule.CombotCasRule$validate$1.invoke(Object, Object) Rules.kt |
| | | |
| | +---kotlinx.coroutines.flow.SafeFlow.collect(FlowCollector, Continuation) Builders.kt:53 |
| | | |
| | +---kotlinx.coroutines.flow.FlowKt__CollectionKt.toCollection(Flow, Collection, Continuation) Collection.kt:32 |
| | | |
| | +---kotlinx.coroutines.flow.FlowKt.toCollection(Flow, Collection, Continuation) |
| | | |
| | +---kotlinx.coroutines.flow.FlowKt__CollectionKt.toList(Flow, List, Continuation) Collection.kt:15 |
| | | |
| | +---kotlinx.coroutines.flow.FlowKt.toList(Flow, List, Continuation) |
| | | |
| | +---kotlinx.coroutines.flow.FlowKt__CollectionKt.toList$default(Flow, List, Continuation, int, Object) Collection.kt:15 |
| | | |
| | +---kotlinx.coroutines.flow.FlowKt.toList$default(Flow, List, Continuation, int, Object) |
| | | |
| | +---io.heapy.kotbot.bot.KotBot$onUpdateReceived$1.invokeSuspend(Object) KotBot.kt:38 |
| | | |
| | +---kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(Object) ContinuationImpl.kt:33 |
| | | |
| | +---kotlinx.coroutines.DispatchedTask.run() Dispatched.kt:241 |
| | | |
| | +---kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(Task) CoroutineScheduler.kt:594 |
| | | |
| | +---kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler, Task) CoroutineScheduler.kt:60 |
| | | |
| | +---kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run() CoroutineScheduler.kt:740 |
| | |
| +---Ktor-client-apache Frozen for at least 10m 53s |
| | | |
| | +---sun.nio.ch.EPoll.wait(int, long, int, int) EPoll.java (native) |
| | | |
| | +---sun.nio.ch.EPollSelectorImpl.doSelect(Consumer, long) EPollSelectorImpl.java:120 |
| | | |
| | +---sun.nio.ch.SelectorImpl.lockAndDoSelect(Consumer, long) SelectorImpl.java:124 |
| | | |
| | +---sun.nio.ch.SelectorImpl.select(long) SelectorImpl.java:136 |
| | | |
| | +---org.apache.http.impl.nio.reactor.AbstractIOReactor.execute() AbstractIOReactor.java:255 |
| | | |
| | +---org.apache.http.impl.nio.reactor.BaseIOReactor.execute(IOEventDispatch) BaseIOReactor.java:104 |
| | | |
| | +---org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run() AbstractMultiworkerIOReactor.java:588 |
| | | |
| | +---java.lang.Thread.run() Thread.java:834 |
| | |
| +---Ktor-client-apache Frozen for at least 10m 53s |
| | | |
| | +---sun.nio.ch.EPoll.wait(int, long, int, int) EPoll.java (native) |
| | | |
| | +---sun.nio.ch.EPollSelectorImpl.doSelect(Consumer, long) EPollSelectorImpl.java:120 |
| | | |
| | +---sun.nio.ch.SelectorImpl.lockAndDoSelect(Consumer, long) SelectorImpl.java:124 |
| | | |
| | +---sun.nio.ch.SelectorImpl.select(long) SelectorImpl.java:136 |
| | | |
| | +---org.apache.http.impl.nio.reactor.AbstractIOReactor.execute() AbstractIOReactor.java:255 |
| | | |
| | +---org.apache.http.impl.nio.reactor.BaseIOReactor.execute(IOEventDispatch) BaseIOReactor.java:104 |
| | | |
| | +---org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run() AbstractMultiworkerIOReactor.java:588 |
| | | |
| | +---java.lang.Thread.run() Thread.java:834 |
| | |
| +---Ktor-client-apache Frozen for at least 10m 53s |
| | | |
| | +---sun.nio.ch.EPoll.wait(int, long, int, int) EPoll.java (native) |
| | | |
| | +---sun.nio.ch.EPollSelectorImpl.doSelect(Consumer, long) EPollSelectorImpl.java:120 |
| | | |
| | +---sun.nio.ch.SelectorImpl.lockAndDoSelect(Consumer, long) SelectorImpl.java:124 |
| | | |
| | +---sun.nio.ch.SelectorImpl.select(long) SelectorImpl.java:136 |
| | | |
| | +---org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(IOEventDispatch) AbstractMultiworkerIOReactor.java:340 |
| | | |
| | +---org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(IOEventDispatch) PoolingNHttpClientConnectionManager.java:221 |
| | | |
| | +---org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run() CloseableHttpAsyncClientBase.java:64 |
| | | |
| | +---java.lang.Thread.run() Thread.java:834 |
| | |
| +---nioEventLoopGroup-2-1 Frozen for at least 10m 52s |
| | |
| +---sun.nio.ch.EPoll.wait(int, long, int, int) EPoll.java (native) |
| | |
| +---sun.nio.ch.EPollSelectorImpl.doSelect(Consumer, long) EPollSelectorImpl.java:120 |
| | |
| +---sun.nio.ch.SelectorImpl.lockAndDoSelect(Consumer, long) SelectorImpl.java:124 |
| | |
| +---sun.nio.ch.SelectorImpl.select(long) SelectorImpl.java:136 |
| | |
| +---io.netty.channel.nio.SelectedSelectionKeySetSelector.select(long) SelectedSelectionKeySetSelector.java:62 |
| | |
| +---io.netty.channel.nio.NioEventLoop.select(boolean) NioEventLoop.java:806 |
| | |
| +---io.netty.channel.nio.NioEventLoop.run() NioEventLoop.java:454 |
| | |
| +---io.netty.util.concurrent.SingleThreadEventExecutor$5.run() SingleThreadEventExecutor.java:918 |
| | |
| +---io.netty.util.internal.ThreadExecutorMap$2.run() ThreadExecutorMap.java:74 |
| | |
| +---io.netty.util.concurrent.FastThreadLocalRunnable.run() FastThreadLocalRunnable.java:30 |
| | |
| +---java.lang.Thread.run() Thread.java:834 |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I have following apache http libs on classpath:
httpasyncclient-4.1.4
httpcore-nio-4.4.10
httpclient-4.5.6
httpcore-4.4.10
httpmime-4.5.3
IRus
on 13 Dec 2019
@ok2c looks like I found deadlock in apache client, very similar to this issue https://issues.apache.org/jira/browse/HTTPASYNC-153
DefaultDispatcher-worker-8 Blocked CPU usage on sample: 0ms
org.apache.http.nio.reactor.ssl.SSLIOSession.setEvent(int) SSLIOSession.java:672
and
sun.security.ssl.SSLEngineImpl.getHandshakeStatus() SSLEngineImpl.java:801
sun.security.ssl.SSLEngineImpl.writeRecord(ByteBuffer[], int, int, ByteBuffer[], int, int) SSLEngineImpl.java:185
sun.security.ssl.SSLEngineImpl.wrap(ByteBuffer[], int, int, ByteBuffer[], int, int) SSLEngineImpl.java:136
sun.security.ssl.SSLEngineImpl.wrap(ByteBuffer[], int, int, ByteBuffer) SSLEngineImpl.java:116
javax.net.ssl.SSLEngine.wrap(ByteBuffer, ByteBuffer) SSLEngine.java:479
org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(ByteBuffer, ByteBuffer) SSLIOSession.java:263
Issue possibly related to TLS1.3, disabling TLS1.3 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 and trying to reproduce issue
IRus
on 14 Dec 2019
Confirm that TLSv1.3 causing this issue
IRus
on 14 Dec 2019
Update to org.apache.httpcomponents:httpcore-nio:4.4.12 - fixes issue, @e5l consider update
IRus
on 14 Dec 2019
@e5l @IRus TLSv13 caused us a lot grief but as far as I can say all known issues have been resolved with HttpCore 4.4.12.
ok2c
on 14 Dec 2019
❤2
@IRus, @ok2c Great thanks for the investigation.
It looks like we can't bump all versions in ktor, and need to wait for apache async HTTP-client(https://issues.apache.org/jira/projects/HTTPASYNC/issues/?filter=allopenissues) minor update.
e5l
on 16 Dec 2019
@ok2c, could you tell me when you can update it?
e5l
on 16 Dec 2019
@e5l There are no severe bugs in HttpAsyncClient that would warrant an immediate release. I was thinking about 4.1.5 in Jan 2020.
ok2c
on 16 Dec 2019
I've also found that Apache client consume to much CPU details are described in this issue https://github.com/Kotlin/kotlinx.coroutines/issues/1833#issuecomment-592473146
rumatoest
on 1 Mar 2020
Since there's good alternatives to the Apache client that run on the same platform (JVM) and don't have that CPU burning issue, is there any reason to still support the Apache client in ktor?
LouisCAD
on 1 Mar 2020
Apache Client one of the most feature complete, and production tested clients.
This issue related to the newest JDK releases and particularly TSL v1.3 support.
Still, this https://github.com/ktorio/ktor/issues/1018#issuecomment-565648818 fixes issue
IRus
on 1 Mar 2020
👍1
Merged in master. Closed
e5l
on 17 Jul 2020
Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.
oleg-larshin
on 10 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
gabin8
路
4Comments
PPACI
路
4Comments
scorsi
路
4Comments
baruchn
路
3Comments
chrisjenx
路
4Comments
Most helpful comment
@e5l @IRus
TLSv13caused us a lot grief but as far as I can say all known issues have been resolved with HttpCore 4.4.12.