Spring-cloud-sleuth: ClassCastException with webClient

[oburgosm]

With sleuth 2.1.3, when webClient is instrumented by sleuth, there is a ClassCastException.

I have a project that reproduces the error: https://github.com/oburgosm/test-sleuth-webclient

Error in this project is:

java.lang.ClassCastException: class org.springframework.core.io.buffer.NettyDataBuffer cannot be cast to class xxx.Employee (org.springframework.core.io.buffer.NettyDataBuffer and xxx.Employee are in unnamed module of loader 'app')
...

If I set spring.sleuth.web.client.enabled the error doesn't occur

The problem doesn't occur with sleuth 2.1.2,

[marcingrzejszczak]

Hi can you provide the full stacktrace please?

[oburgosm]

Yes, sorry.

java.lang.ClassCastException: class org.springframework.core.io.buffer.NettyDataBuffer cannot be cast to class com.bracso.test.sleuth.webclient.Employee (org.springframework.core.io.buffer.NettyDataBuffer and com.bracso.test.sleuth.webclient.Employee are in unnamed module of loader 'app')
    at com.bracso.test.sleuth.webclient.SleuthTest.lambda$testWebClient$0(SleuthTest.java:45) ~[test-classes/:na]
    at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onNext(FluxDoOnEach.java:140) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:242) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
    at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
    at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
    at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
    at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:206) ~[reactor-netty-0.8.11.RELEASE.jar:0.8.11.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:329) ~[reactor-netty-0.8.11.RELEASE.jar:0.8.11.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:350) ~[reactor-netty-0.8.11.RELEASE.jar:0.8.11.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:571) ~[reactor-netty-0.8.11.RELEASE.jar:0.8.11.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91) ~[reactor-netty-0.8.11.RELEASE.jar:0.8.11.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) ~[netty-codec-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:315) ~[netty-codec-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:429) ~[netty-codec-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[netty-codec-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[netty-transport-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[netty-common-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.39.Final.jar:4.1.39.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

[marcingrzejszczak]

Hey @bsideup @smaldini. I see that it might be related to https://github.com/spring-cloud/spring-cloud-sleuth/blob/master/spring-cloud-sleuth-core/src/main/java/org/springframework/cloud/sleuth/instrument/web/client/TraceWebClientBeanPostProcessor.java#L374-L385 . @smaldini a couple of months ago you did "a grand refactoring" of Sleuth and added the DataBuffer there.

What was the reason for that? Also it seems that with the latest release some things must have changed in the Reactor version (or I messed up sth in Sleuth) cause there's an apparent breakage in compatibility.

[blake-bauman]

This is important for our org. We've been eager for #1393 but can't take 2.1.3.RELEASE because of this regression.