Hystrix 1.5.5 and Jackson Error on Metrics-Event-Stream

Created on 24 Aug 2016 · 19Comments · Source: Netflix/Hystrix

Last version of hystrix-metrics-event-stream (1.5.5) running on a JBoss AS 7 app server, when accessing the metric stream, give me this error:

Error log

14:30:04,638 java.lang.NullPointerException GtOHbn9FSiIkdARjIGvK6L7R app-d821bd09.prod.foo.local WARN  [org.jboss.modules] (RxComputationScheduler-1) Failed to define class com.fasterxml.jackson.dataformat.cbor.CBORGenerator in Module "deployment.foo-app.ear.foo-app-presentation.war:main" from Service Module Loader: java.lang.VerifyError: class com.fasterxml.jackson.dataformat.cbor.CBORGenerator overrides final method copyCurrentEvent.(Lcom/fasterxml/jackson/core/JsonParser;)V
        at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_72]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800) [rt.jar:1.7.0_72]
        at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:361)
        at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:482)
        at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:277)
        at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:92)
        at org.jboss.modules.Module.loadModuleClass(Module.java:568)
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:205)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)
        at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:66) [hystrix-metrics-event-stream-1.5.5.jar:1.5.5]
        at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:63) [hystrix-metrics-event-stream-1.5.5.jar:1.5.5]
        at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:259) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OperatorOnBackpressureDrop$2.onNext(OperatorOnBackpressureDrop.java:83) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeRefCount$2.onNext(OnSubscribeRefCount.java:120) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:588) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OperatorPublish$PublishSubscriber.onNext(OperatorPublish.java:286) [rxjava-1.1.8.jar:1.1.8]
        at rx.observers.Subscribers$5.onNext(Subscribers.java:235) [rxjava-1.1.8.jar:1.1.8]
        at rx.observers.Subscribers$5.onNext(Subscribers.java:235) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52) [rxjava-1.1.8.jar:1.1.8]
        at rx.Scheduler$Worker$1.call(Scheduler.java:136) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.8.jar:1.1.8]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_72]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_72]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_72]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [rt.jar:1.7.0_72]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_72]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_72]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_72]

And the error I got is MaxConcurrentConnections reached: 5.

Any tips on how to fix that?

Thanks

metrics question

Source

caarlos0

👍1

Most helpful comment

It usually takes a couple of hours from the time I press the button, but it's out of my hands. Can you re-try?

mattrjacobs on 2 Jan 2017

😄1 👍1

All 19 comments

There is a configuration property for the max concurrent connections, the default is 5.

https://github.com/Netflix/Hystrix/blob/v1.5.5/hystrix-contrib/hystrix-metrics-event-stream/src/main/java/com/netflix/hystrix/contrib/metrics/eventstream/HystrixMetricsStreamServlet.java#L55-L56

mattnelson on 24 Aug 2016

@mattflix yeah, as I said, the problem isn't that, the exception is masking it...

The problem seems to be that the jackson version inside jboss as is older that the version used by metrics-stream... but the error shown is the number of connections, which is unrelated to the root cause...

caarlos0 on 24 Aug 2016

FWIW: version 1.5.3 works

caarlos0 on 24 Aug 2016

Hystrix 1.5.4 broke out the serialization of metrics into its own module (hystrix-serialization). That module depends on 2.7.5 Jackson libraries.

In your application, do you have the hystrix-serialization jar? What versions of Jackson do you have there?

mattrjacobs on 24 Aug 2016

@mattrjacobs yes, it was working until I updated it.

It's a dependency hell... just so you have a slight idea:

I'll try to bump all, for now, the report that remains is the misleading error message 👍

caarlos0 on 24 Aug 2016

I know that feeling :) I suspect that the mismatched Jackson is the problem. Please let me know if that resolves things.

mattrjacobs on 24 Aug 2016

@mattrjacobs yeah, I'll try to do that...

caarlos0 on 24 Aug 2016

I got the same problem in 1.5.5 and rectified it by excluding Jackson.

mukteshkrmishra on 26 Aug 2016

👍1

I was able to get this working when using an older jackson dep by adding hystrix like this:

<dependency>
            <groupId>com.netflix.hystrix</groupId>
            <artifactId>hystrix-metrics-event-stream</artifactId>
            <version>1.5.5</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.dataformat</groupId>
                    <artifactId>jackson-dataformat-cbor</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

pablote on 29 Aug 2016

👍1

This looks resolved. Please re-open if there's more to discuss

mattrjacobs on 17 Sep 2016

👍1

Hi, I'm using 1.5.8 of hystrix-metrics-event-stream. First I had the same verifyError like the original poster, I added the exclusion like @pablote suggested then, but now I get the following ClassNotFound:

Exception in thread "RxComputationScheduler-1" java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread. at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/dataformat/cbor/CBORParser at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:66) at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:63) at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:259) at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) at rx.internal.operators.OperatorOnBackpressureDrop$2.onNext(OperatorOnBackpressureDrop.java:98) at rx.internal.operators.OnSubscribeRefCount$2.onNext(OnSubscribeRefCount.java:120) at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:587) at rx.internal.operators.OperatorPublish$PublishSubscriber.onNext(OperatorPublish.java:285) at rx.observers.Subscribers$5.onNext(Subscribers.java:235) at rx.observers.Subscribers$5.onNext(Subscribers.java:235) at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52) at rx.Scheduler$Worker$1.call(Scheduler.java:137) at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189) at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ... 7 more Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.dataformat.cbor.CBORParser at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1333) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167) ... 22 more

Now I'm not sure how to work around this, since there is obviously a Jackson on classpath already, but this is obviously not compatible to the one hystrix is using. Any hints?

mechko on 30 Nov 2016

Do you have the hystrix-serialization jar on your classpath?

mattrjacobs on 3 Dec 2016

Yes. Parts of the application I'm trying to use with Hystrix is depending on Jackson 2.2.1, but at this early time there was obviously no CBORParser, thats why excluding jackson-dataformat-cbor means class CBORParser is not there at all.

I guess I can only solve this dilemma by using a very early version of Hystrix (say 1.4.26 or the like). What do you think, would that make sense?

Edit: Fixed suggested Hystrix Version.

mechko on 3 Dec 2016

So I think the problematic path for your use-case is that SerialHystrixDashboardData has static methods for String/JSON/CBOR. I will experiment if splitting those methods into separate classes allows some of the Jackson JARs to be excluded.

Otherwise, pinning Hystrix back would work as well

mattrjacobs on 6 Dec 2016

👍1

I think a better design of this set of functionality is to be modular for serialization. If you need JSON serialization, then there should be a set of methods with a dependency on jackson-json. If you need CBOR serialization, then there should be a set of methods with a dependency on jackson-cbor.

In this PR, I've deprecated (and hollowed out) the byte[] implementations of serializations/deserializations. They can come back in a different module when I reintroduce the ReactiveSocket impls of streams (#1325).

Thoughts?

mattrjacobs on 20 Dec 2016

Thank you so much Matt, looks good so far. Is there a timeline already when 1.5.9 will be available from Maven Central?

mechko on 22 Dec 2016

It usually takes a couple of hours from the time I press the button, but it's out of my hands. Can you re-try?

mattrjacobs on 2 Jan 2017

😄1 👍1

Hi Matt, I just tested the new release 1.5.9 from Maven Central and voila: It works like a charm now! No class loader errors at runtime. My effective dependency tree looks like this now (the omitted and newer Jackson version does obviously not affect the functionality of the JSON serialization that is needed by Hystrix).

Thank you so much for your quick reaction and that nice release!

mechko on 3 Jan 2017

Great- putting all the serialization options in the same jar was a mistake by me. I'll figure out a better way to do it once we have pieces that need the CBOR serialization.

mattrjacobs on 6 Jan 2017

Was this page helpful?

0 / 5 - 0 ratings