Hystrix: Consider using Reactive Streams types instead of RxJava types

[mattrjacobs]

At the moment, reactive execution of HystrixCommands returns rx.Observables. If it returned org.reactivestreams.Publishers, then it could be consumed by non-RxJava systems by performing the appropriate ReactiveStreams impl.

Is there any interest in this model?

[spencergibb]

Yes, pivotal is interested.

[smaldini]

Hey we are actually experimenting with https://github.com/javaslang/javaslang-circuitbreaker to introduce reactive operators such as "stream.circuitBreaker(options)" where stream is a Publisher. It's really first step for us and would be interested in federating any effort in that direction to find out exactly what can be done.

[mattrjacobs]

@smaldini Cool, thanks for the pointer!

I think the 2 paths that could occur in Hystrix are roughly:

1) Hystrix could expose a toPublisher() method alongside the existing API. RxJava would remain a dependency and we would just do the conversion to the ReactiveStream Publisher when asked-for.

2) Hystrix could remove the RxJava methods toObservable() / observe() and only use toPublisher(). This would obviously be a large breaking change, and Hystrix would still need to implement the Publisher creation, and I would have to pick an implementation of Reactive Streams at that point. Since I would very likely choose RxJava, Hystrix would still maintain a dependency on RxJava, just the public API would change.

I don't think 2) has any upside and has a lot of downside. Do you have any thoughts on the above approaches, or other alternatives?

[smaldini]

@mattrjacobs anytime ! so let's evaluate the options :

1) Hystrix could expose a toPublisher() method alongside the existing API. RxJava would remain a dependency and we would just do the conversion to the ReactiveStream Publisher when asked-for.

That means you implement your own Reactive Streams infrastructure as you can't depend on rxjava 1 and 2 at the same time. I think overall it might be misleading and will discourage users to connect with Publisher because it has no vocabulary beyond subscribe() quickly limiting out of the box usages.

2) Hystrix could remove the RxJava methods toObservable() / observe() and only use toPublisher(). This would obviously be a large breaking change, and Hystrix would still need to implement the Publisher creation, and I would have to pick an implementation of Reactive Streams at that point. Since I would very likely choose RxJava, Hystrix would still maintain a dependency on RxJava, just the public API would change.

Removing dependency on a composition API will face the same limitation than in point 1 I think. Even you as a committer will quickly be frustrated in your integration testings. If you want to stick to RxJava just wait for RxJava 2.0 in that case and expose it.

3) Spring Framework has matured the same question you have over the last months. The realization after going from the steps :

• "We use internal infrastructure for reactive stuff and just expose Publisher`
• "We let a dedicated scope for that project, Reactor with PublisherFactory and a common basic ground for operators like RxJS Lite"
• "Publisher alone is not usable, why not having two simple types for 0..N sequences and 0..1 sequences implementing Publisher"

We created that with Flux and Mono. And that's both our toPublisher() now (or toFlux()) and our infrastructure. In parallel we worked together with @akarnokd on Reactive Streams Commons. That's an infrastructure fully inlined by Flux and Mono where we assembled the best practices from Rx and old-reactor together. This next generation of Rx operators has the extra benefit of implementing a Fusion protocol to actually mitigate overhead in some (a)synchronous stages, getting that closer to bare-metal performances. You can benefit from that protocol as well. Since Flux is RxLite, we have an extra project about to be released called reactor-stream where Fluxion is doing all of what you can expect from an Rx implementation over Reactive Streams.

After that long introduction, you might see where I am going to with that :) I think you will face the same questions than Spring Framework at some point while experimenting with your transition to Reactive Streams. If you want to fast track and start experimenting now, you can use reactor-core 2.5.0.M1 and the pending reactor-stream 2.5.0.M1 (ETA end of the week). Then you can pick and choose or maybe then RxJava 2.0 will be around and you will prefer to come back to it, we talk the same language practically now and we're not possessive ! I introduced this whole idea last week actually here.

[agentgt]

@mattrjacobs @smaldini Hey guys let me know when you start work as I have looked into this myself and ended up just writing some custom stuff for my company.

A massive issue is that Hystrix was designed and has the assumption of a Request/Reply/Single/Promise library than a streaming library. It doesn't do backpressure correctly see: #1089 . I believe this is because of the various rxjava operators that are being applied to add timers/circuit breakers (ie some operators disregard backpressure... the most famous one being flatmap/merge).

Even if the operators were fixed (ie respect backpressure) there are all these ambiguous semantic questions like what is a true timeout. For example Hystrix only times out on the first item being received while streaming. That is if you have a long running stream and the first item comes quick but the others take forever Hystrix does nothing (I don't think there is a correct answer how to handle this but it should at least be configurable).

I looked at the work it almost seems like a greater undertaking to fix and refactor Hystrix to respect streams. The other issue is that streams can be used for more than Request/Reply pattern (e.g. pub/sub). I think it might be best if it were a separate library or decoupled module that interacts with the reactive-streams directly or RxJava 2 as there are just too many difference with handling streams to commands. IMO If any change should be made directly to Hystrix its that it should return rx.Single instead of rx.Observable (ie deprecate the methods toObservable and observe).

As for existing Hystrix I think there needs to be another Reactive Streams like process (ie standard interface) for cold/hot promise like single objects (rx.Single) (ie onSuccess(T), onFailure(Throwable)). See https://github.com/reactive-streams/reactive-streams-jvm/issues/268 .There are just so many libraries implementing promise like objects that the interop is fairly irritating particularly given that some assume hot and others assume cold. Hot being the promise is like a Future with callbacks (aka Guava ListenableFuture). Cold being like rx.Single where a subscribe is called to kick it off.

[mattrjacobs]

@smaldini Thanks for all of your thoughts - that's very helpful context.

@agentgt All great points, and I agree completely with your point about Hystrix starting from the req-resp model and being more well-suited to it.

I'm doing some prototyping work, as we're internally trying to get async network I/O sorted out. I'll share more thoughts once I've gotten further with the prototypes.

I think the cases that are most important to me are:

1) sync req/resp - I think Hystrix already solves this very effectively
2) async req/resp - I think Hystrix works, but is heavyweight. There's no need for any of the threading concerns.
3) async req/"stream of n" - As @agentgt mentions, some concerns like timeout are semantically "off" in this world.
4) async req/infinite stream - Same as 3.

Again, I really appreciate the thoughts and involvement. I want to find great solutions to all of the above. I'll be working on a design proposal and will ask for feedback sometime soon.

[apischan]

I see this issue is old one and no updates for a long time. Is there any plans to integrate Hystrix with project-reactor (Flux, Mono)?
I see that this stream is not about project reactor but I didn't want to create new one.

[mattrjacobs]

There's no work happening at the moment.

[michael-barker]

Any update on whether Reactor support will be added? This would be great to have for the Spring 5 WebClient.

[spencergibb]

I doubt anything is going to happen, though I have successfully used the reactive streams compatibility library to use Hystrix with reactor and webflux

[akarnokd]

The conversion should be straightforward, however, I'm not certain the library is in a shape that would avoid a lot of duplicate development then on.

[michael-barker]

@spencergibb Do you have a link to an example? Does this approach require manual configuration or could it be configured with Spring properties?

[agentgt]

I'm not really sure I understand the use cases here of why people want this so badly (given someone down voted @mattrjacobs comment) . I am assuming its one of the following:

1. Have an existing backed code base in Hystrix (say database access) and want to connect to newer non blocking Reactor that is upstream
2. Would like to use Hystrix on top of Reactor for fault tolerance and protection
3. Don't want to pass around rx Observable all over the place and would prefer a standard (reactive streams).

Use case 1 in my opinion should be handled with microservice separation. Reactor code should be separated from Hystrix code. Even if it isn't ideal I can still understand why someone might want to do 1 temporarily but as @akarnokd pointed its pretty straightforward to write a bridge. As I mentioned earlier if you limit Hystrix to just using Mono types aka rx.Single and consistently use either cold or hot the mapping should be straight forward.

As for use case 2 and 3 I think really there should be a library on top of reactor or as mentioned earlier the fault tolerance circuit breaking logic should be decoupled from all reactive libraries. For the latter I had written my own internal library for my company but eventually we will probably switch to this far better open source solution: https://github.com/resilience4j/resilience4j .

[michael-barker]

@agentgt I've actually been wanting to try out Resilience4j instead of Hystrix but the last I checked a month or so ago there was no Reactor support. Looks like that will no longer be the case :) https://github.com/resilience4j/resilience4j/issues/35

[remesh22]

@spencergibb Can you please provide the link to the example with Hystrix/WebFlux WebClient. I am looking into wrapping WebClient requests using HystrixObservableCommand.

[spencergibb]

https://github.com/spring-cloud/spring-cloud-netflix/blob/master/spring-cloud-netflix-core/src/main/java/org/springframework/cloud/netflix/hystrix/HystrixCommands.java

[kptfh]

Hi All. Here is my version of reactive feign client https://github.com/kptfh/feign-reactive.
Plan to integrate it with HystrixObservableCommand and LoadBalancerCommand (Ribbon) so it can be used in cloud environment.