Hystrix circuit short-circuited and is OPEN

Created on 9 Apr 2015  Â·  30Comments  Â·  Source: Netflix/Hystrix

Hi,

I ran into this Exception, not sure what is the cause. The setup is fairly simple and I am doing functional testing only. All Hystrix config is defaulted with one exception: HystrixCommandProperties.Setter().withFallbackEnabled(false);

Any help and suggestions would be helpful.

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: CreateUserCommand short-circuited and failed retrieving fallback.
        at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:830)
        at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:798)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:76)
        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
        at rx.Observable$ThrowObservable$1.call(Observable.java:10493)
        at rx.Observable$ThrowObservable$1.call(Observable.java:10483)
        at rx.Observable.unsafeSubscribe(Observable.java:8591)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77)
        at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:318)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:310)

Thanks.
Weiping

picture WeipingGuo

Most helpful comment

The state machine describing how Hystrix responds to requests is here: https://github.com/Netflix/Hystrix/wiki/How-it-Works#flow-chart

In your case, there have been enough errors prior to the request above that the Hystrix circuit-breaker is OPEN. When the circuit-breaker is OPEN, the request is automatically short-circuited to shed load. If a fallback is defined and enabled that logic is triggered. Since you have disabled fallback, the exception is generated and thrown to the caller.

Does that make sense, or is there anything in my explanation or the docs that could use clarification?

picture mattrjacobs on 9 Apr 2015
👍11

All 30 comments

The state machine describing how Hystrix responds to requests is here: https://github.com/Netflix/Hystrix/wiki/How-it-Works#flow-chart

In your case, there have been enough errors prior to the request above that the Hystrix circuit-breaker is OPEN. When the circuit-breaker is OPEN, the request is automatically short-circuited to shed load. If a fallback is defined and enabled that logic is triggered. Since you have disabled fallback, the exception is generated and thrown to the caller.

Does that make sense, or is there anything in my explanation or the docs that could use clarification?

mattrjacobs picture mattrjacobs on 9 Apr 2015
👍11

Thank you very much. That helps a lot.

WeipingGuo picture WeipingGuo on 9 Apr 2015

Hi @WeipingGuo @mattrjacobs What are the steps to solve this error "Hystrix circuit short-circuited and is OPEN" ?

juniebonifacio picture juniebonifacio on 12 Jan 2018

Seems like JIRA died today on this:

HTTP Status 500 - com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
type Exception report

message com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.

description The server encountered an internal error that prevented it from fulfilling this request.

exception

com.atlassian.tenant.api.TenantContextException: com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
    com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:132)
    com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
    java.util.Optional.ifPresent(Optional.java:159)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
    com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
    com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
    com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
root cause

com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
    com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
    com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:123)
    com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
    java.util.Optional.ifPresent(Optional.java:159)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
    com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
    com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
    com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
root cause

com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
    com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:826)
    com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:805)
    rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
    rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1473)
    com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1398)
    rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    rx.observers.Subscribers$5.onError(Subscribers.java:230)
    rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)
    rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.subscribe(Observable.java:10247)
    rx.Observable.subscribe(Observable.java:10214)
    rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
    rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
    com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
    com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
    com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
    com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:53)
    com.atlassian.tenant.impl.AbstractTenantContextProvider.lambda$setTenantId$2(AbstractTenantContextProvider.java:124)
    com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
    com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:123)
    com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
    java.util.Optional.ifPresent(Optional.java:159)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
    com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
    com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
    com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
root cause

java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
    com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:980)
    com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:558)
    com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60)
    com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:420)
    com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:414)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    rx.Observable.unsafeSubscribe(Observable.java:10151)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    rx.Observable.subscribe(Observable.java:10247)
    rx.Observable.subscribe(Observable.java:10214)
    rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
    rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
    com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
    com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
    com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
    com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:53)
    com.atlassian.tenant.impl.AbstractTenantContextProvider.lambda$setTenantId$2(AbstractTenantContextProvider.java:124)
    com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
    com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:123)
    com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
    java.util.Optional.ifPresent(Optional.java:159)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
    com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
    com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
    com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
    com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
    com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
    com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.38 logs.

Apache Tomcat/8.0.38
pcj picture pcj on 20 Feb 2018
😄5 👍4

I'm having the same problem, no matter what parameters I use in @HystrixCommand, even though hystrix writes 

ERROR c.n.hystrix.contrib.javanica.command.GenericCommand     - failed to process fallback is the method: 'fallbackAppendData'. exception: 'class com.myException' occurred in fallback was ignored and wrapped to HystrixBadRequestException.

the exception is not wrapped and a HystrixRuntimeException is thrown with 

java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN

as cause.

VaverDariush picture VaverDariush on 5 Mar 2018

I am getting the same issue:

Caused by: java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN

shashijais789 picture shashijais789 on 31 Aug 2018

Seems like JIRA died today on this:

```
HTTP Status 500 - com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.

And Confluence Cloud died on it for my today...

jeroenhabets picture jeroenhabets on 26 Sep 2018

seems confluence died again today

saicharan566 picture saicharan566 on 26 Sep 2018

@pcj @jeroenhabets jira and confluence down again :D

jpita picture jpita on 26 Sep 2018
😄4

Seems like JIRA died today on this:

HTTP Status 500 - com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.

And Confluence Cloud died on it for my today...

Same problem here.

kevin92dev picture kevin92dev on 26 Sep 2018

HTTP Status 500 – Internal Server Error
Type Exception Report

Message TenantContextCommand short-circuited and fallback disabled.

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback disabled.
com.netflix.hystrix.AbstractCommand.handleFallbackDisabledByEmittingError(AbstractCommand.java:1052)
com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:878)
com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:982)
com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:557)
com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.subscribe(Observable.java:10247)
rx.Observable.subscribe(Observable.java:10214)
rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:58)
com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:132)
com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:78)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:158)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:72)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.security.ratelimit.AbstractJiraRateLimitingFilter.doFilter(AbstractJiraRateLimitingFilter.java:74)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:85)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:73)
com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:41)
com.atlassian.jira.internalstatus.RequestStatusFilter.doFilterOnce(RequestStatusFilter.java:54)
com.atlassian.jira.web.filters.FilterOnceAbstractHttpFilter.doFilter(FilterOnceAbstractHttpFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
Root Cause

java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:979)
com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:557)
com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.subscribe(Observable.java:10247)
rx.Observable.subscribe(Observable.java:10214)
rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:58)
com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:132)
com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:78)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:158)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:72)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.security.ratelimit.AbstractJiraRateLimitingFilter.doFilter(AbstractJiraRateLimitingFilter.java:74)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:85)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:73)
com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:41)
com.atlassian.jira.internalstatus.RequestStatusFilter.doFilterOnce(RequestStatusFilter.java:54)
com.atlassian.jira.web.filters.FilterOnceAbstractHttpFilter.doFilter(FilterOnceAbstractHttpFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
Note The full stack trace of the root cause is available in the server logs.

Apache Tomcat/9.0.10

saicharan566 picture saicharan566 on 26 Sep 2018

@pcj @jeroenhabets @jpita guess its time for a small 'break' :D

avrebarra picture avrebarra on 26 Sep 2018
🎉4 😄4

Why did you kill Jira?

Anruin picture Anruin on 26 Sep 2018
😄3

@pcj @jeroenhabets @jpita guess its time for a small 'break' :D

@shrotavre just in time coffee 😄

jpita picture jpita on 26 Sep 2018
😄2

It's alive!

kevin92dev picture kevin92dev on 26 Sep 2018

@shrotavre no break for JIRA engineering team. I guest a couple seconds ago they read this issue. :D

husnulhamidiah picture husnulhamidiah on 26 Sep 2018

@kevin92dev No it is not.

phonemkyaw picture phonemkyaw on 26 Sep 2018
👍2

someone please put 50p in the meter.

smashmirrorcardboardface picture smashmirrorcardboardface on 26 Sep 2018
😄1

@ph0nemyintkyaw for me Confluence tries to get up, but Jira still doesn't work.
You can subscribe for updates here: https://status.atlassian.com/

Anruin picture Anruin on 26 Sep 2018
👍2

Circuit breaker opens again in JIRA :laughing:

DrudgeRajen picture DrudgeRajen on 26 Sep 2018
😄3 🎉1

they should use a better breaker :thinking: like 100 AMP maybe

bizzr3 picture bizzr3 on 26 Sep 2018
😄5 👍1

Jira and confluence both still down for me

Hobbit44 picture Hobbit44 on 26 Sep 2018

Seems like a highest priority and impact!🙃

kami4ka picture kami4ka on 26 Sep 2018

Coffee anyone?

toninofox picture toninofox on 26 Sep 2018
😄3

I am having the same issue on the confluence.

TehreemNisa picture TehreemNisa on 26 Sep 2018

Back to work everyone !! All systems are operational

Tuiituiii picture Tuiituiii on 26 Sep 2018

Seems to be back up for me.

Hobbit44 picture Hobbit44 on 26 Sep 2018

when i increase load on the service. i am getting this issue. i am using 'spring-cloud-starter-hystrix', what is the fix for this?

some error happened stacktrace:com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:979)
[]2019-08-19T12:31:31,200Z INFO c.g.a.o.a.f.ProxyEndPointFallback - Client fallback called for the cause : Hystrix circuit short-circuited and is OPEN
[]2019-08-19T12:31:31,201Z INFO c.g.a.o.a.f.ProxyEndPointFallback - Unknown exception occurred
java.lang.ClassCastException: java.lang.RuntimeException cannot be cast to org.springframework.web.client.HttpClientErrorException
at com.gdn.affiliate.outbound.api.fallback.ProxyEndPointFallback.doPostCall(ProxyEndPointFallback.java:81)
at sun.reflect.GeneratedMethodAccessor343.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at feign.hystrix.HystrixInvocationHandler$1.getFallback(HystrixInvocationHandler.java:123)
at com.netflix.hystrix.HystrixCommand$3.call(HystrixCommand.java:322)
at com.netflix.hystrix.HystrixCommand$3.call(HystrixCommand.java:318)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.Observable.unsafeSubscribe(Observable.java:10151)`

this is the stacktrace

nitish076 picture nitish076 on 19 Aug 2019

Facing the same issue...

naranglakshay picture naranglakshay on 22 Jul 2020

Any updates on this issue? It works for less than 20 failed requests but when I increase the load circuit getting following error
com.netflix.hystrix.exception.HystrixRuntimeException: short-circuited and no fallback available

pratikdandavate picture pratikdandavate on 23 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

chrisgray picture chrisgray  Â·  4Comments
Ahmed-A-Syed picture Ahmed-A-Syed  Â·  7Comments
aliostad picture aliostad  Â·  6Comments
jordandandan picture jordandandan  Â·  5Comments
namjug-kim picture namjug-kim  Â·  8Comments