Hystrix: java.lang.IllegalStateException: Another strategy was already registered (not during testing)

[tpetach1]

After adding HystrixMetricsServlet to war file, we sometimes get 'IllegalStateException: Another strategy was already registered' when first service hit occurs after tomcat startup. If this occurs, all calls to the war will fail. This does not occur every time we start the tomcat/webapp, just sometimes.

Also, we have several tomcats/wars and it does not happen to all of them, only some. All have similar configuration in web.xml in same order.

I have not been able to isolate what may be causing this.
Have seen other issues reported about this message that have to do with testing and initializing more than once within a test, but I am not talking about test. This is - stop tomcat, deploy, start tomcat - situation.

Any information regarding what kind of situation might be causing this?

using hystrix-core-1.4.21.jar, hystrix-metrics-event-stream-1.4.21.jar

example web.xml

xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
id="WebApp_ID" version="2.5">
app1

<welcome-file-list>
    <welcome-file>index.html</welcome-file>
    <welcome-file>index.htm</welcome-file>
    <welcome-file>index.jsp</welcome-file>
</welcome-file-list>

    <context-param>
        <param-name>com.newrelic.agent.APPLICATION_NAME</param-name>
        <param-value>app1</param-value>
</context-param>

    <context-param>
        <param-name>contextConfigLocation</param-name>
     <param-value>WEB-INF/swaggerContext.xml,WEB-INF/cxf-config.xml</param-value> 
    </context-param>

<listener>
  <listener-class>somepath.StartupListener</listener-class>
</listener>
<listener>
  <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>
<listener>
    <listener-class>somepath.circuitbreaker.CircuitBreakerContextListener</listener-class>
</listener>

<filter>
  <display-name>CircuitBreakerRequestContextFilter</display-name>
  <filter-name>CircuitBreakerRequestContextFilter</filter-name>
  <filter-class>somepath.circuitbreaker.filter.CircuitBreakerRequestContextFilter</filter-class>
</filter>
<filter-mapping>
  <filter-name>CircuitBreakerRequestContextFilter</filter-name>
  <url-pattern>/*</url-pattern>

timing
somepath.filter.TimingFilter


timing
/*

<servlet>
    <servlet-name>CXFServlet</servlet-name>
    <servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
    <init-param>
        <param-name>redirects-list</param-name>
        <param-value>.*\.jsp</param-value>
    </init-param>
    <init-param>
        <param-name>redirect-servlet-name</param-name>
        <param-value>jsp</param-value>
    </init-param>
</servlet>
<servlet-mapping>
    <servlet-name>CXFServlet</servlet-name>
    <url-pattern>/*</url-pattern>
</servlet-mapping>

<servlet>  
   <description></description>  
   <display-name>HystrixMetricsStreamServlet</display-name>  
   <servlet-name>HystrixMetricsStreamServlet</servlet-name>  
   <servlet-class>com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet</servlet-class>
</servlet> 
<servlet-mapping>  
   <servlet-name>HystrixMetricsStreamServlet</servlet-name>  
   <url-pattern>/hystrix.stream</url-pattern> 
</servlet-mapping> 

 <filter>
      <display-name>HystrixLoggingFilter</display-name>
      <filter-name>HystrixLoggingFilter</filter-name>
      <filter-class>somepath.hystrix.filter.HystrixLoggingFilter</filter-class>
 </filter>
 <filter-mapping>
      <filter-name>HystrixLoggingFilter</filter-name>
      <url-pattern>/*</url-pattern>
 </filter-mapping>

<listener>
    <listener-class>somepath.healthcheck.HealthCheckListener</listener-class>
</listener>
<servlet>
    <servlet-name>HealthCheckServlet</servlet-name>
    <servlet-class>com.codahale.metrics.servlets.HealthCheckServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>HealthCheckServlet</servlet-name>
    <url-pattern>/health/*</url-pattern>
</servlet-mapping>
<filter>
    <filter-name>HealthCheckFullFilter</filter-name>
    <filter-class>somepath.health.servlets.HealthCheckFullFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>HealthCheckFullFilter</filter-name>
    <url-pattern>/health/full/*</url-pattern>
</filter-mapping>

<servlet>
    <servlet-name>PingServlet</servlet-name>
    <servlet-class>com.codahale.metrics.servlets.PingServlet</servlet-class>
</servlet>

<servlet-mapping>
    <servlet-name>PingServlet</servlet-name>
    <url-pattern>/ping/*</url-pattern>
</servlet-mapping>

<filter>
    <filter-name>PingFilter</filter-name>
    <filter-class>somepath.health.servlets.PingFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>PingFilter</filter-name>
    <url-pattern>/ping/*</url-pattern>
</filter-mapping>

[mattrjacobs]

@tpetach1 I've also run into this in practice. The root cause is that all of the objects held by HystrixPlugins must be singletons, and must be in place upon the first HystrixCommand invocation. As an example, the HystrixConcurrencyStrategy must be non-null while the first HystrixCommand is executing, so that it knows how to wrap Callables and set up a HystrixRequestContext.

The 2 paths for generating the HystrixConcurrencyStrategy are:

• No explicit initialization, and the first command invocation sets up the default HystrixConcurrencyStrategy.
• Explicit initialization via HystrixPlugins.registerConcurrencyStrategy

Note that the latter fails with the IllegalStateException with cause "Another strategy was already registered" if the implicit initialization runs first.

The above is also true for any of the other objects in HystrixPlugins.

So what must be happening is that your application startup ends up creating a race between the first command running (and performing implicit initialization), and an invocation of one of the explicit init calls. If the implicit goes first by a command executing, then you're seeing the error above.

If possible, you should move the explicit plugin init as early as possible to avoid that race. In practice, that has worked for us internally at Netflix.

The full set of calls to look for is:

• HystrixPlugins.registerEventNotifier
• HystrixPlugins.registerConcurrencyStrategy
• HystrixPlugins.registerMetricsPublisher
• HystrixPlugins.registerPropertiesStrategy
• HystrixPlugins.registerCommandExecutionHook

[tpetach1]

So, I have been continuing to fight with this and I have a question.

Sounds like the entity throwing the exception is saying the strategy is already registered.
In that case, could I just catch the error and go on? I know this is not the optimal solution, but would it work?

I see the explicit registration is in the init() of the CircuitBreakerAspect. The circuit breaker wraps the DAO calls.
This is the registration statement that blows up.
It does not happen until the first call wrapped with the circuit breaker

Code fragment:

---

@Aspect
@Component
public class CircuitBreakerAspect
{
private static final DSLogger logger = DSLogger.getLogger(CircuitBreakerAspect.class);

   private static boolean isInitialized = false;

   public static synchronized void init()
   {
          if(!isInitialized)
          {
          logger.info("CircuitBreakerAspect.init() registering ConcurrencyStrategy!!");
           HystrixPlugins.getInstance().registerConcurrencyStrategy(new CDMConcurrencyStrategy());
           isInitialized = true;
          }
   }

---

I have not been successful in figuring out how to force the aspect to always load before the metrics servlet does the implicit registration.

Or would you suggest taking the registerConcurrencyStrategy() out of the aspect class and putting it in
a ServletContextListener.contextInitialized()? Would that even solve it?

Since I don’t see any debug from the metrics servlet I am not sure exactly when it is doing its registration.

From: Matt Jacobs [mailto:[email protected]]
Sent: Tuesday, January 19, 2016 7:41 PM
To: Netflix/Hystrix [email protected]
Cc: Petach, Teresa [email protected]
Subject: Re: [Hystrix] java.lang.IllegalStateException: Another strategy was already registered (not during testing) (#1057)

@tpetach1https://github.com/tpetach1 I've also run into this in practice. The root cause is that all of the objects held by HystrixPlugins must be singletons, and must be in place upon the first HystrixCommand invocation. As an example, the HystrixConcurrencyStrategy must be non-null while the first HystrixCommand is executing, so that it knows how to wrap Callables and set up a HystrixRequestContext.

The 2 paths for generating the HystrixConcurrencyStrategy are:

• No explicit initialization, and the first command invocation sets up the default HystrixConcurrencyStrategy.
• Explicit initialization via HystrixPlugins.registerConcurrencyStrategy

Note that the latter fails with the IllegalStateException with cause "Another strategy was already registered" if the implicit initialization runs first.

The above is also true for any of the other objects in HystrixPlugins.

So what must be happening is that your application startup ends up creating a race between the first command running (and performing implicit initialization), and an invocation of one of the explicit init calls. If the implicit goes first by a command executing, then you're seeing the error above.

If possible, you should move the explicit plugin init as early as possible to avoid that race. In practice, that has worked for us internally at Netflix.

The full set of calls to look for is:

• HystrixPlugins.registerEventNotifier
• HystrixPlugins.registerConcurrencyStrategy
• HystrixPlugins.registerMetricsPublisher
• HystrixPlugins.registerPropertiesStrategy
• HystrixPlugins.registerCommandExecutionHook

—
Reply to this email directly or view it on GitHubhttps://github.com/Netflix/Hystrix/issues/1057#issuecomment-173075976.

[mattrjacobs]

@tpetach1 You have the option of just catching and recovering from the IllegalStateException, but then your custom concurrency strategy isn't registered. Presumable, something else is winning the race to register a HystrixConcurrenyStrategy, and in my experience, it's very often command's being executed while startup is not fully completed.

I think moving it to servlet init likely should move it early and sounds like a good idea.

[mattrjacobs]

Closing due to inactivity. Please re-open if there's more to discuss.

[yeikel]

I encountered this problem when I add spring dev tools to my project. Is there any workaround for this? If I remove it , everything works just fine

Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.lang.IllegalStateException: Another strategy was already registered.
    at com.netflix.hystrix.strategy.HystrixPlugins.registerCommandExecutionHook(HystrixPlugins.java:328)