Conductor: Elasticsearch failed to get local cluster state & disconnecting

[nikhil-babaria]

Hi,

I am trying to install Conductor using pre-build binaries which led to this error.
Source: Conductor Server

1505 [main] ERROR com.netflix.conductor.dao.index.ElasticSearchDAO  - None of the configured nodes are available: [{#transport#-1}{52.87.176.195}{52.87.176.195:9200}]
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{52.87.176.195}{52.87.176.195:9200}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:326)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:223)
        at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:295)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
        at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
        at com.netflix.conductor.dao.index.ElasticSearchDAO.initIndex(ElasticSearchDAO.java:179)
        at com.netflix.conductor.dao.index.ElasticSearchDAO.<init>(ElasticSearchDAO.java:139)
        at com.netflix.conductor.dao.index.ElasticSearchDAO$$FastClassByGuice$$9fa9c95d.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:73)
        at com.google.inject.Guice.createInjector(Guice.java:62)
        at com.netflix.conductor.server.ConductorServer.start(ConductorServer.java:203)
        at com.netflix.conductor.server.Main.main(Main.java:54)

Here is my [ PROPERTY FILE ] : config.properties

I have my elastic search version 6.2 running on: http: 9200 && tcp:9300_

This response is received on upon _curl -GET elasticsearch_ip:9200/tutorial/_ :

{"tutorial":{"aliases":{},"mappings":{},"settings":{"index":{"creation_date":"1520513367332","number_of_shards":"3","number_of_replicas":"2","uuid":"ScpEe4WKS_Ottzj7GnqMMQ","version":{"created":"5060899"},"provided_name":"tutorial"}}}}r

AND _netcat -z -v 9300_ :

Connection to <machine_ip> 9300 port [tcp/*] succeeded!

Is there something I need to configure locally in order build it successfully?

Thank you!

[saidatta]

Can you share your config.properties?

[nikhil-babaria]

I have shared the [ PROPERTY FILE ] in the above post.

Here is the updated file : config.properties

Is this the file you are looking for?

[saidatta]

I encountered this error. For me, it was because I didnt allow open 9300 as well for the node's communication.

[nikhil-babaria]

I actually had made sure that all the required ports are open before running the server.

I had checked it using the following command:

root@ip:/home/ubuntu# netcat -z -v <machine_ip> 9300
Connection to <machine_ip> 9300 port [tcp/*] succeeded!

[saidatta]

Are you using Docker by any chance? When I was using docker-compose i had a required healthy check for the individual dependencies before starting the server.

For e.g., Before starting Conductor server, I asked docker to make sure elasticsearch and dynomite return a healthy check as true.

Here is a sample, Look at the depends_on and healthycheck

version: '2.1'
services:
  conductor-server:
    environment:
      - CONFIG_PROP=config.properties
    image: docker.repo.frg.tech/conductor-server:01
    ports:
      - 8080:8080
    depends_on:
      elasticsearch:
        condition: service_healthy
      dynomite:
        condition: service_healthy
    ....
    healthcheck:
      test: "exit 0"

   .....
  dynomite:
    image: docker.repo.frg.tech/dynomite_conductor:01
    volumes:
      - fanatics-dynomite-db:/var/lib/redis:rw
    ...
    healthcheck:
      test: "exit 0"


  elasticsearch:
    image: docker.repo.frg.tech/elasticsearch:2.4
     ....
    healthcheck:
      test: "exit 0"

[nikhil-babaria]

No, I was trying to install Conductor using pre-build binaries which led to this error.
Source: Conductor Server

Do I need to do anything apart from creating an index in elasticsearch? for e.g. do I need to create a default mapping that suits the requirement of Conductor.

Here is the list of indices created on the elasticsearch cluster:

health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   tutorial otknS6UPQDucbO57JgO5PQ   1   1          0            0       690b           690b

[saidatta]

AFAIK, you don't need to. Once you specify the url address the server should be able to figure it out.

Can you make sure your elasticsearch, dynomite are active and running before you start your conductor server?

[nikhil-babaria]

Yes, both the services are active and running on their respective ports
i.e. elasticsearch on ( _http:9200_ & _tcp:9300_ )
and dynomite/redis on _6379_.

[mykhailoguk]

Hi guys,
I'm getting same error with conductor-server-1.9.3-all.jar or conductor-server-1.10.0-SNAPSHOT-all.jar
I'm using redis and it is up and running. Same for elasticsearch. Ports are opened too, I can telnet both of them.
Config file:
config.txt

I'm getting same error too:
1316 [main] ERROR com.netflix.conductor.dao.index.ElasticSearchDAO - None of the configured nodes are available: [{#transport#-1}{}{:9300}]
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{}{:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:326)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:223)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:295)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
at com.netflix.conductor.dao.index.ElasticSearchDAO.initIndex(ElasticSearchDAO.java:181)
at com.netflix.conductor.dao.index.ElasticSearchDAO.(ElasticSearchDAO.java:141)

Started server on http://localhost:8080/
6205 [elasticsearch[Swordsman][generic][T#3]] INFO org.elasticsearch.client.transport - [Swordsman] failed to get local cluster state for {#transport#-1}{}{:9300}, disconnecting...
NodeDisconnectedException[[][:9300][cluster:monitor/state] disconnected]
11215 [elasticsearch[Swordsman][generic][T#1]] INFO org.elasticsearch.client.transport - [Swordsman] failed to get local cluster state for {#transport#-1}{}{:9300}, disconnecting...
NodeDisconnectedException[[][:9300][cluster:monitor/state] disconnected]
^C14203 [Thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=localhost, ipAddress=null, port=6379, rack: us-east-1e, datacenter: us-east-1, status: Up]

Can someone please suggest something?
Thanks!

[kamileon]

Hey guys,

Has anyone tried to run Conductor with AWS ElasticSearch? That would be my preferred choice to reduce the ES HA setup overhead.

When I run Conductor against AWS ElasticSearch I am getting the same exception as below. As far as I understand this is because AWS ES does not expose the TCP port (9300) and conductor requires it. Only the HTTP traffic is allowed to AWS stack.

The real question is does Conductor really require access the TCP transport?

Exception:
1316 [main] ERROR com.netflix.conductor.dao.index.ElasticSearchDAO - None of the configured nodes are available

[kishorebanala]

Well, Conductor uses TransportClient to connect to Elasticsearch, which is not supported by AWS Elasticsearch. NoNodeAvailableException could occur for 2 possible reasons - Elasticsearch cluster wouldn't support TransportClient connections, or the ports on ES server are not opened for clients.

We might upgrade Conductor to use other clients over TransportClient, but we don't have an ETA on this.

[apanicker-nflx]

Closing this ticket since ES2 has been deprecated and ES5 and ES6 modules both support transport/REST client.