Generator-jhipster: Getting java.lang.IllegalStateException: handshake failed, mismatched cluster name Exception while running embedded Elasticsearch instance

[maheshjadhav007]

I have created UAA service, Gateway and OtpService microservice using Jhipster. but when I ran Gateway and Otpservice , I am getting below exception related to Elastic search cluster. I am using embedded Elasticsearch instance. after throwing below exception services getting registered to Eureka server. I am expecting to run services without exception.

2020-04-26 02:25:11.623 WARN 3372 --- [_connect]][T#6]] o.e.discovery.zen.UnicastZenPing : [1] failed send ping to {127.0.0.1:9300}{JgxF0kBVR-iIrOf73cnwVg}{127.0.0.1}{127.0.0.1:9300}

java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [internal-test-cluster-name146d2dde-6933-4fc3-abea-4e548dce1c15]] - {127.0.0.1:9300}{JgxF0kBVR-iIrOf73cnwVg}{127.0.0.1}{127.0.0.1:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:404)
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:367)
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:366)
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:471)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

2020-04-26 02:25:11.623 WARN 3372 --- [_connect]][T#3]] o.e.discovery.zen.UnicastZenPing : [1] failed send ping to {[::1]:9300}{QX-g01bdRo2aIjad08AccQ}{0:0:0:0:0:0:0:1}{[::1]:9300}

---

Jhipster info

---

INFO! Using JHipster version installed locally in current project's node_modules
INFO! Executing jhipster:info
INFO! Options: from-cli: true
Welcome to the JHipster Information Sub-Generator

JHipster Version(s)

[email protected] D:\jhipster2\gateway
`-- [email protected]

##### **JHipster configuration, a `.yo-rc.json` file generated in the root folder**

.yo-rc.json file

{
  "generator-jhipster": {
    "promptValues": {
      "packageName": "com.tennabo.gatewayservice"
    },
    "jhipsterVersion": "6.2.0",
    "applicationType": "gateway",
    "baseName": "gatewayservice",
    "packageName": "com.tennabo.gatewayservice",
    "packageFolder": "com/tennabo/gatewayservice",
    "serverPort": "8085",
    "authenticationType": "uaa",
    "uaaBaseName": "authservice",
    "cacheProvider": "hazelcast",
    "enableHibernateCache": false,
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "mysql",
    "prodDatabaseType": "mysql",
    "searchEngine": "elasticsearch",
    "messageBroker": false,
    "serviceDiscoveryType": "eureka",
    "buildTool": "maven",
    "enableSwaggerCodegen": false,
    "useSass": true,
    "clientPackageManager": "npm",
    "skipClient": true,
    "testFrameworks": [
      "cucumber"
    ],
    "jhiPrefix": "jhi",
    "entitySuffix": "",
    "dtoSuffix": "DTO",
    "otherModules": [],
    "enableTranslation": false,
    "blueprints": []
  }
}

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory

JDL entity definitions

Environment and Tools

java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

git version 2.23.0.windows.1

node: v10.16.0

npm: 6.14.4

[mraible]

Embedded elasticsearch is used by default in the dev profile:

https://github.com/jhipster/generator-jhipster/blob/master/generators/server/templates/src/main/resources/config/application-dev.yml.ejs#L176

localhost:9200 is used when running with the prod profile:

https://github.com/jhipster/generator-jhipster/blob/master/generators/server/templates/src/main/resources/config/application-prod.yml.ejs#L150

I'm confused because it looks like it's failing to call 9300, yet you say you're running in embedded mode. Did you make any changes to your app after generating it? Embedded elasticsearch + cluster doesn't make sense to me. Can you clarify?

[maheshjadhav007]

Hi matt,

Thanks for respose.

I haven't done any changes in application-dev.yml. Even It confused me. I am attaching my application-dev.yml for your reference.

I am running following applications in my laptop locally.

1)Eureka Server -> Ran perfectly
2)UAA -> Ran Perfectly
3)Gateway -> throws below error. but connect to eureka
4)OTP Service -> throws below error. but connect to eureka

Can you please point out, what should I do next to resolve?
application-dev.txt

`

[mraible]

Can you please wrap the YAML in your previous comment in a markdown code block so it's easier to read?

What's an OTP Service? Is that an app you created with JHipster?

[maheshjadhav007]

Hi matt,

Extremely sorry for inconvenience. I am attaching application-dev file as txt file. Can you please look into it. OTP service application is created with Jhipster.

In my all 3 jhipster application, I am using embedded elastic instance. and I am trying to run on my laptop. Is it fine to run more than one jhipster applications with embedded elastic instance on same machine?

[mraible]

Embedded ES to me means that you're writing to the file system rather than talking to it in a Docker container (or real instance). Do you have the same definition?

I'm trying to understand why you call it embedded rather than just saying you're using JHipster with Elasticsearch.

On Apr 26, 2020, at 07:50, maheshjadhav007 [e-mail] wrote:


Hi mraible,

Extremely sorry for inconvenience. I am attaching application-dev file as txt file. Can you please look into it. OTP service application is created with Jhipster.

In my all 3 jhipster application, I am using embedded elastic instance. and I am trying to run on my laptop. Is it fine to run jhipster application with embedded elastic instance on same machine?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[maheshjadhav007]

Yes matt. Embedded ES writes data in followning location for each microservice.
D:\uaa\target\elasticsearch\data\
D:\gateway\target\elasticsearch\data\
D:\otpservice\target\elasticsearch\data\

I am using JHipster with Elasticsearch. I am trying to run microservices for dev environment.

[mraible]

I haven't experienced this issue, but I have experienced problems when running ES embedded. The problem I experienced is ES fails to restart after it's started once because it thinks the index is corrupted.

If you change your dev ES settings to use the prod settings and start ES in a Docker container, does it solve the problem?

The reason I ask is we might make that the default in JHipster 7.

[maheshjadhav007]

Hi matt,

If I start ES in docker with prod setting, It will work.
but I am more interested in generating application with jhipster and without modifying any setting,
it should run without any exception.

For re-production of same issue, please find below microservices ( changed db password to 'root')
AuthServer -> https://github.com/maheshjadhav007/authserver.git
Gateway -> https://github.com/maheshjadhav007/gateway.git
OTPService -> https://github.com/maheshjadhav007/otpservice.git

To avoid this exception for time being I am disabling elastic search engine property in .yo-rc.json file as following
"searchEngine": "elasticsearch"
to this:
"searchEngine": false

I wanted to use ES in dev environment without any exception.

Thanks,
Mahesh

[mraible]

but I am more interested in generating application with jhipster and without modifying any setting, it should run without any exception.

I agree. It should run without any exceptions. However, this might not be possible. You might've found a bug and we're unable to fix it unless we use Docker in dev mode too. I don't know how to fix your problem. If you figure out a solution, please create a PR.

[maheshjadhav007]

If possible, Can you please check with repositories which I have shared whether you are getting same exception or not. I just wanted to confirm from expert.

[mraible]

@maheshjadhav007 Can you give step-by-step instructions to reproduce the problem? I'm sorry, I've never used UAA so I'm not sure how to run everything. Instructions should be something like:

git clone [email protected]:maheshjadhav007/authserver.git
cd authserver
// Docker Compose commands
// start command

git clone [email protected]:maheshjadhav007/gateway.git
cd gateway
// Docker Compose commands
// start command

git clone [email protected]:maheshjadhav007/otpservice.git
cd otpserver
// Docker Compose commands
// start command

Then steps to reproduce your issue, for example, running npm run e2e in the Gateway project.

[pascalgrimaud]

It's not the first time someone report a problem with embedded Elasticsearch.
Maybe it's time to switch definitively to the docker-compose Elasticsearch in dev too.

[maheshjadhav007]

Hi @mraible ,

Thanks for your reply.

Please find the below steps to reproduce the issue.
There is no need to run docker coomand. You can run below application locally.

1. Start Eureka Server. Use mvnw command from eureka directory if you are using windows OS, else use "./mvnw"
   you can download Eureka Server from below github link
   https://github.com/jhipster/jhipster-registry

1. Start AuthServer. Use mvn spring-boot:run -DskipTests command to run the app without test cases
2. Start Gateway. Use mvn spring-boot:run -DskipTests command
3. Start OtpServer. Use mvn spring-boot:run -DskipTests command

All other services will get Register on Eureka Server Registry. So you have to run Eureka Server first.
After that you should run AuthServer because other microservices required Oauth Token. after that you can run run Gateway Service or OtpService. There is no specific order to run Gateway service or OtpServer.

Thanks in advance.

Mahesh Jadhav.

[mraible]

I'm sorry, but this is a UAA issue and I'm not a supporter of UAA. I recommend developers use OIDC. 😊

Tagging @xetys, who is our UAA stream lead.

[maheshjadhav007]

Hi @mraible ,

I am not getting any issue in UAA. I am getting the IllegalStateException: handshake failed, mismatched cluster name only when I am running more than one microservice.

If I run only Eureka, Gateway Service and UAA Service , I don't get any exception. but when I tried to run otpservice or any other microservice I encounter above mentioned exception.

So the point is here, one microservice ES is trying to find other microservice ES. but there cluster name is mismatched. Hence it throws IllegalStateException: handshake failed, mismatched cluster name.

Thanks matt for your reply.

I am still looking for solution, :)

[pascalgrimaud]

@maheshjadhav007 : just gave a try to your issue, and I can confirm it. Anyway, the explication is really simple:

• your gateway started an embedded Elasticsearch
• your microservice tried to start another embedded Elasticsearch and it failed

You would have the exact problem if your gateway use postgresql (with port 5432) and another microservice with postgresql (with same port 5432), so only one can work.

What I can suggest is :

• change the config in dev for your gateway and your microservice: use the similar config in prod
• start only once the docker-compose yaml file, related to Elasticsearch
• start both applications -> it will work as it will connect to the same Elasticsearch

Then, as there is more an issue related to how to use microservice, there is nothing we can do on our side.
Anyway, we'll probably migrate to docker-compose Elasticsearch in dev for v7, it's far better.

For now, let's close this.