Skywalking: There is wrong when I use cluster mode in version 3.2.6.

[hepyu]

version3.2.6.

I use elasticsearch cluster as storage, like this:

the content of collector config

cluster:
zookeeper:
hostPort: ip1:2181, ip2:2181, ip3:2181
sessionTimeout: 100000

storage:
elasticsearch:
cluster_name: trace_online
cluster_transport_sniffer: true
cluster_nodes: ip1:9300, ip2:9300, ip3:9300, ip4:9300
index_shards_number: 2
index_replicas_number: 0
ttl: 7

but when I start collector, exception occur like this:

I am confused a lot.

[hepyu]

The elasticsearch version I used is 5.6.8, is that too high for cluster mode? I used es-5.6.8 in single mode, its ok.

[hepyu]

The exception is threw here:

When collector load index info from es into guava, it found the indexes hadn't been created. But why?

There is no exception threw when judge and create index:

The only reason I could thought out is es version problem, sky-v3.2.6 used only es-5.5.0, I used es-5.6.8.

[wu-sheng]

I don't know that. Could you try the v5? Which are most users using. Also the community supports.

[hepyu]

Yes, I used es-5.6.8, thats v5 either.

And I has changed elasticsearch from v5.6.8 to v5.5.0 that used in skywalking-3.2.6, then its ok.
But the third collector report the exception:
https://github.com/apache/incubator-skywalking/issues/999.
the other two collector nodes has no reporting this exception.

I think this is elasticsearch api conflict between es-client-5.5.0 and es-server-5.6.8, because skywalking operate es just only using es-client-api.

Skywalking is not work for every subversion in elasticsearch-5.x, I think the best choice is that the same version as 5.5.0 that skywalking used, in skywalking version 3.2.6. And not only about skywalking-version-3.2.6, but also skywalking new version.

Keep the es-client-version and the es-server-version as same is not bad choice.

This is necessary, for example, the redis, if redis-client version is not same as redis-server version, it always not work. The conflicts in these good things are so common.

[wu-sheng]

Out driver is 5.5.0, I haven't test 5.6.8 in all cases, someone reported they used.

Keep the es-client-version and the es-server-version as same is not bad choice.

I can't agree more about this. Feel free to send pull request to upgrade es client version to latest of v5.

[hepyu]

In my test, es-v5.6.8 is ok if in the es single mode, but not work in the cluster mode, it will not create index in the es, after changing es to v5.5.0, all work, using the same config.

This is not important, I just describe what I had met, just only to provide the info to community to discuss.

And I want to ask for advice about these three questions:

First, https://github.com/apache/incubator-skywalking/issues/999 exception occur, but skywalking is still work. Is this will affect the usability and stability?

Second, I start three collector nodes using collector cluster mode, and of course there exists three nodes in zookeeper, but how they work? Only one is active and the other two nodes are only backup? Or there nodes are active and provide service at same time?

Third, I found some confused thing about es cluster nodes in zookeeper, I config four es nodes, but in the zk path "/skywalking/storage/elasticsearch", at time point1, there exists two es nodes, at time point2, change to three es nodes, I am confused.

[wu-sheng]

This is not important, I just describe what I had met, just only to provide the info to community to discuss.

I think it is important to do upgrade, really, most of 5.x bug fixes are coming from community. So if you have time, don't hesitate to send pull request to fix.

First, #999 exception occur, but skywalking is still work. Is this will affect the usability and stability?

It should not happen, but we don't find why. And don't receive any report in latest version. Sorry, I don't have the answer.

Second, I start three collector nodes using collector cluster mode, and of course there exists three nodes in zookeeper, but how they work? Only one is active and the other two nodes are only backup? Or there nodes are active and provide service at same time?

All are working, for sure.

Third, I found some confused thing about es cluster nodes in zookeeper, I config four es nodes, but in the zk path "/skywalking/storage/elasticsearch", at time point1, there exists two es nodes, at time point2, change to three es nodes, I am confused.

Is this done by our project? I am not following about this.

[hepyu]

got it.

[hepyu]

I found the reason, my fault, there exist chinese space in the config.