Elasticsearch: 6.7 docker build_type breaking 6.6 client

[brackxm]

a 6.6 high level rest client cannot parse the MainResponse from a 6.7 server
due to the new docker build_type (#39378 )
previously a docker distribution was returning the tar type

this prevents 6.6 -> 6.7 upgrades with server before client
client before server upgrades are also not possible due to other 7.0 forward compatibility changes in 6.7
this leaves no 6.6 -> 6.7 or 7 upgrade path

[elasticmachine]

Pinging @elastic/es-core-features

[javanna]

Thanks for reporting this @brackxm , this is indeed a problem. We will discuss possible solutions with the team.

[javanna]

cc @hub-cap I marked this team-discuss .

[jasontedor]

Thanks for reporting this @brackxm and sorry for the issue. As @javanna mentions, we will consider some options and see what we can come up with.

[hub-cap]

Hi @brackxm, Thank you for pointing this out. This did not get caught due to some testing inefficiencies, which I will be hopefully addressing soon. But in the meantime I plan on going thru all of our response parsers manually to see if they suffer from this issue potentially, and if so, correct them such they will not hamper compatibility between versions. #40705 is the first which addresses the parsing problem of this class.

[iremmats]

Would there be any risc overriding the /usr/share/elasticsearch/bin/elasticsearch-env file and at the bottom of it change ES_DISTRIBUTION_TYPE=docker to ES_DISTRIBUTION_TYPE=tar? I just tried this and it made it possible to connect a 6.7.0 node to a 6.6.2 cluster.

[dongs0104]

@iremmats #40723 will return same result as you https://github.com/elastic/elasticsearch/issues/40511#issuecomment-478910697

[dongs0104]

1. build new docker file from official build and change ES_DISTRIBUTION_TYPE

FROM docker.elastic.co/elasticsearch/elasticsearch:6.7.0

RUN grep ES_DISTRIBUTION_TYPE=docker /usr/share/elasticsearch/bin/elasticsearch-env \
    && sed -ie 's/ES_DISTRIBUTION_TYPE=docker/ES_DISTRIBUTION_TYPE=tar/' /usr/share/elasticsearch/bin/elasticsearch-env

1. rolling upgrade for every nodes MUST Check Elastic cluster version > 6.7.

   • that issue is shown under 6.7.0 Cluster only. (docker type shown at 6.7.0)

2. repeat rolling upgrade(Change Docker images) using official build from ES(docker.elastic.co/elasticsearch/elasticsearch:6.7.0)

It should be work.

[ItamarBenjamin]

@dongs0104
if i got this right, need to roll upgrade the entire cluster using the modified build, and once the entire cluster runs on that build do another rolling upgrade to the original build?

[dongs0104]

@dongs0104
if i got this right, need to roll upgrade the entire cluster using the modified build, and once the entire cluster runs on that build do another rolling upgrade to the original build?

I think when you changed every master nodes it will be work but that is not tested

[ItamarBenjamin]

@dongs0104
thanks, it worked! i didn't do this just for the masters though. installed the entire cluster with the fixed version, then installed the real version again.

[brackxm]

will #40705 be backported to 6.6.x?

[jasontedor]

@brackxm No, as we do not intend to do another 6.6.x release.