Elasticsearch: [TEST] failure in :qa:full-cluster-restart

[olcbean]

I am not able to run gradle check locally on the current master (91dc1c5da682fd832fc27d07f657643a88f4aae5) as :qa:full-cluster-restart fails.

Using gradle 3.4.1

To reproduce run :

gradle :qa:full-cluster-restart:check

In qa/full-cluster-restart/build/cluster/v5.6.0-SNAPSHOT#oldClusterTestCluster node0/elasticsearch-5.6.0-SNAPSHOT/logs/full-cluster-restart.log :

[2017-07-26T12:38:02,047][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: failed to load plugin - Plugin information:
Name: reindex
Description: The Reindex module adds APIs to reindex from one index to another or update documents in place.
Version: 5.6.0
Native Controller: false
 * Classname: org.elasticsearch.index.reindex.ReindexPlugin due to jar hell
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:376) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:140) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:313) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:341) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.client.http.auth.KerberosCredentials
jar1: /home/olcbean/ramdisk/elasticsearch_clone/qa/full-cluster-restart/build/cluster/v5.6.0-SNAPSHOT#oldClusterTestCluster node0/elasticsearch-5.6.0-SNAPSHOT/modules/reindex/elasticsearch-rest-client-5.6.0-SNAPSHOT-deps.jar
jar2: /home/olcbean/ramdisk/elasticsearch_clone/qa/full-cluster-restart/build/cluster/v5.6.0-SNAPSHOT#oldClusterTestCluster node0/elasticsearch-5.6.0-SNAPSHOT/modules/reindex/elasticsearch-rest-client-5.6.0-SNAPSHOT.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:374) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    ... 13 more
[2017-07-26T12:38:02,056][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin - Plugin information:
Name: reindex
Description: The Reindex module adds APIs to reindex from one index to another or update documents in place.
Version: 5.6.0
Native Controller: false
 * Classname: org.elasticsearch.index.reindex.ReindexPlugin due to jar hell
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
Caused by: java.lang.IllegalStateException: failed to load plugin - Plugin information:
Name: reindex
Description: The Reindex module adds APIs to reindex from one index to another or update documents in place.
Version: 5.6.0
Native Controller: false
 * Classname: org.elasticsearch.index.reindex.ReindexPlugin due to jar hell
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:376) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:140) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:313) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:341) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    ... 6 more
Caused by: java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.client.http.auth.KerberosCredentials
jar1: /home/olcbean/ramdisk/elasticsearch_clone/qa/full-cluster-restart/build/cluster/v5.6.0-SNAPSHOT#oldClusterTestCluster node0/elasticsearch-5.6.0-SNAPSHOT/modules/reindex/elasticsearch-rest-client-5.6.0-SNAPSHOT-deps.jar
jar2: /home/olcbean/ramdisk/elasticsearch_clone/qa/full-cluster-restart/build/cluster/v5.6.0-SNAPSHOT#oldClusterTestCluster node0/elasticsearch-5.6.0-SNAPSHOT/modules/reindex/elasticsearch-rest-client-5.6.0-SNAPSHOT.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:374) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:140) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:313) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:341) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.6.0-SNAPSHOT.jar:5.6.0-SNAPSHOT]
    ... 6 more

[hub-cap]

I wonder if this is related to the fact that it has not yet been backported? Im going to backport #25892 first to see if it fixes things (since that, I believe, is pulling 5.6 source as well)

[hub-cap]

Also, thanks so very much for sticking this pain out w/ me @olcbean ❤️

[hub-cap]

ok, so I believe this is a symptom of this not yet being backported. When I pushed my backported change to my own fork, and used the following flag to change the refspec, I was able to get the tests to pass. Ill be pushing it shortly, I would like to run the full check first on it.

```
gradle -Dtests.bwc.refspec=origin/5.6 :qa:full-cluster-restart:check

Configure project :benchmarks
The getTestClassesDir() method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use the getTestClassesDirs() method instead.

at build_aotl8we4aotrppxbrm01p2umt.run(/home/elastic/elasticsearch/benchmarks/build.gradle:31)

Elasticsearch Build Hamster says Hello!

Gradle Version : 4.0
OS Info : Linux 4.4.0-1018-aws (amd64)
JDK Version : Oracle Corporation 1.8.0_131 [OpenJDK 64-Bit Server VM 25.131-b11]
JAVA_HOME : /usr/lib/jvm/java-8-openjdk-amd64
Random Testing Seed : A4530BE86F6212E0
Gradle now uses separate output directories for each JVM language, but this build assumes a single directory for all classes from a source set. This behaviour has been deprecated and is scheduled to be removed in Gradle 5.0

Task :distribution:bwc-stable-snapshot:checkoutBwcBranch
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

0680895 Fix rest client causing jarHell for gradle 3.5+ (#25892)

If you want to keep it by creating a new branch, this may be a good time
to do so with:

git branch 0680895

HEAD is now at acd41d5... Fix rest client causing jarHell for gradle 3.5+ (#25892)

> Configure project :checkout-5.x:benchmarks

Elasticsearch Build Hamster says Hello!

Gradle Version : 4.0
OS Info : Linux 4.4.0-1018-aws (amd64)
JDK Version : Oracle Corporation 1.8.0_131 [OpenJDK 64-Bit Server VM 25.131-b11]
JAVA_HOME : /usr/lib/jvm/java-8-openjdk-amd64

Configure project :checkout-5.x:plugins:repository-hdfs
Could not find vagrant: A problem occurred starting process 'command 'vagrant''
secured hdfsFixture is unsupported, please install Vagrant 1.8.6+ to enable

Task :checkout-5.x:core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Task :checkout-5.x:modules:ingest-common:compileJava
Note: /home/elastic/elasticsearch/distribution/bwc/build/bwc/checkout-5.x/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RemoveProcessor.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Task :checkout-5.x:modules:lang-expression:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:modules:lang-mustache:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:modules:parent-join:compileJava
Note: /home/elastic/elasticsearch/distribution/bwc/build/bwc/checkout-5.x/modules/parent-join/src/main/java/org/elasticsearch/join/query/ParentChildInnerHitContextBuilder.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:modules:percolator:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:modules:reindex:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:modules:transport-netty3:compileJava
Note: /home/elastic/elasticsearch/distribution/bwc/build/bwc/checkout-5.x/modules/transport-netty3/src/main/java/org/elasticsearch/http/netty3/Netty3HttpChannel.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:modules:transport-netty4:compileJava
Note: /home/elastic/elasticsearch/distribution/bwc/build/bwc/checkout-5.x/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpChannel.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Task :checkout-5.x:distribution:zip:generatePomFileForNebulaPublication
Unable to convert /home/elastic/elasticsearch to https form in MavenScmPlugin. Using original value.

Task :qa:full-cluster-restart:v5.6.0-SNAPSHOT#oldClusterTestRunner
==> Test Info: seed=CE74A4F95E9BA266; jvm=1; suite=1

HEARTBEAT J0 PID(29243@ip-10-0-1-202): 2017-07-26T14:40:56, stalled for 11.7s at: FullClusterRestartIT.testSearch

Task :qa:full-cluster-restart:v5.6.0-SNAPSHOT#oldClusterTestRunner
Slow Tests Summary:
39.14s | org.elasticsearch.upgrades.FullClusterRestartIT

==> Test Summary: 1 suite, 8 tests, 1 ignored (1 assumption)

Task :qa:full-cluster-restart:v5.6.0-SNAPSHOT#upgradedClusterTestRunner
==> Test Info: seed=8B90DCDA0CA22068; jvm=1; suite=1
==> Test Summary: 1 suite, 8 tests, 1 ignored (1 assumption)

BUILD SUCCESSFUL in 2m 24s
128 actionable tasks: 43 executed, 85 up-to-date

[hub-cap]

Feel free to give it a whirl @olcbean , I just backported the fix to 5.6

[hub-cap]

Since I forgot to put it on the commit msg, this is in reference to 236d2e5bae9ba06c901e39f91719c36e43be84a3

[hub-cap]

once CI is happy Ill close this

[olcbean]

I just rerun the gradle check on 6d02b45f105cb1fb018872871ae5e59651cbe284 and it passed.

Thanks @hub-cap for the quick fix :)