Elasticsearch: TransportClient's ClusterStatsResponse deserialization failing within JBoss 6.1 (new issue in 5.0.0-alpha3)

[fassisrosa]

Within JBoss 6.1, calling TransportClient::admin().cluster().prepareClusterStats().get() breaks in 5.0.0-alpha3, used to work fine in 5.0.0-alpha2

Elasticsearch version: 5.0.0-alpha3

JVM version: 1.8.0_92

OS version: Windows 2012 R2

Description of the problem including expected versus actual behavior:

We are using TransportClient within JBoss 6.1.

In 5.0.0-alpha2 we were able to make the following call:

ClusterStatsResponse clusterStatsResponse = client.admin().cluster().prepareClusterStats().get();

Now when we do the same call in 5.0.0-alpha3 we are getting the following exception:

java.nio.file.FileSystemNotFoundException: Provider "vfs" not installed

Steps to reproduce:

1. Have the call to TransportClient::admin().cluster().prepareClusterStats().get() within JBoss 6.1

Provide logs (if relevant): Full exception from logs (only client side, not errors on elasticsearch side):

TransportSerializationException[Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse]]; nested: ExceptionInInitializerError; nested: FileSystemNotFoundException[Provider "vfs" not installed];
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:172)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:143)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readFrom(NodeInfo.java:199)
at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readNodeInfo(NodeInfo.java:191)
at org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodeResponse.readFrom(ClusterStatsNodeResponse.java:85)
at org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodeResponse.readNodeResponse(ClusterStatsNodeResponse.java:74)
at org.elasticsearch.common.io.stream.StreamInput.readList(StreamInput.java:769)
at org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse.readNodesFrom(ClusterStatsResponse.java:102)
at org.elasticsearch.action.support.nodes.BaseNodesResponse.readFrom(BaseNodesResponse.java:110)
at org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse.readFrom(ClusterStatsResponse.java:85)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:169)
... 23 more
Caused by: java.nio.file.FileSystemNotFoundException: Provider "vfs" not installed
at java.nio.file.Paths.get(Paths.java:147)
at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:75)
at org.elasticsearch.Build.getElasticsearchCodebase(Build.java:87)
at org.elasticsearch.Build.(Build.java:50)
... 32 more

Describe the feature:

Looks like some changes were made into alpha3 in the way the ClusterStatsResponse object is deserialized, this is impacting our client usage within JBoss 6.1.

Details of cause of issue:

+) Making the call to getting ClusterStatsResponse, elasticsearch server returns stream that needs to be deserialized into ClusterStatsResponse.
+) ClusterStatsResponse inherits from BaseNodesResponse.
+) readFrom method in BaseNodesResponse now has a call to 'readNodesFrom' (line 110 of BaseNodesResponse, alpha3 tag).
+) 'readNodesFrom' in ClusterStatsResponse calls readNodeResponse (line 102 of ClusterStatsResponse, alpha3 tag).
+) Eventually call makes it to ClusterStatsNodeResponse::readFrom, taht class NodeInfo.readNodeInfo (line 85 of ClusterStatsNodeResponse, alpha3 tag).
+) NodeInfo::readNodeInfo ends up calling NodeInfo::readFrom which calls Build.readBuild (line 199 of NodeInfo, alpha 3 tag).
+) Calling Build.readBuild forces the Build class static block to be executed to populate 'CURRENT'.
+) The code in getElasticsearchCodebase uses java nio (see lines 85-87 of Build, alpha3 tag). The URL returned within JBoss 6.1 is 'vfs://'.
*) There is no file system provider for 'vfs://' so exception is thrown.

This use is only for client, which probably does not care much for Build.CURRENT anyway?
Could this code be changed to either lazily calculate 'CURRRENT' or to set it to 'Unknown' if an exception is thrown attempting
to call 'getLeasticsearchCodebase' (probably better as we cannot really calculate the current build with current code)?
As it stands, this code cannot be used within JBoss 6.1.

[jasontedor]

This has nothing to do with the serialization of the ClusterStatsNodeResponse, vfs is a JBoss thing and this silliness with JBoss and jar paths is a known issue. That said, I think we can address this. Can you try building Elasticsearch with this patch and let me know if it resolves the issue? I'll follow-up with a PR if it does.

diff --git a/core/src/main/java/org/elasticsearch/Build.java b/core/src/main/java/org/elasticsearch/Build.java
index f844e3b..7b2b829 100644
--- a/core/src/main/java/org/elasticsearch/Build.java
+++ b/core/src/main/java/org/elasticsearch/Build.java
@@ -19,16 +19,11 @@

 package org.elasticsearch;

-import org.elasticsearch.common.SuppressForbidden;
-import org.elasticsearch.common.io.PathUtils;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;

 import java.io.IOException;
-import java.net.URISyntaxException;
 import java.net.URL;
-import java.nio.file.Files;
-import java.nio.file.Path;
 import java.util.jar.JarInputStream;
 import java.util.jar.Manifest;

@@ -47,9 +42,9 @@ public class Build {
         final String date;
         final boolean isSnapshot;

-        Path path = getElasticsearchCodebase();
-        if (path.toString().endsWith(".jar")) {
-            try (JarInputStream jar = new JarInputStream(Files.newInputStream(path))) {
+        final URL url = getElasticsearchCodebase();
+        if (url.toString().endsWith(".jar")) {
+            try (JarInputStream jar = new JarInputStream(url.openStream())) {
                 Manifest manifest = jar.getManifest();
                 shortHash = manifest.getMainAttributes().getValue("Change");
                 date = manifest.getMainAttributes().getValue("Build-Date");
@@ -80,14 +75,8 @@ public class Build {
     /**
      * Returns path to elasticsearch codebase path
      */
-    @SuppressForbidden(reason = "looks up path of elasticsearch.jar directly")
-    static Path getElasticsearchCodebase() {
-        URL url = Build.class.getProtectionDomain().getCodeSource().getLocation();
-        try {
-            return PathUtils.get(url.toURI());
-        } catch (URISyntaxException bogus) {
-            throw new RuntimeException(bogus);
-        }
+    static URL getElasticsearchCodebase() {
+        return Build.class.getProtectionDomain().getCodeSource().getLocation();
     }

     private String shortHash;
diff --git a/core/src/test/java/org/elasticsearch/BuildTests.java b/core/src/test/java/org/elasticsearch/BuildTests.java
index f02d2f2..7bff832 100644
--- a/core/src/test/java/org/elasticsearch/BuildTests.java
+++ b/core/src/test/java/org/elasticsearch/BuildTests.java
@@ -22,16 +22,16 @@ package org.elasticsearch;
 import org.elasticsearch.test.ESTestCase;

 import java.io.IOException;
-import java.nio.file.AccessMode;
-import java.nio.file.Path;
+import java.io.InputStream;
+import java.net.URL;

 public class BuildTests extends ESTestCase {

     /** Asking for the jar metadata should not throw exception in tests, no matter how configured */
     public void testJarMetadata() throws IOException {
-        Path path = Build.getElasticsearchCodebase();
+        URL url = Build.getElasticsearchCodebase();
         // throws exception if does not exist, or we cannot access it
-        path.getFileSystem().provider().checkAccess(path, AccessMode.READ);
+        try (InputStream ignored = url.openStream()) {}
         // these should never be null
         assertNotNull(Build.CURRENT.date());
         assertNotNull(Build.CURRENT.shortHash());

[fassisrosa]

I completely agree this is a JBoss silliness, it just reflects on the deserialization of this instance... Thanks for the super quick response... I'll give it a try and post my results here.

[fassisrosa]

Works like a charm for me (as url for JBoss ends in 'jar/' not 'jar' this ends up in 'Unknown' version).
If you can make this go into product (alpha4 onwards) that would be absolutely fantastic.

Thanks so much for the incredibly quick turnaround!

[jasontedor]

Works like a charm for me (as url for JBoss ends in 'jar/' not 'jar' this ends up in 'Unknown' version.

More silliness; this appears to be JBCL-177 and I'm inclined to _not_ work around it here but at least we avoid the vfs provider not found now.

If you can make this go into product (alpha4 onwards) that would be absolutely fantastic.

I'll try, no promises on alpha4.

Thanks so much for the incredibly quick turnaround!

Thanks for testing!

[fassisrosa]

No worries on JBCL-177. The trailing '/' does not impact me at all... it does generate the 'Unknown' but I can live with it for now. Avoiding the 'vfs' provider issue does it.

Whatever you can do to make it to product is really appreciated. :)

[jasontedor]

Whatever you can do to make it to product is really appreciated. :)

The fix is integrated and will be in alpha4. :smile:

[jasontedor]

I've marked you as eligible for the Pioneer Program; thanks for reporting!

[fassisrosa]

Thanks! :)