Jetty.project: Using WebSocketClient with jetty-websocket-httpclient.xml in a Jetty web application causes ClassCastException

[michael-nilson-yardi]

Jetty version
9.4.31
Java version
OpenJDK 1.8.0_251
OS type/version
Windows 10 64 bit
Description
Typically, using a JSR-356 javax.websocket.ClientEndpoint annotated websocket client with the javax.websocket.ContainerProvider$WebSocketContainer works fine within a Jetty web application.

However, when you add the jetty-websocket-httpclient.xml file to perform any configuration (e.g. for the ssl context factory) you will see an exception similar to below.

java.lang.ClassCastException: org.eclipse.jetty.client.HttpClient cannot be cast to org.eclipse.jetty.client.HttpClient
    at org.eclipse.jetty.websocket.client.XmlBasedHttpClientProvider.get (XmlBasedHttpClientProvider.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.eclipse.jetty.websocket.client.HttpClientProvider.get (HttpClientProvider.java:37)
    at org.eclipse.jetty.websocket.client.WebSocketClient.<init> (WebSocketClient.java:264)
    at org.eclipse.jetty.websocket.jsr356.ClientContainer.<init> (ClientContainer.java:158)
    at org.eclipse.jetty.websocket.jsr356.server.ServerContainer.<init> (ServerContainer.java:94)
    at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer.initialize (WebSocketServerContainerInitializer.java:210)
    at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer.onStartup (WebSocketServerContainerInitializer.java:277)
    at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup (ContainerInitializer.java:145)
    at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart (ServletContainerInitializersStarter.java:64)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start (AbstractLifeCycle.java:72)
...

This can easily be reproduced by creating a new web-app that contains only a basic jetty.xml, web.xml and jetty-websocket-httpclient.xml (and pom.xml assuming you are using maven). The above ClassCastException will be thrown shortly before the "Started Jetty Server" message.

On a possibly related note, if you instead use the Jetty implemented websocket (org.eclipse.jetty.websocket.client.WebSocketClient) and use the constructor that takes an HttpClient you receive a LinkageError similar to this:

java.lang.LinkageError: loader constraint violation: when resolving method "org.eclipse.jetty.websocket.client.WebSocketClient.<init>(Lorg/eclipse/jetty/client/HttpClient;)V" the class loader (instance of org/eclipse/jetty/webapp/WebAppClassLoader) of the current class, com/example/WebSocketClientWrapper, and the class loader (instance of org/codehaus/plexus/classworlds/realm/ClassRealm) for the method's defining class, org/eclipse/jetty/websocket/client/WebSocketClient, have different Class objects for the type org/eclipse/jetty/client/HttpClient used in the signature
    at com.example.WebSocketClientWrapper.<init>(WebSocketClientWrapper.java:23)
...

[joakime]

You have org.eclipse.jetty.client.HttpClient in your WEB-INF/lib/ too.
Remove it from there.

[michael-nilson-yardi]

I have removed the jetty-client.jar which contains HttpClient from my WEB-INF/lib/ and now am getting a java.lang.ClassNotFoundException: org.eclipse.jetty.client.HttpClient. I'm assuming I need the following modules in my start.ini so I have added them server,client,webapp,websocket

[joakime]

@lachlan-roberts can you take a look at this issue?

[michael-nilson-yardi]

We stumbled across this article yesterday and it seemed very similar to our circumstances when encountering the LinkageError. We gave it a try both with the approach that worked for the OP as well as the alternate approach you had provided Joakime but we still had the same result; both when programatically manipulating the SslContextFactory and when relying on the jetty-websocket-httpclient.xml approach.

[joakime]

The LinkageError is when you have the same class in multiple classloaders, but from different origins, and there's an attempt to use the classes across the classloaders.

The jetty-websocket-httpclient.xml is loaded by the org.eclipse.jetty.websocket.client.WebSocketClient class.
The WebSocketClient loads the org.eclipse.jetty.client.HttpClient class from the server classloader too.

Both the WebSocketClient and the HttpClient should only exist on the server classloader.
Removing the jetty-client.jar from your WEB-INF/lib was a partial solution.

The websocket-client.jar should also be removed, along with any other Jetty specific jar files in WEB-INF/lib, as there is rarely a reason for them to be in WEB-INF/lib.

Your exception occurs at org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer which is a server component that loads the javax.websocket implementation from the Server classloader and configures it into the webapp's ServletContext layer.

[michael-nilson-yardi]

I've been attempting to recreate the problem and have used your sample project found here as a starting point. I have adapted it slightly to add the config to make it a webapp and add a servlet for testing purposes. I've uploaded my repo here. I've uploaded a gist including the jetty log here. As you can see on line 608, when running it without the jetty jars included in web-inf/lib we are encountering a ClassNotFoundException for HttpClient when it parses the jetty-websocket-httpclient.xml file.

I'm certain this is something I'm doing incorrectly for config on my jetty-distribution but haven't been able to nail it down yet. Any help would be appreciated. Thanks for your feedback thus far, it has been helpful.

[michael-nilson-yardi]

I recognized that we were still dropping a lot of warnings due to having the javax.servlet jar included in our web-inf/lib. I've pulled that jar out and uploaded a new gist that contains the startup logs. Still the same issue though: ClassNotFoundException for org.eclipse.jetty.client.HttpClient.

[lachlan-roberts]

@michael-nilson-yardi I have been able to replicate this issue and I don't think it is due to incorrect configuration on your part. You can try adding these lines to your start.ini as a workaround while we work on a better solution.

jetty.webapp.addSystemClasses+=,org.eclipse.jetty.client.
jetty.webapp.addServerClasses+=,-org.eclipse.jetty.client.
jetty.webapp.addSystemClasses+=,org.eclipse.jetty.util.ssl.
jetty.webapp.addServerClasses+=,-org.eclipse.jetty.util.ssl.

[michael-nilson-yardi]

Thanks for the help @lachlan-roberts . After adding the above config we saw the below output in the logs after making a websocket connection.

2020-09-28 09:07:29.969:DBUG:oejw.WebAppContext:Thread-20: isServerClass==true interface org.eclipse.jetty.util.component.LifeCycle
Exception in thread "Thread-20" java.lang.NoClassDefFoundError: org/eclipse/jetty/util/component/LifeCycle
    at org.eclipse.jetty.demo.App.lambda$stop$0(App.java:47)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.component.LifeCycle
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:565)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 2 more

I tried adding similar config for org.eclipse.jetty.util.component which got us past the NoClassDefFoundError for LifeCycle but then the jetty server would stop immediately after the websocket connection closed. I can get more debug info if it is helpful.

[lachlan-roberts]

@michael-nilson-yardi if you are calling LifeCycle.start(webSocketClient) or LifeCycle.stop(webSocketClient) try replacing that with webSocketClient.start() / webSocketClient.stop().

Or try adding this to start.ini

jetty.webapp.addSystemClasses+=,org.eclipse.jetty.util.
jetty.webapp.addServerClasses+=,-org.eclipse.jetty.util.

[michael-nilson-yardi]

@lachlan-roberts the issue with jetty stopping was related to an oversight in our sample project. Will there be an issue we can track for watching for the "better solution"? In the meantime we'll proceed with testing the start.ini config you have provided with our actual product instead of the sample. Thanks again.

[joakime]

This is the issue to track.

The suggestion from @lachlan-roberts is merely a temporary workaround until we can get a fix into place (and a release with it in place).

[lachlan-roberts]

This is fixed in 9.4.34.

[lachlan-roberts]

Fixed int 10.0.x with PR #5563.