Strimzi-kafka-operator: [Question] Elastic Connector - Caused by: java.lang.ClassNotFoundException: com.google.common.collect.ImmutableSet

[tirelibirefe]

Hello;
I'm trying to install ElasticSearch Connector via Strimzi.
My Docker image contains Elastic plugin.

devadmin@vdi-mk2-ubn:~/kafka$ kubectl exec -i kafka-beytepe-kafka-0 -n kafka --  curl -s -XGET http://mssql-minio-connect-cluster-connect-api:8083/connector-plugins|jq '.[].class'
Defaulting container name to kafka.
Use 'kubectl describe pod/kafka-beytepe-kafka-0 -n kafka' to see all of the containers in this pod.
"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector"
"io.confluent.connect.s3.S3SinkConnector"
"io.confluent.connect.storage.tools.SchemaSourceConnector"
"io.debezium.connector.sqlserver.SqlServerConnector"
"org.apache.kafka.connect.file.FileStreamSinkConnector"
"org.apache.kafka.connect.file.FileStreamSourceConnector"
"org.apache.kafka.connect.mirror.MirrorCheckpointConnector"
"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector"
"org.apache.kafka.connect.mirror.MirrorSourceConnector"
"uk.co.threefi.connect.shell.ShellSinkConnector"

...but when I fire up the connector, the following execption is throwed.

`  "name": "elastic-files-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "10.244.1.39:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "FAILED",
      "worker_id": "10.244.1.39:8083",
      "trace": "java.lang.NoClassDefFoundError: com/google/common/collect/ImmutableSet\n\tat io.searchbox.client.AbstractJestClient.<init>(AbstractJestClient.java:38)\n\tat io.searchbox.client.http.JestHttpClient.<init>(JestHttpClient.java:43)\n\tat io.searchbox.client.JestClientFactory.getObject(JestClientFactory.java:51)\n\tat io.confluent.connect.elasticsearch.jest.JestElasticsearchClient.<init>(JestElasticsearchClient.java:149)\n\tat io.confluent.connect.elasticsearch.jest.JestElasticsearchClient.<init>(JestElasticsearchClient.java:141)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:122)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:51)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:305)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.ClassNotFoundException: com.google.common.collect.ImmutableSet\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:382)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n\tat org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n\t... 16 more\n"
    }
  ],
  "type": "sink"
}`

I think If it's CLASSPATH issue, the first command wouldn't list available connector, isn't it? I need to your help again.

Thanks & Regards

[scholzj]

The list of connectors lists only the classes implementing the Kafka connector interfaces. It does IMHO not do any deep checking of all the required classes. So this could be some missing library which contains this class and is not in the classpath?

I do not really know how all the different connectors work. So I cannot help much more.

[tirelibirefe]

ok, I'm closing. It's related with Elastic connector not related with Strimzi.

[tirelibirefe]

Thanks for all your attention @scholzj
it looks like there are no many people face with the problem. ElasticSearch Connector is one of the most important connectors widely used. Even if it is not a Strimzi issue, maybe Strimzi helps me debugging. Does Strimzi help me go into deeper of the problem like debugging or anything else like that?

https://stackoverflow.com/questions/62015340/elasticsearch-connector-doesnt-work-java-lang-noclassdeffounderror-com-googl

[wtayyeb]

me too, i have upgrade strimzi from 0.17 to 0.18, and the connector plugin from 5.4.2 to 5.5.0. also upgrade kafka from 2.4.0 to 2.5.0.
from that point the java.lang.NoClassDefFoundError: com/google/common/collect/ImmutableSet bug is raising and KafkaConnector stopped working.

[wtayyeb]

I review changes to the plugin but don't find anything lead to a bug, they add new compressed config with a false default value only:

https://github.com/confluentinc/kafka-connect-elasticsearch/compare/v5.4.2...v5.5.0

[wtayyeb]

for now, I remain on strimzi 0.18, kafka 2.5.0, but downgrade the connector to
use strimzi-kafka:0.17.0-kafka-2.4.0 +( confluentinc/kafka-connect-elasticsearch 5.4.2 ).
currently connector doesn't raise error and works good.

[tirelibirefe]

hmmmm thanks @wtayyeb
My Strimzi is 0.18 too. I check also others and may downgrade too. I writte the result.

[scholzj]

TBH, I do not think I can help you with this - I just don't know enough about neither Connect nor the particular connector.

The missing class com/google/common/collect/ImmutableSet is IMHO part of the Guava library. That library was removed from Kafka in 2.5. So in 2.4.0 it was part of Apache Kafka and was on the class path by default but n 2.5.0 it is not there anymore as KAfka does not need it. So I guess you need to get the JAR and add it to the directory where your connector JARs are?

I cannot speak for Confluent, but I assume that they develop the connectors primarily for Confluent Platform which is not the same as Apache Kafka - so there might be some JARs in their class path which are not in ours. So if the license allows it, you have the add them to your connectors manually.

[wtayyeb]

@scholzj ok, i found the guava jar for the ImmutableSet. I could add it to custom Dockerfile for elastic connector, but in srimzi/kafka image where do I put it? do I need to extract it or put it as is?

as you mention for license, the guava is licensed under Apache2. i dont think it will hurt, do it?

ADD https://repo1.maven.org/maven2/com/google/guava/guava/29.0-jre/guava-29.0-jre.jar plugins/  # ???

[scholzj]

I guess you would need to try it - but I think you should put it right next to the connector. So for exmaple if you follow this: https://strimzi.io/docs/operators/latest/deploying.html#creating-new-image-from-base-str and add your elastic-search-connector.jar to /opt/kafka/plugins/elastic/elastic-search-connector.jar just add it to the same directory as for example /opt/kafka/plugins/elastic/guava.jar. But I never tried this particular problem. So you might need to try it and see how it works.

[tirelibirefe]

@wtayyeb If you look at my detaily explanation here
[https://stackoverflow.com/questions/62015340/elasticsearch-connector-doesnt-work-java-lang-noclassdeffounderror-com-googl](https://stackoverflow.com/questions/62015340/elasticsearch-connector-doesnt-work-java-lang-noclassdeffounderror-com-googl) , I've already added guava and also other missing jars to the worker at the same folder with existing jars. It's sad that it didn't work.

[tirelibirefe]

Thank you all guys!
Adding missing guava-20.0.jar to /opt/kafka/libs/ at worker node container fixed the problem. My environment is;
Strimzi 0.18
Kafka 2.5.0

COPY guava-20.0.jar /opt/kafka/libs/