Cp-docker-images: `cub kafka-ready` fails with non-plaintext and config.provider

[strigona-worksight]

When a config.providers is specified and connecting with a security.protocol other than PLAINTEXT then cub kafka-ready fails.

When using non-PLAINTEXT, the generated kafka-connect.properties which contains the relevant config.providers properties. The following error is thrown:

[main] ERROR org.apache.kafka.clients.admin.AdminClientConfig - ClassNotFoundException exception occurred: org.ggt.kafka.config.provider.KafkaEnvConfigProvider
[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.
org.apache.kafka.common.config.ConfigException: Invalid value java.lang.ClassNotFoundException: org.ggt.kafka.config.provider.KafkaEnvConfigProvider for configuration Invalid config:org.ggt.kafka.config.provider.KafkaEnvConfigProvider ClassNotFoundException exception occurred
    at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:537)
    at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:477)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
    at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:208)
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:69)
    at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
    at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:138)

Relevant code in the ensure file:
https://github.com/confluentinc/cp-docker-images/blob/5.4-preview/debian/kafka-connect-base/include/etc/confluent/docker/ensure#L24-L28

The config provider used works just fine with the strimzi images and works with PLAINTEXT.

[mubeta06]

Is there any know work around here? We have just hit this and strimzi is not an option for us.

[mubeta06]

Update: The issue appears to be associated with cub's inability to load up the config providers jar file. The work around for us was to append to CUB_CLASSPATH the path to the org.ggt.kafka.config.provider.KafkaEnvConfigProvider jar file. e.g. CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar:/usr/local/share/java/kafka-connect/plugins/kafka-env-config-provider/kafka-env-config-provider-0.0.1.jar

[strigona-worksight]

Thanks I'll give that a shot! I had poked around a bit, but settled on overriding the entrypoint script to omit running the ensure script, bypassing cub kafka-ready entirely.

Did you set the CUB_CLASSPATH at the docker/kubernetes level?

[mubeta06]

I am not familiar enough with the project nor the composition of the docker container to manipulate the start up scripts but setting the CUB_CLASSPATH environment variable upon firing up docker i.e. docker run .... -e CUB_CLASSPATH=... should resolve the stack trace from above.

[jennybrown8]

I seem to also have this issue with a different config provider (AWSSecretProvider from lenses.io). In my case, the cub classpath does not impact the results, but changing CONNECT_SECURITY_PROTOCOL from SASL_SSL to PLAINTEXT does resolve it (even without touching any classpaths). I shouldn't keep it on plaintext for production use, so I still hope to see a deeper fix for this. I hope this helps to further isolate the problem for fixes.

[jelledv]

Since image version 6.0.0, the CUB_CLASSPATH environment variable has been updated to "/usr/share/java/cp-base-new/*". You can add your config provider jars into this directory and it will be picked up at startup:
ADD kafka-env-config-provider-0.0.1.jar /usr/share/java/cp-base-new/. This way you don't have to update the CUB_CLASSPATH environment variable in your worker configuration.