Cp-docker-images: Docker Kafka-connect does not start. Having issues with onfiguration key.converter

Created on 5 Feb 2018 · 5Comments · Source: confluentinc/cp-docker-images

Hi,

When I try to start the confluentinc/cp-kafka-connect:latest image, but without success. I got the following sack trace:

===> Launching kafka-connect ... 
dev-kafka-connect        | Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value "io.confluent.connect.avro.AvroConverter" for configuration key.converter: Class "io.confluent.connect.avro.AvroConverter" could not be found.
dev-kafka-connect        |      at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:715)
dev-kafka-connect        |      at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:460)
dev-kafka-connect        |      at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453)
dev-kafka-connect        |      at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
dev-kafka-connect        |      at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
dev-kafka-connect        |      at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:218)
dev-kafka-connect        |      at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:272)
dev-kafka-connect        |      at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:72)

Here is an extract of my docker-compose file:

    kafka-connect:
        image: confluentinc/cp-kafka-connect:latest
        restart: always
        container_name: dev-kafka-connect
        depends_on:
            - zookeeper
            - kafka-broker
            - kafka-schema-registry
        env_file:
            - ${ENV_BASE_DIR}/confluent/global.env
            - ${ENV_BASE_DIR}/confluent/kafka-connect.env
        network_mode: host
        hostname: localhost
        extra_hosts:
            - "moby:127.0.0.1"
        volumes:
            - /tmp/quickstart/file:/tmp/quickstart
            - /tmp/quickstart/jars:/etc/kafka-connect/jars

Here is my variables (cat of kafka-connect.env):

CONTROL_CENTER_ZOOKEEPER_CONNECT=127.0.0.1:2181
CONTROL_CENTER_BOOTSTRAP_SERVERS=127.0.0.1:9092
CONTROL_CENTER_REPLICATION_FACTOR=1
CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS=1
CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS=1
CONTROL_CENTER_STREAMS_NUM_STREAM_THREADS=1
CONTROL_CENTER_CONNECT_CLUSTER=http://127.0.0.1:8082
CONTROL_CENTER_LOG4J_LOGGERS="kafka.controller=ERROR"
CONTROL_CENTER_LOG4J_ROOT_LOGLEVEL=ERROR
CONTROL_CENTER_TOOLS_LOG4J_LOGLEVEL=ERROR

Bellow, some additional information's:

uname -a

Linux HOSTNAME 4.14.15-1-ARCH #1 SMP PREEMPT Tue Jan 23 21:49:25 UTC 2018 x86_64 GNU/Linux

docker --version

Client:
 Version:       18.01.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    03596f51b1
 Built: Sun Jan 14 23:10:39 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.01.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   03596f51b1
  Built:        Sun Jan 14 23:11:14 2018
  OS/Arch:      linux/amd64
  Experimental: false

docker-compose --version

docker-compose version 1.18.0, build unknown

Thanks for your help.
JR

Source

gmocquet

Most helpful comment

@toniiiik ,@edwardpang-computime : try removing the double quotes in the value.

For example, instead of

    - CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter"

use

    - CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter

etc ...

ybyzek on 6 Jan 2019

👍10

All 5 comments

I don't think your connect.env file is correct... The CONTROL_CENTER variables won't be loaded. Plus, I think you are missing CONNECT_PLUGIN_PATH, or have overwritten it or CLASSPATH

OneCricketeer on 31 Aug 2018

I have the same problem as @teaguecole described. When I run kafka connect from docker-compose I got the error:

org.apache.kafka.common.config.ConfigException: Invalid value "io.confluent.connect.avro.AvroConverter" for configuration key.converter: Class "io.confluent.connect.avro.AvroConverter" could not be found.
connect_1  |    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:724)
connect_1  |    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:469)
connect_1  |    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:462)
connect_1  |    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
connect_1  |    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
connect_1  |    at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:345)
connect_1  |    at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:272)
connect_1  |    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:76)
demo_connect_1 exited with code 2

here is my docker-compose:

version: '3.1'
services:
  zoo:
    image: confluentinc/cp-zookeeper:5.0.1
    environment:
    - ZOOKEEPER_CLIENT_PORT=2181 
    - ZOOKEEPER_TICK_TIME=2000
    ports:
    - 2181:2181
  kafka:
    image: confluentinc/cp-kafka:5.0.1
    ports:
    - 9092:9092
    environment:
    - KAFKA_ZOOKEEPER_CONNECT=zoo:2181 
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 
    - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 
    links:
    - zoo
  schema:
    image: confluentinc/cp-schema-registry:5.0.1
    environment: 
    - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zoo:2181
    - SCHEMA_REGISTRY_HOST_NAME=schema
    - SCHEMA_REGISTRY_LISTENERS=http://schema:8081
    links:
    - zoo
    ports:
    - 8081:8081
  connect:
    image: confluentinc/cp-kafka-connect:latest
    environment:
    - CONNECT_BOOTSTRAP_SERVERS=kafka:9092
    - CONNECT_REST_PORT=8083
    - CONNECT_GROUP_ID="quickstart-avro"
    - CONNECT_CONFIG_STORAGE_TOPIC="quickstart-avro-config"
    - CONNECT_OFFSET_STORAGE_TOPIC="quickstart-avro-offsets"
    - CONNECT_STATUS_STORAGE_TOPIC="quickstart-avro-status"
    - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
    - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
    - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
    - CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter"
    - CONNECT_VALUE_CONVERTER="io.confluent.connect.avro.AvroConverter"
    - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL="http://schema:8081"
    - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL="http://schema:8081"
    - CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter"
    - CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter"
    - CONNECT_REST_ADVERTISED_HOST_NAME="localhost"
    - CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG
    - CONNECT_PLUGIN_PATH=/usr/share/java/
    links:
    - schema
    - kafka
    ports:
    - 8083:8083

when I run connect outside of compose with the following command:

docker run -d \
  --name=kafka-connect-avro \
  --net=demo_default \
  -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
  -e CONNECT_REST_PORT=28083 \
  -e CONNECT_GROUP_ID="quickstart-avro" \
  -e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-avro-config" \
  -e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-avro-offsets" \
  -e CONNECT_STATUS_STORAGE_TOPIC="quickstart-avro-status" \
  -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter" \
  -e CONNECT_VALUE_CONVERTER="io.confluent.connect.avro.AvroConverter" \
  -e CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL="http://zoo:8081" \
  -e CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL="http://zoo:8081" \
  -e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_REST_ADVERTISED_HOST_NAME="localhost" \
  -e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
  -e CONNECT_PLUGIN_PATH=/usr/share/java \
  confluentinc/cp-kafka-connect:latest

...it works.

toniiiik on 15 Nov 2018

same to @toniiiik & @teaguecole

docker-compose fail, but same config works using command line docker run

error says:

[2019-01-06 08:38:31,246] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
org.apache.kafka.common.config.ConfigException: Invalid value "org.apache.kafka.connect.json.JsonConverter" for configuration key.converter: Class "org.apache.kafka.connect.json.JsonConverter" could not be found.
        at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:718)
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:471)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:464)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
        at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:360)
        at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:272)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:76)

edwardpang-computime on 6 Jan 2019

@toniiiik ,@edwardpang-computime : try removing the double quotes in the value.

For example, instead of

    - CONNECT_KEY_CONVERTER="io.confluent.connect.avro.AvroConverter"

use

    - CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter

etc ...

ybyzek on 6 Jan 2019

👍10

@ybyzek THX, it works

edwardpang-computime on 6 Jan 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Please add KAFKAREST_HEAP_OPTS to documentation

jmarcos-cano · 4Comments

/var/lib/kafka permissions

fritchie · 4Comments

Unable to override default dataDir and LogDir for zookeeper

gunjanarora · 4Comments

Set Support Metrics to false

Fobhep · 6Comments

Rolling upgrade cp-kafka 4.0.0-3 to 4.1.1-1

mariuscoto · 4Comments