Cmak: Exceptions when accessing a broker running with docker-compose

[1123]

Hi all,

first of all, thanks for the great tool. It makes using kafka very transparent.

I am using this project for running a kafka cluster with docker-compose on a Mac: https://github.com/simplesteph/kafka-stack-docker-compose .

While within the docker network, the kafka brokers are accessible via kafka1:19092, they are only accessible via localhost:19092 on my local work station, where I run kafka-manager.

This is probably why I get the exceptions below in the logs of kafka-manager. I can still see all topics, their partitions, and repartition topics via kafka-manager, but probably I cannot use all functionality it provides. Is there any way I can tell kafka-manager to use the ports on localhost for accessing the brokers?

Thanks!

[warn] o.a.k.c.ClientUtils - Removing server kafka1:19092 from bootstrap.servers as DNS resolution failed for kafka1 [error] k.m.a.c.KafkaStateActor - consumerProperties:{key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, bootstrap.servers=kafka1:19092, group.id=kafka.manager.actor.cluster.KafkaStateActor.KafkaTopicOffsetGetter, value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, security.protocol=PLAINTEXT} WARNING arguments left: 1 [error] k.m.a.c.KafkaStateActor - KafkaTopicOffsetGetter exception org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:793) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:624) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$10$$anonfun$apply$mcV$sp$14.apply(KafkaStateActor.scala:1490) ~[kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$10$$anonfun$apply$mcV$sp$14.apply(KafkaStateActor.scala:1468) ~[kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na] at scala.collection.Iterator$class.foreach(Iterator.scala:891) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174) ~[org.scala-lang.scala-library-2.11.12.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$10.apply$mcV$sp(KafkaStateActor.scala:1468) ~[kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na] at scala.util.control.Breaks.breakable(Breaks.scala:38) ~[org.scala-lang.scala-library-2.11.12.jar:na] Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:66) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:706) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na]

[jsargiot]

I don't think there is a way to use a specific ip for the brokers, because (IIRC) kafka-manager gets the broker info from Zookeeper.

Try changing the order of this line: https://github.com/simplesteph/kafka-stack-docker-compose/blob/master/full-stack.yml#L25

Change this:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092

so it looks like this:

KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,LISTENER_DOCKER_INTERNAL://kafka1:19092

I think the other option is to add 127.0.0.1 kafka1 to your /etc/hosts file...

[1123]

Great. This works. I no longer see the exceptions.