Cp-docker-images: Kafka Docker - How to increase memory settings

[Tin-Nguyen]

I'm using Confluent Kafka image to run Kafka via docker container on my machine. the kafka broker is often hang and unable to be connected when I did run some tests. It doesn't back to work normally even I killed the container and re-inited it or restarted Docker, etc... I had to use Reset to Factory default , then init the docker container again to make it back to work normally.

It takes my time. I dig into investigating the problem and figured out it may be caused by the memory configuration in the Java process running Kafka

java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp /usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /etc/kafka/kafka.properties

It seems 1Gb memory isn't enough and made the kafka in the container hang. I'm looking for a way to increase that configuration to be higher. Anyone knows which ENV var or configuration helps to do that?

[OneCricketeer]

KAFKA_HEAP_OPTS - https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L231-L239

[ethicalmohit]

KAFKA_HEAP_OPTS - https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L231-L239

how to pass this option to the docker image of the confluent Kafka?

[OneCricketeer]

That's the environment variable name, so pass it like all the others

[ethicalmohit]

That's the environment variable name, so pass it like all the others

KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"

I passed it and I am getting below error.

===> Launching kafka ...
hostname: Name or service not known
Error: Could not find or load main class "-Xmx512M

[OneCricketeer]

Should work, there's an example here that also logs JVM events out to a directory

https://docs.confluent.io/current/installation/docker/development.html#write-garbage-collection-logs-to-an-external-volume

[ethicalmohit]

Should work, there's an example here that also logs JVM events out to a directory

https://docs.confluent.io/current/installation/docker/development.html#write-garbage-collection-logs-to-an-external-volume

@cricket007 This works. KAFKA_HEAP_OPTS=-Xmx512M -Xms512M without the double quote. Thanks.