Onpremise: Installation of on premise erroring out with Connection to Kafka failed

[khangtoh]

Cloning master version to ubuntu 18.04 and running the install script

2020-05-05 09:29:21,527 Connection to Kafka failed (attempt 59)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
Traceback (

ubuntu@ubuntu-s-6vcpu-16gb-sgp1-01:~/onpremise$ docker -v Docker version 19.03.8, build afacb8b7f0

ubuntu@ubuntu-s-6vcpu-16gb-sgp1-01:~/onpremise$ docker-compose -v docker-compose version 1.25.5, build unknown

[cristicotet]

Same error when updating Sentry from older 10.1.0-dev0 to latest but in case of update it seems to start correctly if i run docker-compose up -d
I'm not sure if the update is complete :)

[ChibangLW]

Looks like reverting #465 will "fix" this. With kafka and zookeeper on 5.5 both containers kept failing for me.

[maximal]

Had a similar issue in January: see #331

Removing some containers and Kafka volume helped me back then. But it does not resolve the issue this time.

[BYK]

@maximal maybe you need to remove the zookeeper volumes and containers this time too?

Apologies for the issues, I was expecting this upgrade to be painless but seems like that's not really the case.

[maximal]

@maximal maybe you need to remove the zookeeper volumes and containers this time too?

Removing zookeeper volumes helped even with the newer kafka and zookeeper versions in docker-compose.yml, thanks!

docker-compose down
docker volume rm sentry-kafka sentry-zookeeper
docker volume rm sentry_onpremise_sentry-kafka-log sentry_onpremise_sentry-zookeeper-log
./install

Apologies for the issues, I was expecting this upgrade to be painless but seems like that's not really the case.

No worries! We’re using unstable branch here, testing it so I suppose we all realize the consequences :-)

[BYK]

No worries! We’re using unstable branch here, testing it so I suppose we all realize the consequences :-)

Eh, if only we were providing a "stable" version, you'd be right. This is around the corner though so stay tuned and thanks again!

[cristicotet]

Solution proposed by @maximal works in my case (update from older 10.1.0-dev0)

[BYK]

If anyone has still access to ZooKeeper logs, can you share them here? You can run docker-compose logs zookeeper to get them.

[BuSHari]

I'm getting this error too.

Here is the zookeeper logs:

Attaching to sentry_onpremise_zookeeper_1
zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=a82a7743d431
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...
zookeeper_1                | ===> Running preflight checks ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                | ===> Launching ...
zookeeper_1                | ===> Launching zookeeper ...
zookeeper_1                | [2020-05-07 19:40:27,042] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                | [2020-05-07 19:40:28,135] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-07 19:40:28,140] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-07 19:40:28,833] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
zookeeper_1                | java.io.IOException: No snapshot found, but there are log entries. Something is broken!
zookeeper_1                |    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
zookeeper_1                |    at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
zookeeper_1                |    at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
zookeeper_1                |    at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
zookeeper_1                |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
zookeeper_1                |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=a82a7743d431
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...

And kafka:

Attaching to sentry_onpremise_kafka_1
kafka_1                    | ===> ENV Variables ...
kafka_1                    | ALLOW_UNSIGNED=false
kafka_1                    | COMPONENT=kafka
kafka_1                    | CONFLUENT_DEB_VERSION=1
kafka_1                    | CONFLUENT_PLATFORM_LABEL=
kafka_1                    | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1                    | CONFLUENT_VERSION=5.5.0
kafka_1                    | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1                    | HOME=/root
kafka_1                    | HOSTNAME=02cd0ee0887c
kafka_1                    | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1                    | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1                    | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1                    | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1                    | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1                    | KAFKA_VERSION=
kafka_1                    | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1                    | LANG=C.UTF-8
kafka_1                    | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1                    | PWD=/
kafka_1                    | PYTHON_PIP_VERSION=8.1.2
kafka_1                    | PYTHON_VERSION=2.7.9-1
kafka_1                    | SCALA_VERSION=2.12
kafka_1                    | SHLVL=1
kafka_1                    | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1                    | _=/usr/bin/env
kafka_1                    | ===> User
kafka_1                    | uid=0(root) gid=0(root) groups=0(root)
kafka_1                    | ===> Configuring ...
kafka_1                    | ===> Running preflight checks ...
kafka_1                    | ===> Check if /var/lib/kafka/data is writable ...
kafka_1                    | ===> Check if Zookeeper is healthy ...
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=02cd0ee0887c
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-166-generic
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=955MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=14315MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=966MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1                    | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.192.10:42838, server: zookeeper/192.168.192.5:2181
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper/192.168.192.5:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.io.IOException: Connection reset by peer
kafka_1                    |    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
kafka_1                    |    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
kafka_1                    |    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
kafka_1                    |    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
kafka_1                    |    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
kafka_1                    |    at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
kafka_1                    |    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main] ERROR io.confluent.admin.utils.ClusterStatus - Timed out waiting for connection to Zookeeper server [zookeeper:2181].
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
kafka_1                    | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
kafka_1                    | ===> ENV Variables ...
kafka_1                    | ALLOW_UNSIGNED=false
kafka_1                    | COMPONENT=kafka
kafka_1                    | CONFLUENT_DEB_VERSION=1
kafka_1                    | CONFLUENT_PLATFORM_LABEL=
kafka_1                    | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1                    | CONFLUENT_VERSION=5.5.0
kafka_1                    | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1                    | HOME=/root
kafka_1                    | HOSTNAME=02cd0ee0887c
kafka_1                    | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1                    | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1                    | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1                    | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1                    | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1                    | KAFKA_VERSION=
kafka_1                    | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1                    | LANG=C.UTF-8
kafka_1                    | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1                    | PWD=/
kafka_1                    | PYTHON_PIP_VERSION=8.1.2
kafka_1                    | PYTHON_VERSION=2.7.9-1
kafka_1                    | SCALA_VERSION=2.12
kafka_1                    | SHLVL=1
kafka_1                    | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1                    | _=/usr/bin/env
kafka_1                    | ===> User
kafka_1                    | uid=0(root) gid=0(root) groups=0(root)
kafka_1                    | ===> Configuring ...
kafka_1                    | ===> Running preflight checks ...
kafka_1                    | ===> Check if /var/lib/kafka/data is writable ...
kafka_1                    | ===> Check if Zookeeper is healthy ...
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=02cd0ee0887c
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-166-generic
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=955MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=14315MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=966MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1                    | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1                    | [main-SendThread()] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper: Temporary failure in name resolution
kafka_1                    |    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
kafka_1                    |    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
kafka_1                    |    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper: Temporary failure in name resolution
kafka_1                    |    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
kafka_1                    |    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
kafka_1                    |    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    |    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    |    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    |    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    |    at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    |    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)

[pnghai]

Here's yours:

zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=0e076b9fc016
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...
zookeeper_1                | ===> Running preflight checks ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                | ===> Launching ...
zookeeper_1                | ===> Launching zookeeper ...
zookeeper_1                | [2020-05-14 10:45:54,588] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                | [2020-05-14 10:45:54,790] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:45:54,790] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:45:55,004] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
zookeeper_1                | java.io.IOException: No snapshot found, but there are log entries. Something is broken!
zookeeper_1                |    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
zookeeper_1                |    at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
zookeeper_1                |    at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
zookeeper_1                |    at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
zookeeper_1                |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
zookeeper_1                |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=0e076b9fc016
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...
zookeeper_1                | ===> Running preflight checks ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                | ===> Launching ...
zookeeper_1                | ===> Launching zookeeper ...
zookeeper_1                | [2020-05-14 10:46:13,605] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                | [2020-05-14 10:46:13,823] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:46:13,823] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:46:14,001] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
zookeeper_1                | java.io.IOException: No snapshot found, but there are log entries. Something is broken!
zookeeper_1                |    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
zookeeper_1                |    at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
zookeeper_1                |    at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
zookeeper_1                |    at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
zookeeper_1                |    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
zookeeper_1                |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
zookeeper_1                |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

If anyone has still access to ZooKeeper logs, can you share them here? You can run docker-compose logs zookeeper to get them.

[jellevanhees]

https://issues.apache.org/jira/browse/ZOOKEEPER-3056

this might fix the issue in zookeeper. but i don't know the correct way to pass this setting to the container in this setup.

[BYK]

I think there's a fix upcoming. In the meantime, can someone try adding ZOOKEEPER_SNAPSHOT_TRUST_EMPTY: 'true' under zookeeper's environment section and see if it helps?

[xf-]

@BYK stops the loop (error appears once) and install continues

[BYK]

@xf- so that means adding this value solves the issue?

[xf-]

@BYK like i wrote. I get the same error but only once and the update continues.
= Looks like it works, but still the same error in console (only once) and maybe somebody will reopen/create a new issue.

[caugner]

@maximal maybe you need to remove the zookeeper volumes and containers this time too?

Removing zookeeper volumes helped even with the newer kafka and zookeeper versions in docker-compose.yml, thanks!

docker-compose down
docker volume rm sentry-kafka sentry-zookeeper
docker volume rm sentry_onpremise_sentry-kafka-log sentry_onpremise_sentry-zookeeper-log
./install

This solution did not work for me when upgrading from 10.1.0.dev0 (e54cbeb).

I think there's a fix upcoming. In the meantime, can someone try adding ZOOKEEPER_SNAPSHOT_TRUST_EMPTY: 'true' under zookeeper's environment section and see if it helps?

This workaround by @BYK did not work either.

PS: According to docker stats, there are no issues with CPU or Memory consumption. Both kafka and zookeeper seem to be running despite the timeout.

Update: At least I was able to revert to my previous version (with latest git checkout): SENTRY_IMAGE=getsentry/sentry:e54cbeb ./install.sh

[BYK]

@caugner that's quite peculiar that going back to an earlier Sentry image fixing this. I bet it's not Sentry getting reverted itself but instead Snuba and other services getting updated or restarted. I'd recommend running ./install.sh without the SENTRY_IMAGE override and see what happens.

[BYK]

Hey all, can you check whether #511 fixes the issue for you? I tried it locally with a broken ZK and it did for me but trying to be sure that it works for everyone.

/cc @caugner @xf-

[jansorg]

@BYK I've pulled 83160e8bdbebfea2c0f02849b545b9cf1b30c324 and I'm still seeing the errors of this issue when I run ./setup.sh.

^T2020-05-25 11:20:03,768 Connection to Kafka failed (attempt 59)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 11, in <module>
    load_entry_point('snuba', 'console_scripts', 'snuba')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaExcepti

[BYK]

@jansorg it is normal to see it a few times. If the installation finishes successfully, just ignore it. If your install doesn't succeed, please file a new issue and share your Kafka and Zookeeper logs.

[caugner]

I was able to update from e54cbeb to df57809 now, but it could also be that my connectivity issues (https://github.com/getsentry/onpremise/issues/505#issuecomment-635604243) played a role here.