Pulsar: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger

[dsambandam]

Expected behavior

Successful Bookie sanity after starting bookies in IPV6

Actual behavior

Bookie sanity fails - Unable to connect to Bookie after writing to Ledger

Steps to reproduce

1. Update the configuration file for Bookies with Zk Server entries
2. Start the bookie ( No major errors reported)
3. Run Bookie sanity
4. Issue : Connection lost to bookie after writing to ledgers

12:47:57.979 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: [id: 0x426035ec, L:/2001:ssss:a06:xxxx:f1:fef:0:cddd%0:4760 - R:2001:ssss:a06:xxxx:f1:fef:0:cddd/2001:ssss:a06:xxxx:f1:fef:0:cddd:3181]
12:47:57.989 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 2001:ssss:a06:xxxx:f1:fef:0:cddd:3181
12:47:58.009 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x426035ec, L:/2001:ssss:a06:xxxx:f1:fef:0:cddd%0:4760 - R:2001:ssss:a06:xxxx:f1:fef:0:cddd/2001:ssss:a06:xxxx:f1:fef:0:cddd:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
12:47:58.078 [main] INFO org.apache.bookkeeper.bookie.BookieShell - Written 10 entries in ledger 7
12:47:58.099 [bookkeeper-io-3-2] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0xf86d5071]/2001:ssss, current state CONNECTING :
io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: /0.0.7.209:4888
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:104) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.unix.Socket.connect(Socket.java:257) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:732) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:717) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:559) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1346) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1004) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:259) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.NoRouteToHostException
... 30 more
12:47:58.105 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.ReadLastConfirmedOp - While readLastConfirmed ledger: 7 did not hear success responses from all quorums
12:47:58.112 [main] WARN org.apache.bookkeeper.bookie.BookieShell - Error in bookie sanity test
org.apache.bookkeeper.client.BKException$BKLedgerRecoveryException: Error while recovering ledger
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncOpenCallback.openComplete(SyncCallbackUtils.java:143) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:225) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.LedgerOpenOp$1.safeOperationComplete(LedgerOpenOp.java:194) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.LedgerOpenOp$1.safeOperationComplete(LedgerOpenOp.java:1) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.util.OrderedGenericCallback.operationComplete(OrderedGenericCallback.java:55) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.proto.BookkeeperInternalCallbacks$TimedGenericCallback.operationComplete(BookkeeperInternalCallbacks.java:131) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.LedgerRecoveryOp.submitCallback(LedgerRecoveryOp.java:145) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.LedgerRecoveryOp.access$1(LedgerRecoveryOp.java:137) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.LedgerRecoveryOp$1.readLastConfirmedDataComplete(LedgerRecoveryOp.java:124) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:134) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.proto.BookieClient$2.safeRun(BookieClient.java:272) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
12:47:58.124 [main] INFO org.apache.bookkeeper.bookie.BookieShell - Deleted ledger 7

System configuration

Pulsar version: 2.0

[sijie]

@dsambandam I think there is a problem on bookkeeper handling ipv6 ip address. I will file an issue on bookkeeper to address that. for getting around on this issue, I would suggest setting useHostNameAsBookieID to true.

https://github.com/apache/incubator-pulsar/blob/master/conf/bookkeeper.conf#L97

[sijie]

@dsambandam same thing might happen to brokers as well. you might consider using advertisedAddress for broker to advertise hostname. https://github.com/apache/incubator-pulsar/blob/master/conf/broker.conf#L44

[dsambandam]

@sijie useHostNameAsBookieID to true (worked fine) but having issues with brokers

14:34:08.223 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x9faf4b3a, L:/2001:dddd:a06:ssss:f1:fef:0:dgdg%0:5317 - R:host1/2001:ssss:a06:xxxx:f1:fef:0:ffff:6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
14:34:08.224 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
14:34:08.225 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
14:34:08.226 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available

[sijie]

@dsambandam "Not enough non-faulty bookie" means there are not enough bookies.

a couple of questions:

• how many bookie are you running?
• can you run bin/bookkeeper listbookies -rw to show me the list of bookies advertised in the cluster?

[dsambandam]

I am running 3 bookies ...how should I run the below command
[root@carocnxodapa1v apache-pulsar-2.0.0-rc1-incubating]# ./bin/bookkeeper listbookies -rw
JMX enabled by default
JAVA_HOME not set, using java from PATH. (/usr/bin/java)
Error: Could not find or load main class listbookies

[dsambandam]

Tried ./bin/bookkeeper shell listbookies -rw and it shows up lot of logs ..do you need them ?

[dsambandam]

All the bookies are up . Below is the broker conf update

Zookeeper quorum connection string

zookeeperServers=host8:2181,host9:2181,host10:2181

Configuration Store connection string

configurationStoreServers=host8v:2181,host9:2181,host10:2181
advertisedAddress=host1 (Broker hostname)

[sijie]

Tried ./bin/bookkeeper shell listbookies -rw and it shows up lot of logs ..do you need them ?

not all of them, can you paste the lines that have `3181?

[dsambandam]

Unfortunately nothing on port 3181..Is my configuration above for starting broker correct? I have 3 zookeepers and 3 bookies running ..For some strange reason brokers aren't able to see them

[dsambandam]

Also when useHostNameAsBookieID is set to true and bookie restarted , process fails with the below error.

https://github.com/apache/incubator-pulsar/blob/master/conf/bookkeeper.conf#L97

The only way I could get the sanity passed is when the above setting is false , bookie started and sanity run with bookie conf file updated

18:22:49.968 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
18:22:50.095 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
18:22:50.106 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
bookieHost: "host5:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"
] is not matching with [4
bookieHost: "2001:ffff:a06:ssss:f1:fef:0:ssss:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"

[sijie]

okay. so the problem is the bookies didn't start correctly. there is a mechanism in bookkeeper to prevent bookies startup with malformed state which is called cookie. this ensures disk layout and bookie identifier unchanged between different bookie restart cycles.

in this case, since we changed the bookie to use hostname, it changes the bookie identifier. bookkeeper prevents its startup. what I would suggest to do:

on each bookie node, run "bin/bookkeeper shell bookieformat". this would reinitialize your bookie nodes.

[dsambandam]

Should I restart bookie after bookieformat?

[sijie]

@dsambandam yes.

[dsambandam]

19:00:27.594 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
19:00:27.702 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
Fails with the same error

19:00:27.722 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
bookieHost: "carocnxodapa4v:3181"

[dsambandam]

Fails with the same error after "bookieformat"

[sijie]

@dsambandam sorry you might need to remove cookie as well.

try bin/bookkeeper shell bookieformat -deleteCookie

[dsambandam]

Bookie doesn't start ..

19:22:42.675 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
19:22:42.702 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
bookieHost: "carocnxodapa4v:3181"

[sijie]

@dsambandam do you have more logging around this? also can you share your bookkeeper.conf?

[dsambandam]

Attached logs for your reference

bookkeeper.conf.txt
pulsar-bookie-host4.log

[sijie]

@dsambandam I see still the same error. actually can you do following:

1) on the bookie node, stop the bookie process
2) run bin/bookkeeper shell bookieformat -deleteCookie -f
3) check the file by cat data/bookkeeper/journal/current/VERSION and cat data/bookkeeper/storage/current/VERSION. I would like to see what is recorded in the VERSION file.
4) then restart the bookie again.

[dsambandam]

When bookie format is attempted, I can't start back the bookie process again since it fails with the same error report above and data directories are empty.

Working Bookie Node version with useHostNameAsBookieID set to False.

bookieHost: "2001:dddd:a06:dddd:f1:fef:0:ffff:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"

And by default only below directories are created and no storage nodes

[root@host5 bookkeeper]# ls -lrt
total 8
drwx------ 3 root root 4096 Jun 12 11:50 journal
drwx------ 3 root root 4096 Jun 12 11:50 ledgers

[dsambandam]

Ledger version:

[root@host5 current]# cat VERSION
4
bookieHost: "2001:dddd:a06:dddd:f1:fef:0:ffff::3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"

[sijie]

interesting, so it seems that VERSION file is still using ipv6 ip as the bookie identifier. what is the output when you using bin/bookkeeper shell bookieformat -deleteCookie -f?

[dsambandam]

bin/bookkeeper shell bookieformat -deleteCookie -f
JMX enabled by default
JAVA_HOME not set, using java from PATH. (/usr/bin/java)
Are you sure to format Bookie data..? (Y or N) Y
08:03:42.304 [main] INFO org.apache.bookkeeper.bookie.Bookie - Bookie format completed successfully
08:03:42.328 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
08:03:42.328 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
08:03:42.328 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
08:03:42.347 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+hierarchical://host8:2181;host9:2181;host10:2181/ledgers : zkServers = host8:2181,host9:2181,host10:2181, ledgersRootPath = /ledgers.
08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=Bookiehost
08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_131
08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/java/jdk1.8.0_131/jre
08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/conf:::/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.beust-jcommander-1.48.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.carrotsearch-hppc-0.7.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.ea.agentloader-ea-agent-loader-1.0.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.core-jackson-annotations-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.core-jackson-core-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.core-jackson-databind-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.datatype-jackson-datatype-joda-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.github.ben-manes.caffeine-caffeine-2.3.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.api.grpc-proto-google-common-protos-0.1.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.auth-google-auth-library-credentials-0.4.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.code.findbugs-jsr305-3.0.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.code.gson-gson-2.8.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.errorprone-error_prone_annotations-2.0.19.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.guava-guava-20.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.instrumentation-instrumentation-api-0.4.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.protobuf-protobuf-java-2.4.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.protobuf-protobuf-java-util-3.3.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-beanutils-commons-beanutils-1.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-beanutils-commons-beanutils-core-1.8.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-cli-commons-cli-1.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-codec-commons-codec-1.10.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-collections-commons-collections-3.2.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-configuration-commons-configuration-1.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-digester-commons-digester-1.8.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-io-commons-io-2.5.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-lang-commons-lang-2.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-logging-commons-logging-1.1.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.squareup.okio-okio-1.6.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.wordnik-swagger-annotations-1.5.3-M1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.yahoo.datasketches-memory-0.8.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-3.10.1.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-all-4.1.21.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-codec-http2-4.1.12.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-codec-socks-4.1.12.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-handler-proxy-4.1.12.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-tcnative-boringssl-static-2.0.7.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient_common-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient_hotspot-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient_servlet-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.swagger-swagger-annotations-1.5.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.swagger-swagger-core-1.5.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.swagger-swagger-models-1.5.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.annotation-javax.annotation-api-1.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.validation-validation-api-1.1.0.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.websocket-javax.websocket-api-1.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/jline-jline-0.9.94.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/joda-time-joda-time-2.7.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/log4j-log4j-1.2.17.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/net.java.dev.jna-jna-4.2.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/net.jpountz.lz4-lz4-1.3.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper-circe-checksum-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper.http-http-server-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.commons-commons-collections4-4.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.commons-commons-lang3-3.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.distributedlog-distributedlog-core-shaded-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-managed-ledger-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-broker-common-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-client-admin-original-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-client-original-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-client-tools-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-common-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-connect-core-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-discovery-service-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-api-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-api-examples-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-proto-shaded-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-runtime-all-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-runtime-shaded-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-worker-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-proxy-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-testclient-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-websocket-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-zookeeper-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.aspectj-aspectjrt-1.8.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.aspectj-aspectjweaver-1.8.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.asynchttpclient-async-http-client-2.1.0-alpha26.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.asynchttpclient-async-http-client-netty-utils-2.1.0-alpha26.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.bouncycastle-bcpkix-jdk15on-1.55.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.bouncycastle-bcprov-jdk15on-1.55.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-client-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-continuation-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-http-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-proxy-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-security-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-servlets-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-api-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-client-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-common-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-server-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-servlet-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-hk2-api-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-hk2-locator-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-hk2-utils-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.bundles.repackaged-jersey-guava-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.core-jersey-client-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.core-jersey-common-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.core-jersey-server-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.media-jersey-media-multipart-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.javassist-javassist-3.21.0-GA.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.projectlombok-lombok-1.16.20.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.reactivestreams-reactive-streams-1.0.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.rocksdb-rocksdbjni-5.8.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.slf4j-slf4j-api-1.7.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.yaml-snakeyaml-1.15.jar:
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-693.21.1.el7.x86_64
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating
08:03:42.360 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host8:2181,host9:2181,host10:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@4f5991f6
08:03:42.393 [main-SendThread(host8:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server host8/2001:dddd:a06:ssss:f1:fef:0:xxxx:2181. Will not attempt to authenticate using SASL (unknown error)
08:03:42.400 [main-SendThread(host8:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to host8/2001:dddd:a06:ssss:f1:fef:0:xxxx:2181, initiating session
08:03:42.409 [main-SendThread(host8:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server host8/2001:dddd:a06:ssss:f1:fef:0:xxxx:2181, sessionid = 0x163f5d957ec001d, negotiated timeout = 30000
08:03:42.423 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
08:03:42.447 [main] WARN org.apache.bookkeeper.bookie.BookieShell - No cookie to remove :
org.apache.bookkeeper.bookie.BookieException$CookieNotFoundException: Bookiehost:3181
at org.apache.bookkeeper.discover.ZKRegistrationManager.readCookie(ZKRegistrationManager.java:339) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.Cookie.readFromRegistrationManager(Cookie.java:332) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.Cookie.readFromRegistrationManager(Cookie.java:316) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.BookieShell$BookieFormatCmd.lambda$0(BookieShell.java:423) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.meta.MetadataDrivers.lambda$1(MetadataDrivers.java:363) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:338) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:363) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.BookieShell$BookieFormatCmd.runCmd(BookieShell.java:421) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:232) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2816) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2907) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
08:03:42.453 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x163f5d957ec001d closed
08:03:42.453 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x163f5d957ec001d

[dsambandam]

Version file still uses IPV6 since bookie never started successfully after modifying conf file

[sijie]

No cookie to remove :
org.apache.bookkeeper.bookie.BookieException$CookieNotFoundException: Bookiehost:3181
at

so the bookieformat didn't succeed. I think the problem might be failing on handling the ipv6 ip address.

so I would suggest doing following:

1) stop all your bookies.
2) on one bookie node, run 'bin/bookkeeper shell metaformat -f', this will clean up the bad state in metadata. run this only once.
3) on each bookie, update bookkeeper.conf to set advertisedAddress to the hostname and then run bin/bookkeeper shell bookieformat -f, then start the bookie.

[dsambandam]

Bookies came up fine after above changes and I see hostname configured for bookie host.

Do you recommend any changes on Brokers conf other than advertisedAddress before I restart the same ?

[dsambandam]

Brokers seems to be working now and able to discover the bookies configured

13:01:31.329 [bookkeeper-io-12-11] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: host5:3181
13:01:31.332 [bookkeeper-io-12-17] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: host4:3181

[sijie]

@dsambandam yeah broker just needs to set advertisedAddress. bookies are tricky since it has a lot of defense mechanisms to prevent things go wrong, since it is a storage system.

Glad to hear that it is working for you now!

[dsambandam]

Thanks for your help..I shall keep you posted for any further questions.

[sijie]

great! feel free to ping us via github issues, mailing list or slack channels. Glad we helped you.