Pulsar: kubectl appply -f bookie.yaml andthen Error

[yangmingqi218]

14:32:40.592 [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.16.0.204:2181. Will not attempt to authenticate using SASL (unknown error)
14:32:40.597 [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to zookeeper/172.16.0.204:2181, initiating session
14:32:40.603 [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.16.0.204:2181, sessionid = 0x200e9db95fb0024, negotiated timeout = 30000
14:32:40.606 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
14:32:40.683 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
14:32:40.695 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: instanceId 098b7177-8e27-4c57-a38a-ddc230840b86 is not matching with aff0aa33-b168-4735-82f7-281900942137
at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:142) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:147) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.bookie.Bookie.verifyAndGetMissingDirs(Bookie.java:371) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:434) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:691) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:106) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:314) [org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]

[sijie]

@yangmingqi218 :

Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: instanceId 098b7177-8e27-4c57-a38a-ddc230840b86 is not matching with aff0aa33-b168-4735-82f7-281900942137

It seems the bookkeeper cluster is reformatted by metaformat. so the instance id stored in the bookie is different from the instance id in the cluster.

Can you describe more about the sequence you run for deploying the cluster? I guess you are using old persistent volumes but the zookeeper metadata is reformatted.

[sijie]

@yangmingqi218 any more details about this issue?

[raulnegrean]

@sijie I'm having the exact same issue. As mentioned in the documentation, I'm deploying the zookeeper stateful set first and then, as soon as the pods are up and running, I'm deploying the cluster metadata job. Next, I'm deploying the bookie daemon set, and that's when I'm getting the aforementioned exception:

$ kubectl logs bookie-zg92v
...
15:31:12.873 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
15:31:12.996 [main] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
15:31:13.012 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: instanceId 23ba2d60-6136-4300-960f-af622a870ec4 is not matching with c36550e6-a701-4b05-be49-709b0d2654c5
    at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:142) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:147) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.Bookie.verifyAndGetMissingDirs(Bookie.java:371) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:434) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:691) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:106) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:314) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]

Also, here's a list of all the available resources:

$ kubectl get all
NAME                                     READY   STATUS             RESTARTS   AGE
pod/bookie-zg92v                         0/1     CrashLoopBackOff   6          8m31s
pod/pulsar-cluster-metadata-init-4ptvl   0/1     Completed          0          8m45s
pod/zk-0                                 1/1     Running            0          9m20s
pod/zk-1                                 1/1     Running            0          9m5s
pod/zk-2                                 1/1     Running            0          8m56s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/bookkeeper   ClusterIP   None         <none>        3181/TCP            8m31s
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP             10m
service/zookeeper    ClusterIP   None         <none>        2888/TCP,3888/TCP   9m20s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/bookie   1         1         0       1            0           <none>          8m31s

NAME                  READY   AGE
statefulset.apps/zk   3/3     9m20s

[sudeepamin]

I get the same error. The deployment instructions seem to be incorrect. For example the aws folder does not have a cluster-metadata.yaml file. The bookkeeper.yaml fails to create bookies.

[valiantljk]

any update on this?

[rocketraman]

I'm having the same issue.

[TC-robV]

i found that if you use the deployment files in the repo that it creates directories on the nodes at /mnt/disks.

if you have installed it previously, you must blow away these directories. once i did that the next install worked fine.

[bry00]

I have the same issue. Any update on this?

[cutiechi]

Because the bookie nodes use host path, delete /mnt/disks/ssd0 and /mnt/disks/ssd1, then reapply

[cutiechi]

@yangmingqi218 Could you try it?

[yannick]

i also have this issue on k8s, however i only have the journal and ledger PVC mounted and no ssd....

[sijie]

for people how encountered issues here, the yaml files are pretty out-of-dated. It is hard for us to maintain individual yaml files for different platforms. we have decided to remove those yaml files.

Please consider using the latest helm chart for deploying Pulsar on Kubernetes. The Helm chart is now protected and verified using the Helm chart testing framework and it is the official support Helm chart by the Pulsar PMC.

• follow the instructions (http://pulsar.apache.org/docs/en/kubernetes-helm/) to get started
• follow the instructions to configure (http://pulsar.apache.org/docs/en/helm-deploy/) the helm chart to install a production setup