Pulsar: Several issues with the kubernetes generic example

[pvanderlinden]

Expected behavior

It should be able to be adjusted to run more then one bookie.

Actual behavior

When you adjust the kubernetes example to run more then one bookie without multiple kubernetes nodes, the second bookie will crash with There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
now it's just on bookie-0 instead of bookie-1`

Steps to reproduce

Change the DaemonSet to be a Statefulset, and change replicas to 2

System configuration

Pulsar version: 2.2.0

More info

Debugged together with @ivankelly.
Several suggestions:

• change from DaemonSet to Statefulset as in the GKE example
• remove the environment to set the advertisedAddress from the hostIP, let it just use it's POD ip (like the GKE example)
• Remove the initContainer as there is a job to initialize the cluster metadata
• Change the container to wait for zookeeper to be up (apparently there is a script for this), as this removes the need to spin up zookeeper first as the documentation suggests.
• have PVC's for all zookeeper and bookie instances (not really part of the issue)

[pvanderlinden]

The zookeeper fix might not be needed as kubernetes will just restart the pod and by that time zookeeper should be available

[sijie]

copied the conversation from slack to here. I think the main problem is the deployment was changed to using statefulset deployment, but it still using hostIp as the advertised address. It is a problem of documentation. I would update the documentation to address it.

sijieg [9:05 AM]
@paultjuh looks like the issue is because you were trying to change a daemon set k8s deployment to a statefulset deployment, but still using hostIp as the advertised  address.

FYI. Generic is for daemonset deployment because it is assuming there is no PVs available in a generic deployment; if you want to use a statefulset deployment, a cloud one like GKE is better to use or modify.

I will take a closer look at your issue though.


paultjuh [9:32 AM]
Yeah, I realize that now.  I don't think the hostIp is needed though (edited)

ivan [9:46 AM]
podIp should work for both cases. or hostnameAsBookieId

paultjuh [10:09 AM]
without picking it up manually it also works as far as I know

sijieg [10:12 AM]
Well, that’s not true. The principle is using stable identifier for your bookkeeper deployment. In daemonset, only hostIp is a stable identifier to use; while in statefulset, the pod hostname is the one to use.
Using podIp is not correct for bookkeeper deployment
PodIP changes when the pod restarts
The suggestion of using PodIP is misleading
The problem of using PodIP only occurs after you restart pods

paultjuh [10:14 AM]
ah, my mistake
I'm not sure if there is anything against a statefulset with a preferred anti-affinity set
or it might be good to document(maybe with comments in the yaml), why certain things are set like that

sijieg [10:17 AM]
I think there is one section documenting that

paultjuh [10:17 AM]
it would be helpful, as now with generic I expected get something similar as GKE, but instead it's pretty different (no persistent storage, no replication for bookies)
Did I miss that?

sijieg [10:18 AM]
BK has its own replication. So there is no technical differences between statefulset and daemonset.
StatefulSet is good for cloud or your K8 already have persistent volumes. But the downside is you have extra replication in persistent volumes
Daemonset is good for a k8 cluster that doesn’t have persistent volumes.

paultjuh [10:21 AM]
ok, that might make sense indeed
although not sure if their would be many clusters without PV's
usually there is some support available
I can't find it in the docs though

sijieg [10:22 AM]
In either deployment, the key is to choose an identifier as the bookie id, the identifier should not be changed upon pod restarts
Why the examples in generic is daemonset is because people typically use it for trying it out and customize to its own k8 deployment. In that way, using a daemonset for generic examples is making sense for most of k8 clusters
Let me find the documentation when I am close to a laptop

paultjuh [10:24 AM]
thanks!

sijieg [10:40 AM]
@paultjuh - I think the documentation was in bookkeeper. http://bookkeeper.apache.org/docs/latest/deployment/kubernetes/#deploy-bookies I will update them to pulsar as well.

paultjuh [10:41 AM]
Ah nice, thanks!

[adamdyga]

Not sure if it's related, but I get similar error message when running a single bookie using the K8S examples from deployment/kubernetes/generic in minikube. After restarting minikube, I also get the following error:

16:16:49.822 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
16:16:49.822 [main] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
16:16:49.834 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: 
    at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:470) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:691) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]

What do I have to change to make it work with minikube?

[ivankelly]

@adamdyga probably related. If I remember correctly, you have to change the bookie deployment to be a stateful set and tell bookkepeer to use the hostname as the advertised address and to use persistent volumes for the bookkeeper storage. I'm not sure if this works with minikube, but it works well for us with kind [1] (with rancher local storage [2])

[1] https://kind.sigs.k8s.io/
[2] https://github.com/rancher/local-path-provisioner

[mingfang]

I'm still seeing this problem with 2.4.0.
I'm deploying bookie as stateful set and just using in container storage for testing.
The error seems to be related to zookeeper info and not related to storage.

[agilob]

I'm not using k8s, but this bug is transferable to docker and standalone too. I can confirm that running bin/bookkeeper shell metaformat fixes the issue of bookkeeper not starting.

I run this to fix the bookies not starting because of the error above

docker run -it --rm \
    --network "traefik-network" \
    --env BK_zkServers=zookeeper:2181 \
    --env BK_DATA_DIR=/data/bookkeeper \
    --env BK_journalPreAllocSizeMB=32 \
    apache/bookkeeper:4.10.0 \
    bin/bookkeeper shell metaformat

https://bookkeeper.apache.org/archives/docs/master/bookkeeperConfig.html

[sijie]

Since we have deprecated the generic k8s yaml files and moved to helm chart, please try the helm chart. Close this issue for now. If you have any issues related to helm chart, please let us know.

[mingfang]

@sijie Not everyone use helm

[sijie]

@mingfang Because a Pulsar cluster includes different components to deploy and there are dependencies between components. If we don't leverage a packaging tool like Helm to manage the dependencies, it is hard for us to maintain the project in the long run. I understand not everyone is using Helm. People can still retrieve the helm chart and use the helm template to render the YAML files. Hope that works for you.

[Lanayx]

I got this issue with fresh helm installation, the failing bookie didn't return back until I run _bookkeeper shell metaformat_. @sijie are you aware of any other less disruptive fixes for this issue?

[sijie]

@Lanayx what was the issue you encountered? Missing cookie? If it is a single bookie problem, you don't need to run "shell metaformat". We need to know what is the issue that you encountered and deal with that bookie.

[Lanayx]

@sijie The one from the header
There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
Yes, it was a single bookie problem, the bookie couldn't start and failed in crash loop. Since it was in crash loop I couldn't access it's terminal

[sijie]

Okay. What storage provisioner are you using? The error basically says a bookie attempts to start with empty directories. It means your disks are reformatted. Bookie prevents startup with this corrupted state. So it would be good to understand why the disks are reformatted.

To repair it, you can run decommissionbookie to decommission the bookie first. The decommission command will re-replicate the data and delete the cookie in the zookeeper. So the bookie can restart freshly.

But anyway the most important thing is to figure out why the bookie starts with empty directory.

[Lanayx]

Thank you for the explanation! This is a test cluster so we are using host directory and that was a storage failure test, so some error was expected. What was unexpected is that we had no idea how to "clean up" the bookie to return it back into cluster. Looks like decommissionbookie command fits this case well.