Pulsar: What official pulsar helm chart request 15GB memory for bookeeper,zookeeper and broker?

[benjaminhuo]

Expected behavior

The memory should be set to a reasonable size
The manifest(bookie.yaml) deploy bookeeper as daemonset, why not statefulset with persistent volume claim?

Actual behavior

Official pulsar helm chart request 15GB memory for bookeeper,zookeeper and broker

The manifest(bookie.yaml) deploy bookeeper as daemonset, why not statefulset with persistent volume claim?

Steps to reproduce

System configuration

Pulsar version: x.y

[sijie]

@benjaminhuo

Official pulsar helm chart request 15GB memory for bookeeper,zookeeper and broker

the helm chart was contributed from a production deployment in google cloud. so those values are set to pretty high. I think we can improve it by introducing 3 different groups of settings, e.g. small, medium and large. so people can choose the proper settings to deploy to their environment. I am wondering if you are interested in contributing a change to this helm chart :)

The manifest(bookie.yaml) deploy bookeeper as daemonset, why not statefulset with persistent volume claim?

I think helm chart deploys bookkeeper as statefulset. https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/bookkeeper-statefulset.yaml

You mean the kubernetes manifest here: https://github.com/apache/pulsar/blob/master/deployment/kubernetes/generic/bookie.yaml#L33

The reason why it is using daemonset for generic deployment is following: as a generic deployment, people might not enable any persistent volumes enabled in their k8s environment. so for they to tryout pulsar quickly, daemonset is an easy solution to start.

In cloud environment, for example, aws and google cloud, we are recommending people deploying using statefulset.

https://github.com/apache/pulsar/blob/master/deployment/kubernetes/aws/bookkeeper.yaml#L67

but since k8s is now supporting local volume claim, I think we can change generic deployment to use statefulset with local volume claim. wondering if you are interested in contributing to this as well?

Hope this answer your question?

[benjaminhuo]

Thanks for the explanation and suggestion.
I'd like to contribute.

Regards,
Ben

[benjaminhuo]

@sijie Another irrelevant question regarding pulsar:
Is the connection to broker per client or per producer?
If I create multiple producers using the same client, there will be multiple connections or just one connection?

PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Producer .topic("my-topic")
.create();

// You can then send messages to the broker and topic you specified:
producer.send("My message".getBytes());

[sijie]

@benjaminhuo

Pulsar is using netty, so the connections will be managed per client.

If I create multiple producers using the same client, there will be multiple connections or just one connection?

it will be one connection per broker to the brokers that the client will connect to.

[benjaminhuo]

Great!
Thanks very much.

Ben

[sijie]

Closed this issue since the question was answered