Pulsar: set offload bucket per topic and per tenant/namespace
Describe the solution you'd like
Do you think it's easy to set s3 bucket for offload to topic level and tenant/namespace instead of cluster level? (i.e. be able to configure one bucket per topic, and one bucket per tenant/namespace).
KannarFr
All 6 comments
I think the best idea is to have a default bucket which is defined in broker.conf and be able to configure one bucket per tenant and/or tenant/namespace and/or tenant/namespace/topic. Then when offload is threw, it will take the more specified defined. WDYT?
KannarFr
on 21 Jun 2019
@sijie I started to dev this feature. Where do we store the offload configuration per tenant/namespace? In BK as it's done for schemas? Or in ZK?
KannarFr
on 15 Jul 2019
(I started an implementation copied from Schemas usages).
KannarFr
on 15 Jul 2019
@KannarFr I think it should be part of namespace policies. Please check this class : https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/Policies.java
sijie
on 16 Jul 2019
Ok.
KannarFr
on 16 Jul 2019
Really looking forward to this feature. The following can be implemented once the directory structure for tenant, namespace and topic is there: https://github.com/apache/pulsar/issues/5978
shiv4289
on 2 Jan 2020
Related issues
frankjkelly
路
3Comments
dzmitryk
路
4Comments
pruthvivrp
路
4Comments
tfhappy
路
3Comments
Tielem
路
3Comments
Most helpful comment
I think the best idea is to have a default bucket which is defined in
broker.confand be able to configure one bucket per tenant and/or tenant/namespace and/or tenant/namespace/topic. Then when offload is threw, it will take the more specified defined. WDYT?