Pulsar: Tag Pulsar ledgers in order to distinguish from other ledgers in the same Bookkeeper cluster

[eolivelli]

Is your feature request related to a problem? Please describe.
I need a way to distinguish Pulsar ledgers from ledgers created by other applications.

Describe the solution you'd like
An idea to solve the problem in the short term is to add some "custom metadata" on every ledger created by Pulsar.
It would be good to add metadata to link the ledger to the owner component in Pulsar (topic...)

Describe alternatives you've considered
We can start on BookKeeper a new feature to add a (optional) "Application Type" client configuration which would be written to Ledgers metadata.
This can be a better solution in the mid term but it requires more coordination with BookKeeper community and it can happen before BookKeeper 4.10.
My need is to have such tags for Pulsar + BookKeeper 4.9.

[eolivelli]

@merlimat @sijie @ivankelly how does this sounds to you ?

[jiazhai]

@eolivelli Would you mind share a little about How the tag will be used?

[ivankelly]

Sounds like a good idea. I take it that these tags will are aimed at debug/ops usecases?

[ivankelly]

While you're at it, you could change managed ledger to use WriteHandle. It already uses ReadHandle in the read path.

[eolivelli]

@jiazhai @ivankelly these tags in my case are only for debug/ops.

When an operator (using an automatic tool) looks at the contents of BookKeeper he must be able to understand which application/sub-system created the ledger.
Especially the case is for orphan ledgers, the application creates the ledger then it crashes and so there is no track of that ledger, it will remain forever on metadata.

[eolivelli]

@ivankelly

While you're at it, you could change managed ledger to use WriteHandle. It already uses ReadHandle in the read path.

I can do this change but I guess it will be a different patch, I did not look at Pulsar write path yet but if it is using the old API it will be using callbacks and not CompletableFuture.
I will be happy to contribute a patch

[eolivelli]

@jiazhai @ivankelly Am I in time to send a patch at least for the new custom metadata and have it in the new release of Pulsar ?
I am asking just to know if I have to work on this as soon as possible or I will eventually see mu change within months.
AFAIK the new Pulsar release is just waiting for BK 4.9 and this will happen very soon (hopefully)

thanks

[sijie]

Enrico, yes, you are still in time for a patch for adding tags for ledger metadata. Agreed that changing to WriteHandle should be in a separate patch. It would be great if you can create a GitHub issue, so that people in the community can pick it up at their convinience.

[eolivelli]

Another question:
Should I add a flag?
Custom metadata are there aince bk 4.5, so there should not be a real compatibility issue

[merlimat]

@eolivelli I think we should use the LedgerMetadata.customMetadata and add few infos there,
something like :

{
   "application" : "pulsar",
   "component" : "managed-ledger",
   "topic" : "persistent://tenant/ns/my-topic"
}

For cursor ledger, we should also have the subscription :

{
   "application" : "pulsar",
   "component" : "managed-ledger",
   "topic" : "persistent://tenant/ns/my-topic",
   "subscription" : "my-subscription"
}

I agree that it's not a problem anymore, since we don't need to maintain the compatibility to rollback a live cluster to BK 4.3 anymore.

[eolivelli]

Great, it is time to write code now!

[eolivelli]

@merlimat @ivankelly I am not sure how to work in BookkeeperSchemaStorage, any hint ?

[ivankelly]

How do you mean? I would just change the component to "schema", everything else should be similar.

[eolivelli]

@ivankelly
I am not sure how to get the 'schema name' from the objects I have around createLedger

[ivankelly]

Just pass in SchemaId from the calling methods to BookkeeperSchemaStorage#addNewSchemaEntryToStore

[eolivelli]

@ivankelly thank you.
I am totally new to Pulsar internals !

[eolivelli]

@ivankelly this is my workbench #3525