Pulsar: Support for delayed message delivery

[kesavkolla]

Scheduled and delayed message delivery is a good feature to support in pulsar. Basically individual message can have a header which will be set by publisher and based on the header value the broker should hold message in queue. Upon the interval is over message should be delivered to subscribers

[codelipenghui]

Is any idea to implement this feature? I'm interest to join the discussion of design and contribute for this feature.

[skyrocknroll]

This is an excellent feature we heavily depend on the current messaging system.

[pmwqld]

We would like to use such a feature to implement an AUTO Dead Letter Queue (DLQ). Any messages that cannot be processed from the main topic can be sent to the AUTO DLQ topic and only become "visible" after a particular time depending on the retry and backoff algorithm.

[codelipenghui]

@pmwqld

We would like to use such a feature to implement an AUTO Dead Letter Queue (DLQ). Any messages that cannot be processed from the main topic can be sent to the AUTO DLQ topic and only become "visible" after a particular time depending on the retry and backoff algorithm.

I think if all messages with the same delay time in the topic, use Dead Letter Queue can match this scene. otherwise the dispatcher logic in broker and limited by the receive queue will affect the delay result.

[codelipenghui]

Delayed message redelivery solution based on expose redelivery count to client. #3030

[Ah39]

mark ,the feature have finished ??

[codelipenghui]

@Ah39 PR #3033

[sijie]

@Ah39 it is not completed yet. there is an ongoing PIP for it.

[skyrocknroll]

Getdelivery count does not cover all the use cases. We are pushing the logic to determine whether the delay has expired to the client(Implementing this in all the client languages will be repetitive & some language clients may not even get this feature implemented). If the broker sends the message to the client after the expiry of the delay then it would be simple and straightforward to remember.

[skyrocknroll]

@sijie is there any link to the PIP definition to share?

[sijie]

@skyrocknroll : https://lists.apache.org/thread.html/977a8042581819769809f298bd2a2e9fd6b78598792c98069eb691e5@%3Cdev.pulsar.apache.org%3E this is the discussion thread.

[sijie]

PIP-26: https://github.com/apache/pulsar/wiki/PIP-26:-Delayed-Message-Delivery