Pulsar: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.

[codelipenghui]

Master issue #4077
Is your feature request related to a problem? Please describe.
Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.

[feeblefakie]

Let me add a related issue and some discussion I and @codelipenghui had about this.

• Key_Shared consumers with different subscriptions get messages for the same key in an inconsistent order
• https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1585527971073400

I think sticky consumers won't really solve it since consumers can be easily down in failure cases and it is pretty hard for applications to manage scalability of consumers.

[poulhenriksen]

How come this is marked as a feature and not a bug?

It violates the ordering guarantee of Key_Shared, and basically makes Key_Shared useless if you dynamically add consumers.

I ask, because I would like to ensure this bug gets the proper attention, and is not regarded as a nice-to-have feature.

[feeblefakie]

@codelipenghui Could you share updates on this if any ?
Sorry for pushing, but I feel it is a really critical bug and Key_Shared users will have so much troubles if they just use it without doubt.

[codelipenghui]

@feeblefakie We are already advancing this matter and I have drafted an approach. I will create a PIP or push a PR soon.

[feeblefakie]

@codelipenghui Thank you for sharing the update ! Good to know. :)

[merlimat]

So, I think the problem here should be treated by having new consumers joining in a "paused" state.

For example, consider this sequence:

1. Subscriptions has c1 and c2 consumers
2. c3 joins. Some of the keys are now supposed to go to c3.
3. Instead of starting delivering to c3. We mark the current readPosition (let's call it rp0_c3) of the cursor for c3.
4. Any message that now hashes to c3 and that has messageId >= rp0_c3 will be deferred for later re-delivery
5. Any message that might get re-delivered (eg: originally sent to c1, but c1 has failed) to c3 and that has messageId < rp0_c3 will be sent to c3
6. When the markDelete position of the cursor will move past rp0_c3 the restriction on c3 will be lifted.

Essentially, c3 joins but can only receive old messages, until everything that was read before joining gets acked.

[feeblefakie]

@merlimat
I'm not fully sure about the solution but does it guarantee key ordering ?
For example, a message X with key "A" is first consumed in c1, and another message Y with key "A" can possibly go to c3 ? (can we make sure message X is consumed and acked before message Y ?)

[merlimat]

For example, a message X with key "A" is first consumed in c1, and another message Y with key "A" can possibly go to c3 ? (can we make sure message X is consumed and acked before message Y ?)

Yes, the above proposal takes care of that.

The important point is that X could be re-sent to c3, in case c1 fails just after c3 has joined.

[codelipenghui]

@merlimat I like the approach you have mentioned.

[codelipenghui]

@pouledodue @feeblefakie @merlimat I have pushed a PR #6977 for this issue. Please help review, thanks.

[codelipenghui]

Reopen it via https://github.com/apache/pulsar/issues/5819#issuecomment-644510797.

[rocketraman]

PR #7106 referred to by https://github.com/apache/pulsar/issues/5819#issuecomment-644543109 is marked as closed and merged, with milestone 2.6.0. Should this issue be closed with 2.6.0 milestone after all?

[codelipenghui]

@rocketraman As discussed with @feeblefakie in slack channel, looks there are some problems with auto split mechanism. So I reopen this issue for tracking this problem. And @feeblefakie has verified consistent hash mechanism is work as expected. I will take a look at the auto split mechanism later.

[feeblefakie]

@codelipenghui Should I create an open issue for this ?
I realized that I just created how to reproduce doc in my repo and shared it in only the closed issue.

[codelipenghui]

@feeblefakie Yes, thanks.

[feeblefakie]

@codelipenghui #7455 PTAL.

[wolfstudy]

The issue will be tracked in #7455 so move the issue to release/2.6.2

[wolfstudy]

It seems that #8292 has solved this problem, let us consider closing this issue.