Azure-sdk-for-js: [Event Hubs] Batch Receive or flow control on event processor host

[elvinm]

Is your feature request related to a problem? Please describe.
When using EPH it is possible to get into a situation where an earlier message never finished processing but a later message does and successfully checkpoints. If the service dies at this point, upon restarting the EPH messages will start from this later offset even though the earlier message was never successful.

I wrote some sample code I'm happy to share to demonstrate this where I send 11 messages, allow all but 1 in the middle and the latest one to checkpoint. Then upon restarting I see message 11 but my message 6 for example never got to process (i.e. due to a longer processing time).

Describe the solution you'd like
I'm sure there are a bunch of different possible solutions here for example a batch mechanism would allow me to receive those 10 messages as a batch and checkpoint sequentially or checkpoint the latest only if all 10 succeed.

Otherwise possibly some flow control to not send events for all 10 messages until earlier ones checkpoint successfully (of course configuration might need to be applied here or different methods of consumption).

If there are already ways to handle this situation I haven't thought about that would be even better.

[kurtzeborn]

Thank you for opening this issue! We are routing it to the appropriate team for follow up.

CC: @ramya-rao-a, @AlexGhiondea

[ramya-rao-a]

Thanks for reporting @elvinm and apologies for the delay in response

Event if the current model that uses callbacks is updated to return batches of events instead of single events, we would still have the same problem because the callback for the second batch will not wait for the first batch to complete.

What we need here is newer apis that would let you wait until the current event or batch of events are processed before proceeding to the next

The api to work in batches is available in the underlying library @azure/event-hubs which you can try. But you will have to maintain checkpoints at your end if you use this library instead of EPH.

We will keep this issue open for tracking to add the same feature to EPH

[princjef]

Just wanted to check in on this. Are there any plans for adding this pattern to the JS EPH implementation? Without it there's no way for us to control concurrency/rate limit or guarantee at least once delivery.

[ramya-rao-a]

We are currently working on previews for the next version of the EventHubs library where we will be providing the same EPH support with back pressure such that a callback for the next batch of events will not be called unless the previous call to the callback finished.

By next month, we will have a preview out with the above feature. I'll post an update here then

[ramya-rao-a]

Hello all,

We have come a long way working on the next version of the Event Hubs library.
We are still in preview, but would like your thoughts and feedback on the new model.

Please checkout v5.0.0-preview.6 of the Event Hubs library. Below are some useful links to get started

• Package (npm)
• Readme
• API Reference Documentation
• Samples

[ramya-rao-a]

One more point, the capabilities of the older Event Processor Host is now folded into the main Event Hubs library. See the sample for receiving events with load balanced for details

[ramya-rao-a]

Update:
We have released another version of the Event Hubs library, please checkout the version 5.0.0-preview.7.
It has small API changes from the previous version

We will keep this issue open until the above library moves out of the preview state

[ramya-rao-a]

Hello all,

We are happy to announce the version 5 of @azure/event-hubs library is now out of preview.
Among other things, this replaces the need for the @azure/event-processor-host library and provides the flow control requested in this issue.

Please checkout the migration guide to see how you can migrate your application to use the latest version of the event hubs library.

We look forward to hearing any feedback you have on the new apis, samples and/or docs.