Azure-sdk-for-js: [Event Processor Host]: same event processed multiple times after checkpoint has been set

[danielepattuzzi1]

• Package Name: @azure/event-processor-host
• Package Version: 1.0.6
• Operating system: Linux
• [x] nodejs
  
  
  
  • version: 8.16.0
  
  
• [ ] browser
  
  
  
  • name/version:
  
  
• [ ] typescript
  
  
  
  • version:
  
  
• Is the bug related to documentation in
  
  
  
  • [ ] README.md
  
  
  • [ ] source code documentation
  
  
  • [ ] SDK API docs on https://docs.microsoft.com
  
  

Describe the bug
Scenario: multiple services (4 microservice instances of the same docker image running on Kubernetes) are listening from an EventHub using Event Processor Host library. Each service is a updating partition checkpoint every 100 events (checking that eventData.sequenceNumber % 100 === 0) by PartitionContext's checkpointFromEventData method:

await pc.checkpointFromEventData(ed);

Issue: noticed that sometimes, for each partition, the services are receiving the same events multiple times, even if the checkpoint has been previously set. As a consequence, our logs also state that the services are trying to set the same checkpoint multiple times.

To Reproduce
Steps to reproduce the behavior:

1. Send some messages to EventHub and verify that each message will be processed only once

Expected behavior
Message not processed again once checkpoint has been set

Screenshots

Additional context

[kaerm]

@danielepattuzzi1 thanks for reporting this, our team will look into it! //cc: @ramya-rao-a

[chradek]

@danielepattuzzi1
Can you share a little more about how quickly you are receiving events and if your processing is non-blocking?

The reason I ask is because one thing that could be happening is that you have multiple calls to checkpointFromEventData for the same partition at the same time.

This could happen if you have non-blocking tasks in the callback you pass to event processor host because EPH won't wait for those tasks to complete before giving you the next event. This is true even if you're using an async function and await. This could lead to the situation where you have multiple checkpointFromEventData calls to the same partition id, where the later one completes before an earlier one, causing it to later get overwritten by the earlier one.

Putting some logs around the actual checkpointing might help confirm if this is what's happening.

This is something we're addressing in the next version of the Event Hubs library (currently in preview).

[danielepattuzzi1]

Hey,

Yeah that’s exactly the case. We currently are calling async methods in the processing event business logic and EPH is not waiting before receiving the next event.
Also, this was causing our service to not be able to respond to our health service (which periodically pings the container).
We will definitely move to the new library soon. Some teams in my organization started adopting the Event Hubs preview library and it seems working properly ☺

Thanks,
Daniele

[ramya-rao-a]

The root cause of the issue here is that there is no back pressure i.e the callbacks provided to the receive method are called in a fire and forget way i.e we dont wait for a callback to complete any async work it could possible be doing before calling the callback for the next event.

We are currently working on previews for the next version of the EventHubs library where we will be providing the same EPH support with back pressure such that a callback for the next batch of events will not be called unless the previous call to the callback finished.

Please try out the @azure/event-hubs library version 5.0.0-preview.7 which we recently released.

Below are some useful links to get started

• Package (npm)
• Readme
• API Reference Documentation
• Samples

We will keep this issue open until the above library moves out of the preview state

[ramya-rao-a]

Hello all,

We are happy to announce the version 5 of @azure/event-hubs library is now out of preview.
Among other things, this replaces the need for the @azure/event-processor-host library and provides the flow control which resolves the problem described in this issue.

Please checkout the migration guide to see how you can migrate your application to use the latest version of the event hubs library.

We look forward to hearing any feedback you have on the new apis, samples and/or docs.

[laudibert]

Hello @ramya-rao-a,

Sorry to ask this here but does this mean @azure/event-processor-host will be deprecated and people using EPH should migrate to @azure/event-event-hubs ?

I don't see any mention of this on the README files of both library, only comments from you in severa issues concerning EPH.

Thanks in advance.

[laudibert]

Hello @ramya-rao-a,

Sorry to insist on this.

I also cannot put my hand on the migration guide you refer to.

[richardpark-msft]

@Zell211 - the migration guide (if you're still looking for it) is here:
https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/eventhub/event-hubs/migrationguide.md