Prefect: Integrating with an event-driven architecture

[jhagege]

Hi, is there any possibility to integrate a prefect pipeline with an event-driven architecture.
I'd like to have steps trigger external jobs on different infra and block until those jobs communicate their response through existing pubsub infrastructure.
On successful receival, I'd like to unblock the step and pass him the result from the pubsub message.
Can you think of any possibility to handle it ?
Thanks.

[jlowin]

Hi @jhagege, we have explored event-driven flows a few times in the past but failed to find sufficiently broad use cases to motivate building it directly into Prefect, as opposed to using an external service.

However, it seems that just in the last couple weeks we've heard a variety of use cases that may motivate a new feature. Could you take a look at our original design document and let us know how it aligns with your use case; could you also provide any further detail about how you envision it working so our team can include your use case as we explore?

Thank you!

[jhagege]

Hi @jlowin , thanks for your quick reply.
Actually I thought implementing it with asyncio.Future() instances as class constants on the Flow class.
Then I could update their result from an outside pubsub message on event-receiving.
Do you think this could integrate elegantly in prefect as is, or would that require some extra feature on your end ?
Thanks for your insights.

[lauralorenz]

Hi @jhagege! As @jlowin mentioned, Prefect doesn't currently support a first-class integration with event-driven architecture (though we have a group working on it in the public slack if you'd like to join us for the discussion, and the design document @jlowin mentioned has the basis of the current thinking of how to achieve that).

Just to circle back more specifically on your case, you may be interested enough to put something together yourself without an extra feature on our end, which more or less revolves around the ability to trigger a flow with the graphQL client or, more simply call flow.run() from some intermediate you set up (like a lambda function or a Kafka worker) that is triggered by or polls for your event. If I understand your use case properly the Prefect flow could be written with custom tasks that handle submitting the job, blocking until completion, and communicating back to your pubsub infra.

On the asyncio suggestion, I may be missing exactly what you mean but currently flow tasks (and flows themselves) are not called async in an event loop, so I don't think that would integrate in out of the box.

[jhagege]

@lauralorenz thanks much for your detailed suggestions.

[lauralorenz]

@jhagege I'm closing this issue for now as I think your original question was answered, but I'll keep you in mind to ping you as we keep researching and eventually move forward with an implementation for stream processing in prefect :+1: