Fastapi: [QUESTION] Example for running celery task which includes db transcation in async fastapi app

[codesutras]

Is there any working example of running a celery task, which has some DB transactions ( async DB transaction) ??

Note: I'm using MongoDB (motor) for db utils.

This URL is a well-documented example for running fastapi with MongoDB in async fashion.

I've checked all project template generator as well. None of them have any example for having a celery task interacting with db transactions. ( I'll be okay to contribute on this once these current issues will get resolve)

I've found that there is no support for an async function in current (stable) celery release. Until then, it's going to be a massive rewriting for me,if I wanted to use fastapi with celery ( Need to drop off MongoDB async way).

Is there any better way to deal with this situation? So, that fastapi can have few endpoints work with MongoDB (motor) transactions and a few endpoints ( which have celery task) can consume db functions in sync way to work well.

In a nutshell,
Q1: How to handle async db transactions in the celery task?
Q2: Can we have async and sync db connection in fast api?

The reason, I'm asking these questions because I'm getting below exception in the celery tasks. which are depending on db calls ( mongodb motor).

kombu.exceptions.EncodeError: Object of type coroutine is not JSON serializable
@tiangolo Any thought would be appreciated. Since I really don't want to lose the power of async and it's performance advantages with fastapi.

[retnikt]

Can you provide a code sample for this problem?

[codesutras]

@retnikt I've achieved to run celery task with DB transactions in async fashion. Though, it was not at all straight forward solutions. I have to do some cleanup in code and will share it after that.

I'll create a git repo with detailed information for using celery task + scheduling + fast API in next few days.

[tiangolo]

Thanks for reporting back and closing the issue. :+1:

If using async functions for distributed background tasks, I would try ARQ :wink:

[codesutras]

Thanks for reporting back and closing the issue.

If using async functions for distributed background tasks, I would try ARQ

@tiangolo I'd tried ARQ already. But, it can't handle too much load. It got failed in my load testing. Celery was the only solution in my case. Though ARQ works pretty well if you don't have a much-complicated task to be performed.

Nevertheless, Celery coroutine support is in pipeline and will get a release in celery 5.0 as per the github discussion. Code is almost ready for it. But, I'd reported an issue in it.