Asyncpg: Is asyncpg supported by Django?

[lospejos]

Is asyncpg supported by Django?
AFAIK Django requires some additional features should be implemented by the driver to be able to use it.

[1st1]

No. Django uses synchronous IO, whereas asyncpg is designed for asynchronous IO. We have plans to add sync IO support in the future, but have no ETA. Closing this one.

[rougeth]

@1st1 any changes on this issue?

[meshy]

Now that django-channels is backed with asyncio, this could be a very interesting possibility.

[pzelnip]

Not really a change, but https://www.aeracode.org/2018/06/04/django-async-roadmap/ discusses some of the complicating factors around making Django IO async.

[timkofu]

How are things looking in 2019?

[dbarbeau]

Django 3.0 is on its way. I guess that now that the fundamental bricks are being laid down, it would be a matter of reimplementing this whole class https://github.com/django/django/blob/194d1dfc186cc8d2b35dabf64f3ed38b757cbd98/django/db/backends/postgresql/base.py#L66 against asyncpg instead of psycopg. Probably a bit of a moving target for the moment?

[1st1]

Probably a bit of a moving target for the moment?

This is some great news! I doubt that we'll have resources for maintaining Django compatibility layer. It should be a separate PyPI package maintained by the community.

[darealzz]

How are things going in 2020?

[Andrew-Chen-Wang]

Hey all, I wouldn't mind starting a ticket in Django to get asyncpg in as a backend. I just ported django-redis+redis-py to django-async-redis with aioredis, so I don't think the task is too daunting as much of the functionality is the same between psycopg2 and asyncpg (AFAI can tell).

So long as there are people who can help me (or at least provide guidance as to what's different between psycopg2 and asyncpg), then we can get started by October 21st, day of my last midterm :) The only problem I can foresee is finally testing it with the ORM/builder stuff. For now... It's just getting that backend in.

My plan? Copy pasta pretty much all of psycopg2's backend to asyncpg's codebase and see where the caveats are and what needs to be changed. Most of the psycopg2's backend is SQL or some basic database adapter functionality which is similar to asyncpg (I was originally planning on doing this with psycopg3, but saw its development being slowed down by busyness).

Hopefully this can take 10 days of porting including work/school. If anyone's interested, just leave a comment here and ping me!

[ckcollab]

@Andrew-Chen-Wang I'm not sure I have the expertise to help out much, but I'd be very curious in following along if you don't mind linking PRs/etc. progress here. I'll try to help out as I can/weekends! maybe I can write a test case or fix a failing test case or two?

[Andrew-Chen-Wang]

That'd be great @ckcollab ! I can link a PR here once I get a first commit down. Any help with just a bit of knowledge in asyncpg is greatly appreciated!

One of the biggest issues I faced making django-async-redis (which backed me up nearly 5 days) was some blocking async context manager in aioredis that blocked pytest from properly moving on to the next test case because of an open socket conn. This was all due to me just never using aioredis before.

In other words, I've never used asyncpg before :P So any experience and assistance is greatly appreciated.

[Andrew-Chen-Wang]

Hey @ckcollab I've recently finished adding async caching to Django core at https://github.com/django/django/pull/13508. I'll be working on asyncpg now at ticket-32092. If you or anyone else who would like to help me make this PR happen have a Discord (sorry, I'm young) or some neat form of communication, please let me know via email which you can find in my profile.

[ckcollab]

@Andrew-Chen-Wang

    def clear(self):
        with self._lock:
            self._cache.clear()
            self._expire_info.clear()

    async def clear_async(self):
        async with self._lock_async:
            self._cache.clear()
            self._expire_info.clear()

I'm curious if most of the other porting/etc. has been done this way? it seems quite cumbersome to have so much duplicated code but I'm unsure if this is best practice or not for some reason?

[Andrew-Chen-Wang]

@ckcollab Unfortunately, for LocMem, they use threading.Lock but in an async context, I'm forced to use asyncio.Lock() which make me using async with.

And yea, a lot of the porting had to be done with very little DRYness to them. If you look at that 1,000 line test addition, there was no way of not duplicating because the way DEP 9 is specified, we have to add the new methods: get_async and set_async and then we have to await them. If I were to try to abstract these usages, then there would be some more issues down the line as the ORM/DB is core to everything. It's very noticeable when you take a look at pickle.dumps in LocMemCache's set_async.

What ended up happening was picklewas evaluating the queryset which ended up raising a SynchronousError

And then finally, to abstract the small chunks of code would be adding more lines than if I simply just copied and pasted everything.

---

Regarding asyncpg in Django, it's really going to be the same thing. Some things can be copied and pasted but the majority needs reworking with async context in mind.

That's probably the price to pay when supporting two different contexts. When using sync, you can develop much faster. When developing async, you have to be careful since you're trading development time for performance (which is why they first recommend you program everything in sync first).

Edit: just to make it clear, most of porting done by Andrew Godwin has been using asgiref.sync.sync_to_async which makes everything somewhat DRY, but that decreases performance significantly as you have to continuously switch contexts.

In retrospect, LocMem could have been more DRY in most methods which I think I can abstract away... I just worry that down the road, when it comes to synchronous blocking, there'd be issues. But for sure, that can be made more DRY!

[ckcollab]

Gotchya, just double checking -- that makes sense.

In retrospect, LocMem could have been more DRY in most methods which I think I can abstract away... I just worry that down the road, when it comes to synchronous blocking, there'd be issues. But for sure, that can be made more DRY!

can always optimize/refactor after it's working, ty for your work on this! I will try to pull away from work and help in the upcoming weeks. btw, I think the best place to poke at people for this would be IRC? #django on freenode

can try it out here (channel #django):
https://webchat.freenode.net/

you'll need to register first:
https://freenode.net/kb/answer/registration

hope this is helpful! I am sure there are discord communities for django as well, but IRC is pretty great

[Andrew-Chen-Wang]

Oh yeah they've got an IRC!

I will try to pull away from work and help in the upcoming weeks

That's awesome! I've been snooping around asyncpg and there are a lot more differences between asyncpg and psycopg2 and 3... I'm also just waiting to get the ticket triaged before I begin. The other thing that might back this up is that everything is connected via a Dictionary called connections (I think). That might need a bit of reworking beforehand, but I'm not sure.

Btw, I did end up trying to abstract it. Turns out, git diff says the number of lines is the same lol (32+, 32-).