Asyncpg: How to override json encode/decode?

[mike-hart]

• asyncpg version: 0.8.4
• PostgreSQL version: 9.5
• Python version: 3.6
• Platform: Linux (Alpine, on Docker)
• Do you use pgbouncer?: no
• Do you use the wheel package or built locally
  (if locally, which version of Cython was used)?: wheel
• Can you the issue be reproduced when running under
  uvloop?: yes

I'd like to override json's encode/decode to avoid the need to dumps/loads my data-structure myself. However, I don't see any examples of how to override an existing codec so I end using connection pool's setup flag (as it's the only way I see to call set_type_codec). I end up with something like:

async def setup(conn):
    await conn.set_type_codec(
        'json', encoder=json.dumps, decoder=json.loads, schema='pg_catalog'
    )

This doesn't seem to do anything. So I add the binary=True flag and get:

cannot override codec for type 114

Since this runs on every .acquire (and, I guess, since there's an existing codec). I'd need to somehow prevent that since I only need to call it once per connection _creation_. So, I tried using a weakvaluedictionary:

CODECS_SET = weakref.WeakValueDictionary()


async def setup(conn):
    conn_id = id(conn)
    if conn_id not in CODECS_SET:
        await conn.set_type_codec(
            'json', encoder=json.dumps, decoder=json.loads, schema='pg_catalog', binary=True
        )
        CODECS_SET[conn_id] = conn

But this isn't permitted since connections don't have a __weakref__ attribute due to the use of __slots__:
https://docs.python.org/3.6/reference/datamodel.html#notes-on-using-slots

So this is really a few questions:

1. How do I replace the JSON codec?
2. How am I supposed to do something for connection instantiation rather than acquire? (Like this, or prepared statement preload, etc.)
3. Where can I read more about how the various codecs are laid out in binary format?

[elprans]

How do I replace the JSON codec?

Current asyncpg version does not allow replacing codecs for builtin types. I don't remember off top of my head why this restriction was placed, but it doesn't seem necessary any more. I'll merge #75 once the buildbots are happy.

How am I supposed to do something for connection instantiation rather than acquire? (Like this, or prepared statement preload, etc.)

There's no defined Pool API for that currently. Right now you can subclass asyncpg.Pool and override the _new_connection method.

Where can I read more about how the various codecs are laid out in binary format?

There's no formal documentation AFAIK. Read asyncpg (asyncpg/protocol/codecs) and PostgreSQL (src/backend/utils/adt) source.

[mike-hart]

Thanks, your merge has all the info I needed for now. Likewise thanks for the additional detail re: _new_connection.

[vladyslav2]

Any plans to push it to master branch ?

[vladyslav2]

by the way, I use to get this error:

unsupported jsonb version number 34

  File "/Users/admin/projects/gftornado/uv/lib/python3.6/site-packages/asyncpg/connection.py", line 174, in execute
    True, timeout)
  File "asyncpg/protocol/protocol.pyx", line 165, in bind_execute (asyncpg/protocol/protocol.c:54999)

  <class 'asyncpg.exceptions.InternalServerError'> unsupported jsonb version number 34
------------------------------------------------------------------------ Captured stderr

so I updated my encoder to

def _encoder(value):                                                              
    val = bytes(json.dumps(value).encode('utf-8'))                                
    val = b'\x01' + val                                                           
    return val 

\x01 helped but I'm not sure if that a right solution

[elprans]

75 has been merged. Closing.

[1st1]

BTW, in asyncpg 0.10.0 (about to be released) you shouldn't override _new_connection(). Use the new init kwarg of create_pool().