Gino: Question about the transactions

[mister-vio]

• GINO version: 0.8.3
• Python version: 3.7.2
• asyncpg version: 0.18.3
• aiocontextvars version: 0.2.2
• PostgreSQL version: 11.4

Description

I'm trying to do a fixture which will wrap all tests with transactions and call rollback after test been done. And it looks like the fixture and test has a different connection because of all test works directly without a transaction.

If it is expected behaviour, can you please tell me how can I solve my issue?

My fixture

@pytest.yield_fixture(autouse=True)
async def rollback_db(loop):
    from proj.models import db
    async with db.transaction() as tx:
        try:
            yield None
        finally:
            tx.raise_rollback()

[wwwjfy]

I took a look at pytest code and it turns out that async fixtures are executed in its own context: https://github.com/pytest-dev/pytest-asyncio/blob/master/pytest_asyncio/plugin.py#L99-L102

One way I can think of is to have a fixture that initializes a transaction and returns the connection, and the tests use the connection to make calls. It'll be verbose, but I don't have a better idea now. :(

[mister-vio]

@wwwjfy how about to add an additional option to start a global transaction?

[wwwjfy]

It's indeed an option but it's less preferred because we may need to tweak core code. I'm not so willing to do that just for easier testing code.
Let me have some time to think about it. Maybe I can find a way to workaround py.test

[mikekeda]

@EasyLovv Why do you need to do rollback?
You could drop test database or truncate table instead.

[mister-vio]

@mikekeda yeah I know about this option, and actually, I made this workaround for now, but the canceling transaction will fork faster. So would be really tasty to have this functionality.

[wwwjfy]

Closing due to inactivity. Feel free to reopen.

[ricardogsilva]

I'd like to have this be reopened please.

I'm facing a similar issue as the one described originally.

If I understand correctly @wwwjfy stated that, as it stands now, in order to be able to test DB interaction via pytest-asyncio I need to explicitly use the connection that has the transaction inside the tests.

I guess this might work OK if my test function is creating the DB records. However it seems this will not work when I want to test my actual codebase and some function which is supposed to create records. I wouldn't want to rewrite my app's code to rely on explicit connections since it is already using implict connections. I also think this approach will not work when I start testing my app's web API, as I won't be able to pass the connection around.

I guess I'll have to create/delete my DB upon every test case, which will be slower.

Or might there be another option?

[wwwjfy]

If I understand correctly, it's not the same issue. Using context manager, you should be able to use the connection in an implicit way. Something like

async with conn.transaction():
    get_a_user()

[ricardogsilva]

@wwwjfy maybe I am not understanding what you propose, but this is what I am trying to do and it does not seem to work:

# file: conftest.py
# assume there is already a fixture named `create_test_database` that does create the DB correctly

@pytest.fixture()
async def db_connection(create_test_database):
    async with db.with_bind(config.DATABASE_URL) as engine:
        async with engine.acquire() as connection:
            async with connection.transaction() as transaction:
                yield connection
                transaction.raise_rollback()
        # verify if the rollback worked
        existing_styles = await db.all("SELECT * FROM styles")
        print(f"existing_styles after rollback: {existing_styles}")


# file: test_integration_db_crud.py

@pytest.mark.asyncio
async def test_create_style(db_connection):
    async with db_connection.transaction():
        await models.Style.create(
            id="4b1c383a-1a23-407b-a084-e6a8f2dd716e",
            name="test_simple1",
        )
    existing_styles = await models.Style.query.gino.all()
    print(f"existing_styles before rollback: {existing_styles}")

# pytest execution
$ pytest -k test_simple -s
platform linux -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
plugins: raises-0.10, asyncio-0.10.0

existing_styles before rollback: [(UUID('4b1c383a-1a23-407b-a084-e6a8f2dd716e'), 'test_simple1')]
existing_styles after rollback: [(UUID('4b1c383a-1a23-407b-a084-e6a8f2dd716e'), 'test_simple1')]                                                                                                                                                

As you can see, the contents of the DB are the same before and after rolling back the transaction.

Maybe I am misunderstanding your suggestion?

[wwwjfy]

What I mean is for this issue However it seems this will not work when I want to test my actual codebase and some function which is supposed to create records. I wouldn't want to rewrite my app's code to rely on explicit connections since it is already using implict connections., it can be solved by context manager, but NOT fixtures. We don't need to use explicit connection objects to let the "actual codebase" know, but the connection or transaction has to be created inside the test context, not in a fixture.

I do understand having such fixtures can make things easier, but I don't have a solution to that now. :(

[ricardogsilva]

Ah OK, I understand it now, pytest-async's fixtures are not really compatible with this stuff.

Thanks for your patience :wink:

[remarkov]

Does anyone found a good solution for this? Recreating the test database for each test works but it is quite slow on a large amount of tests. Explicitly opening a new transaction in each test is also quite inconvenient. Any advice would be much appreciated.
@ricardogsilva Could you please share what was your solution to this in the end?

[ricardogsilva]

@remarkov

I start out each test by opening a transaction, then I perform the test and finally I do a rollback - this is all done without using pytest fixtures, in order to use the same asyncio execution context.

It is not very elegant, but works great. More importantly, it is fast :smile:

Something like this:

from myproject import db  # this is the usual gino db

@pytest.mark.asyncio
async def test_something():
    async with db.bind.transaction() as transaction:
        # - first load whatever data into the db
        # - then perform test logic
        # - finally, clean up the DB by rolling back 
        transaction.raise_rollback()