Saleor: Tests takes long time to complete

[akjanik]

As the project is growing incredibly fast, tests needs a lot of time to complete, both on travis and local machine. We could try to investigate which tests takes most time, simplify and remove obsolete ones if needed.
It may not directly impact end user, but developer life would be much easier - I personally practice to run pytest before each push, just to make sure nothing is accidentally broken.

Possibly related with #2049 #2050 #2080

[Pacu2]

Have you tired running pytest on multiplecores? Any better?

[akjanik]

Yes I did, it runs faster, but it isn't an ultimate solution - what if a developer posses a bit older hardware? Also, decreasing overall test time could speed up PR process as sometimes we wait for travis to complete.
It's better to try to find a memory leak before you buy new RAM stick!

[Pacu2]

Sometimes hours spent on searching for memory leak might be hundreds times more costly than RAM stick itself... 😄

What's the total test time on your machine at this moment? With or without multiprocessing?

I think we need some investigation to check, if there's some problem with tests itself, or that's a natural outcome of adding new features.

[Pacu2]

List of tests sorted by time taken:
https://gist.github.com/Pacu2/18f534e60e8ce8d10696927e36ed3ae1

On my Macbook Pro (i7, 4x up to 4.1Ghz / 16GB Ram) tests took 187s to execute without multiprocessing
And 71s with multiprocessing enabled

I don't necessary consider that slow, however looking at the above list, there is some place for improvements, not big, but still.

[salwator]

There is nothing anyone could do with it in short term, apart from minor tweaks.

I can think of one strategy to deal with problem in long term:

• Write tests for new functionality / rewrites on lower level, mocking external side effects and perform only actions related to single test assertion
• Limit number of tests using views as entry point and treat them as integration tests, at this level check flow, not details. Do not perform those tests to every branching in code logic!
• Divide tests into separate pytest baskets, UT in one for quick execution, developer can run this basket on every source file changed and it is almost instantaneous. Integration tests moved to separate basket are run before pushing and obviously on CI
• Promote TDD practice among project contributors, it is self optimizing way of keeping tests quick and to the point
• Every major release (month or two) should be reviewed for test run time and sanity

[korycins]

@akjanik put this in your test settings.

PASSWORD_HASHERS = [
    'django.contrib.auth.hashers.MD5PasswordHasher',
]
DEBUG=False

I've reduced execution time from 250 sec to 120 sec.
pytest -n=auto --ds "tests.settings" --reuse-db tests/

[maarcingebala]

@korycins Should we add the above to our test settings as default?

[NyanKiyoshi]

@akjanik put this in your test settings.

PASSWORD_HASHERS = [
    'django.contrib.auth.hashers.MD5PasswordHasher',
]
DEBUG=False

I've reduced execution time from 250 sec to 120 sec.
pytest -n=auto --ds "tests.settings" --reuse-db tests/

Thinking back to this reply, you could add --reuse-db to PYTEST_ADDOPTS if you are running tests directly from a software/tool instead of a command line.

You would have:

export PYTEST_ADDOPTS='--reuse-db'

It will invoke --reuse-db by default in tests. Thus would make tests faster to run as they won't have to reapply migrations.

On Linux you would add that export to .bashrc. On MacOS you would run launchctl setenv PYTEST_ADDOPTS --reuse-db.

Then you would restart you shell and/or software (especially on Mac OS).

For example on PyCharm I use launchctl setenv PYTEST_ADDOPTS "-n 0 --reuse-db" first and foremost to be able to run the tests as PyCharm does not support pytest-xdist, then to reuse the db.

The only issue, is when you rewrite some migrations of course. In that case, you have to change the environment variable's value to recreate the db (or manually delete the test database).

Thanks @korycins! Passing from 10-20 seconds to 5 seconds.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.