Codeception: Add ordering for suites

[LiamBest]

I need to run suites in a certain order, however there does not appear to be functionality for this.

I have the same issue with the tests.. but I get around it by naming the files in a certain way.

(Relates to: https://github.com/Codeception/Codeception/issues/418)

[tiger-seo]

Tests shouldn't have influence on each other, they must be isolated and that's the best practise, that's how you can avoid issues with tests execution order.

[LiamBest]

The tests aren't influencing each other, however one suite needs to run first as it creates the user - the second suite then using that userid to run further tests.

[Tro95]

Then they are actually influencing each other. What we do is we fire some SQL to create all the necessary things to test with before each test and then revert it at the end. If you need the user for testing, you should create it at the start of every test.

[tiger-seo]

I have similar needs, and my way is such that I've done helper-methods to get me fixture before conducting any assertions, and for those users which are being used among of the majority of tests I use sql-dump, but such user is read-only, to avoid influence between tests.
I don't remove fixtures right after the test, they all are left in DB till the next tests run, so that DB will be repopulated from scratch, and that is done for performance reasons.
p.s. In the past I struggle a lot from tests wasn't isolate, but those times are past :)

[tiger-seo]

here an example:

$I = new ApiGuy($scenario);
$I->am('Guest');
$I->wantTo('test forgot password functionality using API');
$I->haveHttpHeader('Content-Type', 'application/json');

$user = $I->addUserFixture(
    [
        'email'              => '[email protected]',
        'confirmation_token' => null
    ]
);

$I->seeInDatabase('nmtn_user', ['email' => $user['email'], 'confirmation_token' => null]);
$I->sendPOST("/api/v2/users/forgot.json", ['email' => $user['email']]);
$I->seeResponseIsJson();
$I->seeResponseCodeIs(HTTP_RESPONSE_OK);
$I->seeResponseContainsJson(['success' => true]);
$I->seeInDatabase('nmtn_user', ['email' => $user['email']]);
$I->dontSeeInDatabase('nmtn_user', ['email' => $user['email'], 'confirmation_token' => null]);

$I->addUserFixture() is my helper-method which creates and returns user (array type) using method argument, but I can omit user data, so that method will evaluate and use defaults for those which are not specified.

[Tro95]

I would recommend reading http://codeception.com/docs/09-Data to follow on from tiger-seo.

[DavertMik]

The final answer: won't be added in core.
probably we would allow this to be done via extensions.

[eolexe]

Actually I also found that could be useful. I my case I want to restrict dev's to rely on configuration in unit tests, but not in integration tests.
Example: I have 2 different suites one for integration another for units. If integration suite run first it loads configuration constants globally (Yii is used), and dev's still able to use. So if user runs both suites his bad unit test that rely on global config wouldn't fail, but if he run just unit test suite, it will fail.

[wasmithee]

Can we use @depends annotations across multiple CEST classes?

[buffcode]

I think it is kind of useful when thinking of "fast-fail".
In our CI system we run all suites, but unit tests are much faster the the acceptance tests.

I would like to order the unit suite first, so any failure aborts further load-intensive tests as they might or might not produce correct results, based on the correctness of the units.

[Ryuske]

I have the same use-case as @buffcode. I would like the tests to fail as quickly as possible. So my ideal order would be unit -> api -> functional -> acceptance. It's literally just so I waste the least amount of time possible waiting for tests to finish if one of them is going to fail.

[Faryshta]

It also makes sense for dependencies

If the unit test UserTest:testCreate fails, the api test ProfileCest:register will fail too.

[nicholascus]

I have usecase for that too. I have module level tests which restore database dumps, which are shared across multiple test classes. I want all tests working on the same database dump to be executed in one go, on top of that I would love to have option to randomize order in the group.

I haven't been looking at how that's implemented in Codeception yet, but I'll get to that for sure in the nearest couple of months.

[ebuildy]

As a developer, we want save time, using fast-fail, when we are working on a particular feature, we want to run in priority tests influenced by this feature.

Also, we want to run "heavy" tests at the end.

End, we may want run suites in a particular order "register" / "login" because we are lazy to make SQL dump ^^

[matthiasnoback]

Talking with @DavertMik I learned about this solution to determine the order of the suites to run:

vendor/bin/codecept run unit,integration,functional

Personally, I like to combine this with using @group annotations above test/cest classes to run tests from all suites that are in the same group (i.e. the feature I'm working on):

vendor/bin/codecept run unit,integration,functional -g [name-of-the-group]

Combined with --fail-fast, this can give you quite a fast feedback loop.