Docker-mailserver: Separate test cases

[17Halbe]

Context

With the increasing amount of test cases testing is becoming more and more the bottleneck in development. Even a fast cached (make all-fast) build on a current PC currently takes around 17 minutes to complete.

My proposal would be to split the tests into separate test files according to the containers they are executed from.
The user could then decide to execute all containers and therefore all tests or just the particular container and its associated tests.
The Makefile would be getting a new optional argument CONTAINER='container1 container2'

Furthermore I would like to introduce a optional argument PERF=(int) setting, to respect different performance capabilities of different setups.
The setting would basically be a factor applied to any sleep command in the Makefile.

Pros

• shorter debugging times(Performance factor and selective testing)
• smaller and therefore more readable test files
• more granular debugging
• less ressource heavy. Containers would be removed after their tests exited successfully.
• downward compatible

Cons

• not one single searchable tests.bats file

[johansmitsnl]

Sounds good and it is indeed an issue.

We could also take advantage of Parallelizing the builds/tests https://docs.travis-ci.com/user/speeding-up-the-build/ ?
The issue is that the docker container needs to be build and building on each node does not sounds efficient. Any ideas on this?

[mwlczk]

What about splitting the big container/services into a "composition" of postfix, dovecot and antivir as proposed in #539. We could have many "service"-specific tests, and some "global" tests at the end.

[17Halbe]

@johansmitsnl That's a great idea, I'm not really familiar with Travis though. But as we would already separate the tests, we could easily perform the tests directly after container creation in parallel. I'll have a look into parallelizing Travis.
If I'm not totally wrong about this, we should be able to deny multithreading to the build target and then create the containers all based on the built test image.
@mwlczk This is an interesting idea, but doesn't really have to do with testing, right? Sounds like a huge undertaking though! ;)

[johansmitsnl]

@mwlczk there is already a ticket about this: #539 take a look at the conversation there.

[crazystick]

It would be great if tests would not break each other. For example, the mail container is used by a bunch of tests, and adding a new test which sends an additional email breaks earlier tests that count emails, and debugging is hard because yet another test rotates the log file and so there's no data to find out why something failed earlier on. Maybe each test should have a separate file which also does setup/teardown?

[17Halbe]

That is indeed desireable but quite a lot of work to do. Like your example regarding those log file checks. Some could probably be targeted more specifically, others don't and would have to be rewritten altogether.
I would suggest to separate the tests, doing multithreaded container starts and tests first and then, if that's all working, we could look into changing individual tests.
I separated the tests into files regarding the container they are run on. I don't see a benefit in separating them even further on a "one test per file" basis.
In the long run I would deem it necessary for anyone writing tests, to clean up everything they changed or added for testing afterwords and before their test case exits.
Another problem with those targets generate-accounts-after-run fixtures in the Makefile. They are test specific but get applied on a global basis because they are too time consuming and/or resource heavy.

[crazystick]

I suppose if each 'test set' has it's own container started and own bats file, why couldn't it have its own of fixture set too, controlled by the container arg?

I agree that refactoring existing tests into individual files is maybe too much, but going forward new tests could be written that way. I avoided making a new container even though it would be simpler, mainly because of the extra time to run the tests.

Would also be good to have a pattern for containers to have their own config directory. I did something like that when I added a container for testing relay hosts, but it felt like it didn't sit nicely in the existing folder structure...

[17Halbe]

I suppose if each 'test set' has it's own container started and own bats file, why couldn't it have its own of fixture set too, controlled by the container arg?

Agreed! 👍

I also like the idea about tests having their own config directories.
Thanks for the input!

[aendeavor]

@17Halbe has this stalled? @martin-schulze-vireso has made some good progress in #1613. You could have a look at this, maybe even contribute something:)