Manim: List of tests to add

[huguesdevimeux]

At the moment we only have last-frames tests. Although they are useful, it's not enough. We would need to have tests for cli flags, that check videos, etc. This issue will keep a list of tests that need to be added. Please make more suggestions for tests!

Tests that need to be added

• [ ] We should have at least one test for each CLI flag.
• [ ] We should test that the cascading priorities in the config system work as expected. Ties in with #201
• [ ] We should create a test to certify a clean install. See #11.
• [ ] Testing video content
• [x] Test that manim runs correctly on all environments/prompts/versions #336
• [ ] Regression test for #240
• [ ] Test z-order stuff #327
• [ ] Test to ensure MovingCamera and ZoomedScene let the camera move as they should. From #388 .
• [ ] Test the functionality of asking the user for which scenes to render #523
• [x] There are no tests for the -s flag or the -t flag
• [ ] Test for add_fixed_in_frame_mobjects which is used in 3D scenes #569, currently broken
• [ ] Tests for SVG images
• [ ] @XorUnison Once video tests are implemented reverse_direction should get one
• [ ] @XorUnison Test for Brace

[naveen521kk]

I recently came up with some good things like coverage.py which is great. What it actually does is checks how many parts of code is executed when the tests are running and how many doesn't(for example checks how many if condition code are not executed as so on). It make beautiful html report or we can something like https://codecov.io for managing that. Pretty much, it will help us determine how good our tests are and to prevent code breakage. I suggest we should set it up soon and it isn't hard to it.

[leotrs]

Before looking into code coverage tools, I would like to see more tests.

[csrabak]

@leotrs :

The two subjects are well interconnected, we could say they're conjoined twins ‼️

Adding tests if do not exercise all the control flow paths may give a sense of security when in fact are only redundancies.

That's why coverage should be considered as integral part of the testing exercise.

my .01999....

[huguesdevimeux]

That's why coverage should be considered as integral part of the testing exercise.

I agree with that

[naveen521kk]

Probably we can open a new issue for that. @huguesdevimeux

[leotrs]

I agree that coverage should be taken into account, I just meant to say that right now the priority should go to writing tests. If anyone is looking into implementing coverage, that time would 100% be better spent by implementing more tests, as of right now.

Having said that, if anyone wants to do that now, then by all means. But deciding upon one more tool should be discussed in a separate issue, and we should talk about a tool that works for doctests as well, at least.

[eulertour]

I just wanted to add that testing frames in the middle of each animation would be very helpful, since a lot of the changes I've been making have passed the tests even if they don't actually render the video properly. The only solution right now is to constantly run several animations and hope they offer enough coverage.