Flank: Create & Run Integration Tests for Flank
Author the user story for this feature
As a developer, I want to run integration test on flank so I can test the library with a set of default test cases as though i was using flank as a user.
Is your feature request related to a problem? Please describe.
It is linked to the need to test a full production based build of flank, and to test if the minimization of flank will not introduce errors. It can also help with bug testing issues where we can replicate the issue by running it on a production version of flank.
Describe the solution you'd like
Either have it built into test_runner project or a seperate project like flank_scripts, whereby we can run a set of integration tests simply and easily.
Sloox
All 3 comments
@bootstraponline @jan-gogo @pawelpasterz @piotradamczyk5 @adamfilipow92
do we have any _preferences/ideas/comments_ as to how integrations testing may look like for this task?
shall it appear as a subproject i.e. test_integration with its own gradle build etc? Will there actually be any code? or all gradle commands? Should we head in the direction of the flank_scripts and have a fully fledged sub project that we can run?
At the very least it should in my opinion start by compiling flank and either copying that lib/binary out of the test_runner folder and running a set of tests on that binary so we have a reference point.
Any ideas or suggestions would be greatly appreciated.
WDYT?
Sloox
on 4 Sep 2020
Should we head in the direction of the flank_scripts and have a fully fledged sub project that we can run?
Yes. I think we'd create a new Kotlin project responsible for running Flank and verifying the tests execute successfully. The integration tests can be JUnit tests. Inside the Junit test, we'd use a freshly built version of Flank to run an existing apk/ipa and verify that the final result from Firebase Test Lab is what we expect.
I think these should be smoke tests.
Smoke tests are a subset of test cases that cover the most important functionality of a component or system, used to aid assessment of whether main functions of the software appear to work correctly.
The 3 critical integration tests I see are:
- Given a test apk on Android, does the test run successfully
- Given a test bundle for iOS, does the test run successfully on macOS
- Given a test bundle for iOS, does the test run successfully on Linux
Each test should be simple, designed to execute quickly, and be stable. If we have 3 solid integration smoke tests, then we can think about adding more advanced tests in the future.
bootstraponline
on 4 Sep 2020
Each test should be simple, designed to execute quickly, and be stable. If we have 3 solid integration smoke tests, then we can think about adding more advanced tests in the future.
Right, I love the idea to get something simple first.
jan-gogo
on 4 Sep 2020
Related issues
bootstraponline
路
7Comments
bootstraponline
路
6Comments
jrodbx
路
4Comments
bootstraponline
路
6Comments
rolyatwilson
路
5Comments
Most helpful comment
Yes. I think we'd create a new Kotlin project responsible for running Flank and verifying the tests execute successfully. The integration tests can be JUnit tests. Inside the Junit test, we'd use a freshly built version of Flank to run an existing apk/ipa and verify that the final result from Firebase Test Lab is what we expect.
I think these should be smoke tests.
The 3 critical integration tests I see are:
Each test should be simple, designed to execute quickly, and be stable. If we have 3 solid integration smoke tests, then we can think about adding more advanced tests in the future.