Riot: tests: Common output format

[roberthartung]

Hey everyone,
I was porting our own board to RIOT and was about to integrate the driver tests into our CI system. However, I noticed that there output of the tests I am using (driver_adxl345, driver_l3g4200d, driver_bmp180 for now) do not have a common output format and also they run "forever".
Is there any "best practice" for tests? E.g. an "[OK]" for initializing, some data output and finally a "[Done]" output? Or should we come up with such thing?

• Robert

[miri64]

I think @smlng had some ideas on that. I'm not sure how @kaspar030's #7258 is taking these kind of infinite tests into account.

[smlng]

My idea was to define common strings to indicate success or failure of a test, which would also allow an automated test framework to looks for these strings. However, until now we do not have such thing and its inconsistently handled in our tests. I personally try to use [SUCCESS] and [FAILURE] as the final output of a test, but we also have tests that run in a loop - but on the other hand, we could say a tests succeeded if it didn't fail?!

[miri64]

but on the other hand, we could say a tests succeeded if it didn't fail?!

We solved the halting problem \o/

[roberthartung]

@smlng I thing a test should always have SUCCESS and FAILURE maybe? But that doesnt apply for examples where a shell is started I guess? And for those running in a loop: Do they have to be in a loop? I guess it makes sense for the sensors, but from a CI Point of View it doesnt make much sense. Nevertheless, those tests could still print out a [SUCCESS] or [FAILURE] and possibly a [FAILURE:] ?

[smlng]

There are lots of things to think about when it comes to automatic testing:

• common messages/labels/tags/returns (what ever we call it) for success and failure
• certainly, infinite loops are not good, so we have to use timeouts in the test framework or introduce a max-counter instead
• timing is also an issue, many tests sleep or wait before continuing - for auto-tests such intervals should be small

to name only a few, and at best these things should be configurable through the build (and CI) system, especially timing intervals and max-counters.

[roberthartung]

@smlng I am currently working on an improved version of our (some colleagues and mine) test framework for those cases. It is a python script configurable by different YAML configuration files (Suite: Directory to RIOT/Contiki, ...; Nodes: Node types, serial devices, IDs, ...'; Tests: which tests to run, which output parser to use, directory, make command, arguments, ...).
I would be happy if we agree on such stuff and create a "test.h"-file that should be included in the test providing some macros/defines to use (e.g. TEST_SUCCESS; or TEST_FAILURE("reason") macros?). And I would also like be happy to contribute this test environment to RIOT, e.g. include it in a future build perhaps so people can use it?

[kaspar030]

I am currently working on an improved version of our (some colleagues and mine) test framework for those cases. It is a python script configurable by different YAML configuration files (Suite: Directory to RIOT/Contiki, ...; Nodes: Node types, serial devices, IDs, ...'; Tests: which tests to run, which output parser to use, directory, make command, arguments, ...).

That sounds painfully close to what I wrote in #7258. Maybe we can share insights and hopefully code?

[smlng]

@roberthartung please coordinate with @kaspar030 on that matter, he is currently working on this on the RIOT side, see referenced PR in his comment, too.

[roberthartung]

@kaspar030 You are not at the hackathon end of the month, right?

[kaspar030]

@kaspar030 You are not at the hackathon end of the month, right?

unfortunately not.

[roberthartung]

@kaspar030 I had a nice discussion with @OlegHahm and @haukepetersen - Let's have a chat / call soon to discuss our solutions and requirements?

[roberthartung]

@kaspar030 ping

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.