Riot: tests: run tests on all boards.

[fjmolinas]

Description

For testing to be performed it must be easy. /dist/tools/compile_and_test_for_board was a stepping stone towards this.

Many boards can now run all tests in a row reliably. But there exception for boards that:

• make reset does not work
  e.g.: arduino, native, z1 (all passing by a BSL)
• make reset blocks
  e.g.: esp32
• make reset causes serial disconnect
  e.g.: atmega2560, mkrwan
• make flash fails without manual intervention
  e.g.: ccs2538dk,
• port with no stdin or stdout
  e.g: all mips

This issue wants to track the needed efforts to be able to achieve this. The following list is not exhaustive at it only involves the issues I'm aware off

Steps

• make reset does not work
  
  
  
  • [x] use test_interactive_sync_util in all possible test, i.e. test not disabling input/output #12461
  
  
  • [x] test application should run in a loop for tests that cant use test_interactive_sync_util (all aplications in tests do this)
  
  
• make reset causes serial disconnect
  
  
  
  • [x] use sync functions and disable reset on term
  
  
  • [x] wait for terminal to become available
  
  
  • [x] reset before opening the terminal.
  
  
• make reset blocks
  
  
  
  • [ ] not reset when terminal is open
  
  
• make flash fails without manual intervention
  
  
  
  • [ ] use usb-bootloader
  
  
  • [ ] blacklist tests that cause hardfaults (tests/ssp)
  
  
• general speed ups:
  
  
  
  • [ ] get rid of uneeded sleeps when possible.
  
  

[miri64]

* `make reset` does not work

I wonder how we would do this on native. It is effectively a process running on the host system, so we somehow need to restart that without effecting make term.

[miri64]

The other question: is it necessary for native?

[kaspar030]

I wonder how we would do this on native. It is effectively a process running on the host system, so we somehow need to restart that without effecting make term.

We could trigger native's "reboot" (which does a self-exec) via signal. For that, there needs to be a way to export the pid, maybe through a CLI argument setting a pid filename...

[miri64]

Signaling and rebootstrapping (not necessarilly via reboot) was also my thought. But saving the PID, especially with multiple native sessions (which especially considering some of our networking test cases is desirable), seems like a challenge.

[kaspar030]

But saving the PID, especially with multiple native sessions (which especially considering some of our networking test cases is desirable), seems like a challenge.

How are multi-node native tests usually run?

[miri64]

On multiple terminals, currently. But having some kind of automation integration should be possible for future use cases.

[miri64]

Have a look at https://github.com/RIOT-OS/Release-Specs/pull/79 for specifics e. g.

[miri64]

See #12517 for a PoC for native reset.

[MrKevinWeiss]

We should also try to get rid of the unneeded sleeps, with the sync tests should run much faster.

[fjmolinas]

The other question: is it necessary for native?

Coming back to this it isn't necessary, but make native resemble more to normal BOARD which also means the test running on native will be much more like the one running on a physical BOARD.

* `make reset` does not work

I wonder how we would do this on native. It is effectively a process running on the host system, so we somehow need to restart that without effecting make term.

But make reset will still not work on native even while there isn't a process running (previous make term), which is not the case for a "physical" board where I can reset, and the call term. This difference needs to be handled.

[fjmolinas]

But make reset will still not work on native even while there isn't a process running (previous make term), which is not the case for a "physical" board where I can reset, and the call term. This difference needs to be handled.

Thinking of it better, it would be the same as for z1, where it just does nothing.

[MrKevinWeiss]

• make reset blocks
  e.g.: esp32

12752 I think fixed that. Is it still the case?