Platformio-core: Add Support for local ("PC") unit tests

[neuhalje]

For CI and testing reasons I would really like to run unit tests. For performance reasons these should run on the local machine (PC), and not on the device (in contrast to #408).

I managed to hack together an external_script (see here) that compiles and runs a single cpp file as unit test.

Adding multiple test files does NOT work, bc the compilation of 'program' lumps together every testfile in one executable and the linker fails with duplicate symbols.

If someone with SCons skills can help me, we could extend platformio ci to actually run unit tests.

[malachib]

Although not exactly the same thing, have you looked into using travis.yml? Works well if your project is on github & open-source

[neuhalje]

@malachib you are right. In fact that is one of the use cases I am thinking of. The current platformio ci infrastructure only allows compilation, but no test execution. That last bit is missing.

[malachib]

Test execution of an embedded device is a very situation-specific thing. Even if Ivan put in full support for something like simavr, it would fall short of a complete suite. Or are you thinking of just IO-less testing which could all run on say an x86?

[ivankravets]

Guys, I hope that I will think something as for this issue.

[malachib]

Reviewing this, you seem to be wanting an x86-only test run situation. That seems fully feasible.

I do it with a distinct directory (test, alongside src) and custom makefiles that travis picks up, then use Catch.hpp unit test framework because it's light and easy with C++

I bet the same approach can work with pio+scons, however my scons skills are too nub-ish to make further recommendations

example: https://github.com/malachib/util.embedded

note: there is also a test.pio folder, but that is not automated testing and not relevant to our discussion, so ignore it

[neuhalje]

@malachib yes, the goal is to run tests on x86.

A dedicated makefile for tests is a good possibility. For starters I will borrow your Makefile :-)

Having worked with java build tools I came to enjoy automatic (test) dependency resolution.

[malachib]

Glad the makefile can provide some interim help !

Which tools did you use for your Java testing? I did a fair bit of Java back in the day, but have been a C# guy for the last 10 years

[neuhalje]

@malachib Mainly JUnit and Mockito. At times also Spock.

[ohhorob]

Is there any indication wether the testing configuration can handle using or emulating the same types as the target platform? Or is that just common, and independent of the target??

i.e. some variables I want to be uint8_t, and other uint32_t. Will the unit test compilation be able to ensure correctly sized integers/floats?

[ivankravets]

@ohhorob No emulation/cross-compilation. All tests will bu run on your machine using OS's GCC compiler.

[eerimoq]

I can't resist to promote Simba in this thread since it has the host PC test execution functionality already in place. I'm not saying you should use Simba, but instead I'm giving an example of how I work developing for embedded devices.

When I started the development of Simba RTOS I compiled and tested everything on my Linux PC. After all, the big job was to set the interfaces, not to execute the binary on an embedded device. Executing the binary on my Linux PC gave the possibility to use Linux tools, for example gdb, gprof, code coverage, etc. Very useful indeed!

Even today I usually implement and test everything on my Linux PC. When I'm satisfied I cross-compile the code and upload the binary to the embedded deivce.

Of course, certain functionality requires the embedded device, but a surprisingly big part does not.

Here is an example of how to build and run for two boards, linux and arduino_due:

   # Linux test compilation and execution (linux is the default board).
   $ make -s run
   <test soutput from linux stdout/stderr>

   # Arduino Due compilation and execution.
   $ make -s BOARD=arduino_due run
   <upload output>
   <test output from serial port from the embedded device>

The output from the test suite today is rather big, but if you have a PC with lots of RAM you can open it here: https://travis-ci.org/eerimoq/simba

Code coverage (press "src" at the bottom of the page): https://codecov.io/gh/eerimoq/simba

Simba project page on GitHub: https://github.com/eerimoq/simba

[ivankravets]

Docs: http://docs.platformio.org/en/latest/unit_testing.html

Examples: https://github.com/platformio/platformio-examples/tree/develop/unit-testing

[jerabaul29]

It is unclear for me how to perform this. The link to docs seems broken (get a missing page error). Do you know of any simple documentation / tutorial on how to do this? I have a project on Arduino Due where I tried to add to my platformio.ini:

[env:local]
platform = native
board = due
framework = arduino
build_flags = -std=gnu++17

But then I get:

> platformio test -e local
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

Processing tests_local in local environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Error: Unknown board ID 'due'
==================================================================================================== [FAILED] Took 0.27 seconds ====================================================================================================

Test         Environment    Status    Duration
-----------  -------------  --------  ------------
tests_local  local          FAILED    00:00:00.271
=============================================================================================== 1 failed, 0 succeeded in 00:00:00.271 ===============================================================================================

While my test is a simple TEST_ASSERT_EQUAL(32, 32); .

[jerabaul29]

Aaah, I think I understand: one cannot test locally some sources that include Arduino.h and their likes, right? May be good to explain it in some "docs for noobs" :) .