Catch2: target embedded devices?

[m-mcgowan]

I work with embedded devices. I've started using Catch to write tests that verify the hardware independent C++ code in my project, compiled using gcc.

It would be super awesome to be able to run the same tests on the embedded device also. On the embedded device (a STM32) there's about 40K of free program memory to play with, so not much, but the ARM gcc compiler does a good job of optimizing code. Are there features in Catch that could be made compile-time optional to squeeze Catch into this constrained environment? Also since there is no stdout could a global stream be made available for all output, so embedded programs can direct output elsewhere (e.g. over a TCP socket or over a serial connection) by setting that global output stream?

There are existing test frameworks for embedded, such as Arduino Unit, but this uses a different syntax to Catch and so means keeping separate source code for on device tests vs off device tests.

Embedded devices are already in the billions and will continue to grow over the coming years.
I hope this possibility gives food for thought - looking forward to an engaging conversation!

[martinmoene]

Apart from Catch's development you may want to have a look at my lest test framework which is inspired on Catch. I've just picked-up development again.

lest started as an exercise to see 1) how a simple C++11 test framework may look like and 2) how Catch's expression decomposition works.

Currently there are three lest C++11 variants and a C++03 version.

The intention is to be able to toggle larger features on and off, e.g. Catch option --duration via lest_FEATURE_TIME_DURATION (soon to be implemented).

lest does have less features than Catch naturally: specifically SECTIONs are not available.

[philsquared]

Lest might be a good option if you want something Catch-like but lighter weight.

As for making some features compile-time optional - that's certainly a possibility. I have a long term arc where I keep trying to move towards making whole sections of Catch more modular (and therefore, potentially, optional). A couple of parts that I'm considering taking out of the core are: generators and matchers (neither of which are fully fledged yet anyway). I don't know how much that will help but we could possibly look at other areas too.
I'd be keen to have Catch usable in the embedded space. It's a few years since I worked in a truly embedded environment but my impression at the time was that even those projects that had moved to C++ were still working with such a limited subset of C++ (both through compiler limitations as well as project preferences) that something like Catch would never have been workable (most projects I worked on didn't even have the std library!). That may have changed (or may have been specific to the segments I worked in). I don't have a feel for how addressable a market embedded C++ is for Catch these days.

You also mentioned stdout. Catch allows you to supply a custom ostream to use instead of std::cout. Actually it just ties the cout buffer to the one you supply - so it still needs std::cout to be available, at least.
Without knowing more about what a C++ without stdout looks like I'm not sure how feasible it would be to support it. Is std::cout available - and if so, what does it do?

Finally: do you work with simulators and/ or emulators. I used to do unit testing almost exclusively in those and never deploy the tests to the device itself (because it was so limited). That's not to say that no code-level testing took place on-device - but it tended to be much more specialised, and involving instrumented builds etc. I even work that way primarily when I do iOS development nowadays (even though it's not really "embedded" anymore).

[m-mcgowan]

Thanks for the extensive reply!

The compiler we are using is gcc 4.8.4 with the ARM backend, so C++11 is fully supported. (yay!)

Attempting to use cin/cout results in linker errors - open, lseek, read, write, isatty, fstat, close which aren't present on the embedded platform. Some way to override cin/cout at compile time is needed so that none of the console functions are linked. As a hack, I would just use a macro to redefine cin/cout to point to local declarations so that the standard libraries don't pull in the stdio bits, but there may be a more elegant way to reach the goal!

At present, we use neither simulators or emulators. For our unit tests, the SUT code is made sufficiently hardware abstracted (hardware functions are external and injected/mocked where necessary) so the test suite can be compiled on regular gcc. We have some integration tests that run on device, which are basically the previous unit tests rewritten to run on the device (hence, integrating the test with the hardware). I have to rewrite the tests to run on-device since it's using a different test framework. Fortunately, we are just at the start of applying TDD to the firmware so I've not had to rewrite much code, just a few tests as PoC. But it's a big enough pain that I know I want to avoid it as the test suite grows! Having Catch on the embedded device would allow test code to be cross-compilable and run both on and off device.

[philsquared]

Ah, I feared as much. That makes it a bit more work to address. Not impossible - but not something I can just throw in.
That said what about std::fstream - is that supported (or std::ostream in general)?

[m-mcgowan]

Yes, the stream library is supported. The main issue is with the default binding of stdout. Although I can easily implement the missing functions (in fact, someone has already done that and pushed stdout over a serial port connection.)

If I solve that then the next problem would be code size, and providing a different entry point than main(), e.g. programmatic configuration (e.g. reporter selection, test name etc..) With static configuration the linker should be able to remove unused reporters, and other features not explicitly called into play.

[philsquared]

Hi @m-mcgowan,

build 6 on develop now allows you to conditionally compile cout/ cerr.
What I have done is replaced all calls to std::cout in Catch with calls to Catch::cout(). I've then implemented Catch::cout() to just return std::cout - but wrapped the implementation in #ifndef CATCH_CONFIG_NOSTDOUT (and the same with cerr).

The upshot is that if you #define CATCH_CONFIG_NOSTDOUT it will not use std::cout or std::cert _but_ you must implement Catch::cout() and Catch::cerr() yourself (to return a std::ostream&).

Does that address this particular need for you?

[jweyrich]

Nice solution @philsquared! :+1: Maybe Catch could provide a std::ostream named cnull and use it when CATCH_CONFIG_NOSTDOUT is defined? Or it would be totally out of scope / non-sense? Not a big task for the user anyway.

For example:

std::ostream cnull(NULL);
std::wostream wcnull(NULL);

According to the standard, these streams will ignore any writes since their streambufs are set to null pointers. I did find a related Q&A on StackOverflow where someone cared to cite the relevant parts from the standard.

A possible implementation:

    std::ostream& cout() {
#ifdef CATCH_CONFIG_NOSTDOUT
        static std::ostream cnull(NULL);
        return cnull;
#else
        return std::cout;
#endif
    }
    std::ostream& cerr() {
#ifdef CATCH_CONFIG_NOSTDOUT
        static std::ostream cnull(NULL);
        return cnull;
#else
        return std::cerr;
#endif
    }

[philsquared]

I did consider something along those lines (although I didn't know you could null-construct the ostream - thanks for the SO link there).
However I was thinking (a) you _are_ going to want to do something with the output and (b) if it silently swallows all output it may not be obvious what's going on - or what you can do about it.

[m-mcgowan]

Thanks for adding conditional compilation of stdout! I'll take it for a spin this week and let you know!

[m-mcgowan]

Hi, I'm sure this helped a lot. However, when I tried this today, still got linker errors. There must be something pulling in a standard output function in libc. (A SSCCE where I just throw an exception is enough to trigger this, so the cpp runtime is doing something with exceptions that requires file access.) Fixing up the linker errors by defining _write, _lseek etc.. allows it to build, but the code is presently too large by some 100k to fit into memory.

I will later be moving to larger hardware that has more memory, so this is a transient issue. In the meantime I will take a look at what large functions have been introduced to see what causes the program size to increase.

Thanks again for for the work you did and making the goal one step closer!

[Trass3r]

Any news on this, @m-mcgowan?

[philsquared]

Reviewing old tickets: It looks like the original issue, here, was potentially resolved - but subsequent issues arose (and it's not clear, now, what the status of those is).
There's a whole set of potential issues when working in the embedded space and it's going to be an ongoing effort to raise specific issues and work through them.

So I'm marking this for closure soon unless someone (@m-mcgowan?) has more to add here?
For other issues relating to embedded targets please raise further tickets as appropriate.

[rgrover]

I am keen to use Catch for my project based on a ARM Cortex-M with only 256K of flash (code) space. Even though Catch allows me to replace std::cout etc with my own lightweight streams/stream-buffers, it still requires me to work with std::ostream. Ostream pulls in a huge amount of locale, time... related support in IOStreams. This is non-essential and bloats my test application to a size beyond the available flash--this in spite of using reasonable compiler/linker flags for optimizing space and removing the unnecessary bits.

The following report contains a table of symbol-sizes and symbol-names (in decreasing order of symbol-size) for the code that gets pulled into the binary as a result of introducing Catch:
report1.txt
It was generated using

nm --demangle --print-size --size-sort --reverse-sort --radix d

Here are a few items from the top of the list (first column is size [in decimal], second column is symbol-type [according to nm]):

00001812 T std::locale::_Impl::_Impl(unsigned int)
00001812 T std::locale::_Impl::_Impl(unsigned int)
00001484 W std::__cxx11::time_get<...>::_M_extract_via_format(...) const
00001480 W std::__cxx11::time_get<...>::_M_extract_via_format(...) const
00001064 T std::locale::facet::_M_sso_shim(std::locale::id const*) const
...

A lot of these are un-necessary and come un-invited as a result of depending on iostreams.

I'd love to use Catch on my embedded device, and advocate its use to my colleagues.

[Trass3r]

Even though Catch allows me to replace std::cout etc with my own lightweight streams/stream-buffers, it still requires me to work with std::ostream. Ostream pulls in a huge amount of locale, time... related support in IOStreams.

Thanks for clarifying that it is iostream striking again. Maybe doctest could work in an embedded setting?

[BitFis]

@rgrover Whats the current state of this, did you kept trying to find a way or did you stop?