Mbed-os: Tests not possible with custom target

Created on 2 Jul 2018  路  25Comments  路  Source: ARMmbed/mbed-os

Description

I have stumbled over the follwoing problem:
When I declare a custom target in costum_targets.json the build process by using mbed compile works fine. But when I try to call mbed test the build fails due to the following error:

Traceback (most recent call last):
  File ".\mbed-os\tools\test_api.py", line 2195, in build_test_worker
    bin_file = build_project(*args, **kwargs)
  File ".\mbed-os\tools\build_api.py", line 554, in build_project
    app_config=app_config, build_profile=build_profile, ignore=ignore)
  File ".\mbed-os\tools\build_api.py", line 328, in prepare_toolchain
    config = config or Config(target, src_paths, app_config=app_config)
  File ".\mbed-os\tools\config\__init__.py", line 468, in __init__
    self.app_config_data.get("custom_targets", {}), tgt)
  File ".\mbed-os\tools\targets\__init__.py", line 127, in generate_py_target
    return target(name, total_data)
  File ".\mbed-os\tools\targets\__init__.py", line 105, in target
    resolution_order = get_resolution_order(json_data, name, [])
  File ".\mbed-os\tools\targets\__init__.py", line 97, in get_resolution_order
    parents = json_data[target_name].get("inherits", [])
KeyError: u'MY_TARGET'

In my opinion this error occurs because in File ".\mbed-os\tools\config\__init__.py", line 468, in __init__ self.app_config_data.get("custom_targets", {}), tgt) the function tries to find the target MY_TARGET (tgt) in the app_config json file, where it is just not present and according to the guidelines should not be present, I guess, because it is already defined in custom_targets.json. Could someone please look into this, since we really like to use the mbed test environment with a custom target without changing the mbed os repository.

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

closed_in_jira tools mirrored bug
Source
picture EmbedEngineer

All 25 comments

I believe the issue is that mbed-enabled dev boards are assigned unique ID numbers that make them automatically identifiable by the test system. Mbed-enabled boards also require a consistent way of resetting, flashing, and getting serial debug information out of the target. So it doesn't seem very possible to use the full native mbed test system at this time.

However, instead of using the full test system (greentea) you can just use the underlying tools that greentea uses. These include mbedhtrun, greentea-client, utest, and unity.

I have used the tools in this way and have had good results with unit testing. You may need to just write a simple custom script on top of these tools to automate the flashing, resetting, and pass/fail message parsing (if you want fully-automated unit testing).

You should take a look at the following repositories/tools used by the greentea test system:

Here's a (somewhat recent) guide on writing unit test apps with these tools.

AGlass0fMilk picture AGlass0fMilk on 2 Jul 2018

As an example, here's a unit test I wrote for a temperature/humidity sensor driver (Si7006). It shows a bunch of different test types (synchronous, asynchronous, tests with setups, tests with teardowns, etc).

The way I would run these tests is by flashing the target with this unit test app, then resetting it. By using a few command line flags with mbedhtrun you can run the unit tests without the full greentea server.

The flags I used are:

mbedhtrun --skip-flashing --skip-reset --port=/dev/ttyUSB0 --baud-rate=115200

You can run mbedhtrun --help to see the other possible flags. You should see pass/fail messages for each test case you add to the cases array.

Another hint is the ability to print out key value pairs using greentea_send_kv("Key", value);. The test script might complain but it's harmless and lets you log information during testing.

```/**

  • @file Si7006-Test.cpp
  • @brief Unit tests for the Si7006 Temperature/Humidity sensor
    *
  • @copyright Copyright © 2018 Heads Up Display, Inc.
    *
  • Created on: 3/31/2018
  • Author: gdbeckstein
    */

include "greentea-client/test_env.h"

include "utest/utest.h"

include "unity/unity.h"

include "drivers/I2C.h"

include "drivers/Timer.h"

//#include "events/EventQueue.h"

include "devices/Si7006.h"

include "PinDefinitions.h"

using namespace utest::v1;

/** Test class to access protected member functions */
class Si7006Test : public hud::Si7006
{

public:

    typedef Measurement_t MeasurementTest_t;

public:

    Si7006Test(mbed::I2C* i2c) : Si7006(i2c) { }

    void big_to_little_endian_wrapper(MeasurementTest_t* measurement)
    {
        big_to_little_endian(measurement);
    }

    bool verify_checksum_wrapper(MeasurementTest_t measurement)
    {
        return verify_checksum(measurement);
    }

    static uint8_t compute_crc8_wrapper(const uint8_t* bytes, const uint8_t length)
    {
        return compute_crc8(bytes, length);
    }

    static float convert_raw_temp_wrapper(uint16_t raw_temp)
    {
        return convert_raw_temperature_to_C(raw_temp);
    }

    static float convert_raw_humidity_wrapper(uint16_t raw_humidity)
    {
        return convert_raw_humidity_to_percent(raw_humidity);
    }

};

mbed::I2C i2c(I2C_SDA_PIN, I2C_SCL_PIN);
mbed::Timer timer;

//events::EventQueue queue;

Si7006Test si7006(&i2c);

void test_endianness_conversion(void)
{
Si7006Test::MeasurementTest_t expected;
Si7006Test::MeasurementTest_t actual;

expected.temperature = 0xABCD;
actual.temperature = 0xCDAB;

si7006.big_to_little_endian_wrapper(&actual);

// Print out the data
greentea_send_kv("Expected", expected.temperature);
greentea_send_kv("Actual", actual.temperature);

TEST_ASSERT_EQUAL_HEX16(expected.temperature, actual.temperature);

}

void test_checksum_calculation(void)
{
timer.reset();
uint16_t input_data = 0xB238;
uint8_t expected_crc = 0x58;
uint8_t actual_crc;

timer.start();
// Test the crc8 computation
actual_crc = si7006.compute_crc8_wrapper((uint8_t*) &input_data, 2);
timer.stop();
TEST_ASSERT_EQUAL_HEX8(expected_crc, actual_crc);

greentea_send_kv("ExecutionTime_us", timer.read_high_resolution_us());
greentea_send_kv("Expected CRC", expected_crc);
greentea_send_kv("Actual CRC", actual_crc);

// When you compute the crc of data with it's own CRC
// You should get 0 out as the CRC8
timer.reset();
uint8_t input_string[3] = { 0x68, 0xD4, 0xA2 };
expected_crc = 0x00;
timer.start();
actual_crc = si7006.compute_crc8_wrapper(input_string, 3);
timer.stop();
greentea_send_kv("ExecutionTime_us", timer.read_high_resolution_us());
TEST_ASSERT_EQUAL_HEX8(expected_crc, actual_crc);
greentea_send_kv("Expected CRC", expected_crc);
greentea_send_kv("Actual CRC", actual_crc);

input_data = 0xD468;
expected_crc = 0xA2;
Si7006Test::MeasurementTest_t measurement;
measurement.temperature = input_data;
measurement.checksum = expected_crc;
TEST_ASSERT_TRUE(si7006.verify_checksum_wrapper(measurement));

}

void test_raw_temp_conversion(void)
{
uint16_t input_raw_temp = 17952;

// The temperature today (4/2/18) in Buffalo (C)...
float expected = 1.28f;
float actual;

actual = Si7006Test::convert_raw_temp_wrapper(input_raw_temp);

TEST_ASSERT_FLOAT_WITHIN(0.1f, expected, actual);

// Test negative temperatures
input_raw_temp = 16316;
expected = -3.1f;
actual = Si7006Test::convert_raw_temp_wrapper(input_raw_temp);
TEST_ASSERT_FLOAT_WITHIN(0.1f, expected, actual);

}

void test_raw_RH_conversion(void)
{
uint16_t input_raw_RH = 17460;
float expected = 27.3f;
float actual;

actual = Si7006Test::convert_raw_humidity_wrapper(input_raw_RH);

TEST_ASSERT_FLOAT_WITHIN(0.1f, expected, actual);

}

void test_manufacturer_info(void)
{
// Get the serial number
uint64_t serial_number;
si7006.get_serial_number(&serial_number);

uint32_t* sn_pointer = (uint32_t*) &serial_number;

greentea_send_kv("SerialNumberUpper", sn_pointer[1]);
greentea_send_kv("SerialNumberLower", sn_pointer[0]);

// The 4th LSB should be 0x06 (indicates Si7006)
TEST_ASSERT_EQUAL_HEX32((Si7006_SERIAL_NUMBER_ID_BYTE << 24), (sn_pointer[0] & 0xFF000000));

// Get the firmware revision
uint8_t firmware_rev;

si7006.get_firmware_revision(&firmware_rev);

// This could be v1 (0xFF) or v2 (0x20)
greentea_send_kv("FirmwareRev", firmware_rev);

}

void test_set_and_get_heater_current(void)
{
uint8_t expected = 0x0E;
uint8_t actual;

si7006.set_heater_current(expected);
si7006.get_heater_current(&actual);
TEST_ASSERT_EQUAL_HEX8(expected, actual);

}

void test_set_and_get_resolution(void)
{
hud::Si7006::Resolution_t actual_resolution;
hud::Si7006::Resolution_t expected_resolution;

expected_resolution = hud::Si7006::RESOLUTION_RH_10BIT_TEMP_13BIT;
si7006.set_resolution(expected_resolution);
si7006.get_resolution(&actual_resolution);
TEST_ASSERT_EQUAL(expected_resolution, actual_resolution);

expected_resolution = hud::Si7006::RESOLUTION_RH_11BIT_TEMP_11BIT;
si7006.set_resolution(expected_resolution);
si7006.get_resolution(&actual_resolution);
TEST_ASSERT_EQUAL(expected_resolution, actual_resolution);

expected_resolution = hud::Si7006::RESOLUTION_RH_12BIT_TEMP_14BIT;
si7006.set_resolution(expected_resolution);
si7006.get_resolution(&actual_resolution);
TEST_ASSERT_EQUAL(expected_resolution, actual_resolution);

expected_resolution = hud::Si7006::RESOLUTION_RH_8BIT_TEMP_12BIT;
si7006.set_resolution(expected_resolution);
si7006.get_resolution(&actual_resolution);
TEST_ASSERT_EQUAL(expected_resolution, actual_resolution);

}

status_t test_setup_read_data(const Case *const source, const size_t index_of_case)
{
// Reset the ALS to default state
si7006.reset();

// Reset the timer
timer.reset();

// Startup wait
wait_ms(20);

return greentea_case_setup_handler(source, index_of_case);

}

// Block read test
void test_read_data(void)
{
float temp, humidity;

// Time the transaction
timer.start();

// Make sure no errors happened during read
TEST_ASSERT_EQUAL(0, si7006.read(&temp, &humidity));

timer.stop();

greentea_send_kv("ExecutionTime_us", timer.read_us());

greentea_send_kv("Temperature", (int)(temp*1000));
greentea_send_kv("Humidity", (int)(humidity*1000));

// Check for valid data
TEST_ASSERT_TRUE_MESSAGE((-5.0f < temp) && (temp < 100.0f), "Is it really hot/cold?");
TEST_ASSERT_TRUE_MESSAGE((0.0f <= humidity) && (humidity <= 100.0f), "What\'s with this humidity?");

}

status_t test_teardown_asynch_read(const Case *const source, const size_t passed,
const size_t failed, const failure_t reason)
{
greentea_send_kv("ExecutionTime_us", timer.read_us());

return greentea_case_teardown_handler(source, passed, failed, reason);

}

void asynch_read_temp_handler(hud::Si7006::Error_t error, float temp)
{

timer.stop();

greentea_send_kv("Temperature", (int)(temp*1000));

// Make sure no errors happened
TEST_ASSERT_TRUE(error == hud::Si7006::NO_ERROR);

// Check for valid data
TEST_ASSERT_TRUE((-5.0f < temp) && (temp < 100.0f));
//TEST_ASSERT_TRUE((0.0f <= humidity) && (humidity <= 100.0f));

Harness::validate_callback();

}

// Tests the asynchronous read temp functionality
control_t test_asynch_read_temp_data(void)
{
// Start the timer
timer.start();

si7006.read_temperature_asynch(asynch_read_temp_handler);

// Timeout test case after 200ms
return CaseTimeout(200);

}

void asynch_read_humidity_handler(hud::Si7006::Error_t error, float humidity)
{

timer.stop();

greentea_send_kv("Humidity", (int)(humidity*1000));

// Make sure no errors happened
TEST_ASSERT_TRUE(error == hud::Si7006::NO_ERROR);

// Check for valid data
//TEST_ASSERT_TRUE((-5.0f < temp) && (temp < 100.0f));
TEST_ASSERT_TRUE((0.0f <= humidity) && (humidity <= 100.0f));

Harness::validate_callback();

}

// Tests the asynchronous read temp functionality
control_t test_asynch_read_humidity_data(void)
{
// Start the timer
timer.start();

si7006.read_humidity_asynch(asynch_read_humidity_handler);

// Timeout test case after 200ms
return CaseTimeout(200);

}

// Turns the heater on, waits a few moments and measures the temperature delta
void test_self_heating(void)
{

}

// Custom setup handler required for proper Greentea support
status_t greentea_setup(const size_t number_of_cases) {
GREENTEA_SETUP(20, "default_auto");

// Reset the LTR329ALS
//als.reset();
//wait_ms(100);

// Call the default reporting function
return greentea_test_setup_handler(number_of_cases);

}

// Specify all your test cases here
Case cases[] = {
Case("Endianness Conversion Test", test_endianness_conversion),
Case("Checksum Calculation Test", test_checksum_calculation),
Case("Raw Temperature Conversion Test", test_raw_temp_conversion),
Case("Raw Rel Humidity Conversion Test", test_raw_RH_conversion),
Case("Manufacturer Info Test", test_manufacturer_info),
Case("Set and Get Heater Current Test", test_set_and_get_heater_current),
Case("Set and Get Resolution Test", test_set_and_get_resolution),
Case("Blocking Read Data", test_setup_read_data, test_read_data),
Case("Asynchronous Read Humidity Data", test_setup_read_data, test_asynch_read_humidity_data, test_teardown_asynch_read),
Case("Asynchronous Read Temperature Data", test_setup_read_data, test_asynch_read_temp_data, test_teardown_asynch_read),
//Case("Self Heating Test", test_setup_read_data, test_self_heating)
};

// Declare your test specification with a custom setup handler
Specification specification(greentea_setup, cases);

int main(void)
{
Harness::run(specification);
return 0;
}
```

AGlass0fMilk picture AGlass0fMilk on 2 Jul 2018
👍1

I think we've been discussing this in Fail to run Greentea test on inherited target #6342.

mattbrown015 picture mattbrown015 on 2 Jul 2018

ARM Internal Ref: MBOTRIAGE-1033

ciarmcom picture ciarmcom on 2 Jul 2018

@AGlass0fMilk Thank you for your respsonse, but that is actually not what I was looking for. Mbed has the capability to mock targets. So if you use the standard onboard mbed enabled debugger with the appropriate mcu and use the mock ability from mbed ls combining the mbed dev board with your own custom target (which is infact nothing other than the mbed dev board with custom pin configuration) the tests should compile with it none the less. But at the moment they don't, so that's the issue.

EmbedEngineer picture EmbedEngineer on 3 Jul 2018

@mattbrown015 thank you for linking this to the other issue

EmbedEngineer picture EmbedEngineer on 3 Jul 2018

@EmbedEngineer I cannot reproduce this on master.

I added a custom_targets.json with:

{
    "MY_TARGET": {
        "inherits": ["NUCLEO_F411RE"]
    }
}

and ran mbed test --compile -t arm -m MY_TARGET and it built tests.

theotherjimmy picture theotherjimmy on 3 Jul 2018

@theotherjimmy I tried to recreate it with your custom_targets.json: doing to following steps:

  1. running mbed import https://github.com/ARMmbed/mbed-os-example-blinky
  2. modifiying mbed-os.lib to point at the master (https://github.com/ARMmbed/mbed-os/)
  3. running mbed deploy to update the repository
  4. creating the costum_targets.json with your target
  5. running mbed test --compile -t GCC_ARM -m MY_TARGET

And I still get the same error. So am I doing something wrong here?

EmbedEngineer picture EmbedEngineer on 4 Jul 2018

@EmbedEngineer You have to get the name correct

costum_targets.json

It's custom_targets.json

Assuming that's a typo, where did you put the custom_targets.json? It must be in the project root.

theotherjimmy picture theotherjimmy on 5 Jul 2018

Also take a look at https://github.com/ARMmbed/greentea/issues/256, we're able to program using mbed test via J-Link, there are some instructions at the bottom of that issue that might help you @EmbedEngineer

Alex-EEE picture Alex-EEE on 6 Jul 2018

@theotherjimmy Yeah this was just a typo in my post, i put the custom_targets.json in the root project folder so my test project dir tree looks like this:
image
My custom_targets.json:

{
    "MY_TARGET": {
        "inherits": ["NUCLEO_F411RE"]
    }
}

mbed compile -t GCC_ARM -m MY_TARGET also works fine. What OS are you compiling on? I am using a Windows 10 platform. So maybe there is the problem?
Or do I have to add somethind to the mbed_app.json since self.app_config_data.get("custom_targets", {}), tgt) in mbed-os\tools\config\__init__.py (where the error occurs) is clearly trying to find the costum targets in it?

EmbedEngineer picture EmbedEngineer on 6 Jul 2018

@theotherjimmy So i tried the following:
I modified the source code in mbed-os\tools\config\__init__.py int the function def __init__(self, tgt, top_level_dirs=None, app_config=None): on Line 460 onwards from:

        if isinstance(tgt, Target):
            self.target = tgt
        else:
            if tgt in TARGET_MAP:
                self.target = TARGET_MAP[tgt]
            else:
                self.target = generate_py_target(
                    self.app_config_data.get("custom_targets", {}), tgt)

to

        if isinstance(tgt, Target):
            self.target = tgt
        else:
            if tgt in TARGET_MAP:
                print("Target found in TARGET_MAP")
                self.target = TARGET_MAP[tgt]
            else:
                print("Target NOT found in TARGET_MAP")
                self.target = generate_py_target(
                    self.app_config_data.get("custom_targets", {}), tgt)

So I just wanted to know if my costum target MY_TARGET is found in the TARGET_MAP which should be generated by the mbed tools.
And i got the following output:

> mbed test --compile -t GCC_ARM -m MY_TARGET -j 1
Target found in TARGET_MAP
Scan: .
Scan: TESTS
Scan: TESTS
Scan: TESTS
Scan: TESTS
Scan: TESTS
Scan: TESTS
Scan: TESTS
Scan: TESTS
Target found in TARGET_MAP
Building library mbed-build (MY_TARGET, GCC_ARM)
Scan: mbed-os-example-blinky
Target found in TARGET_MAP
Scan: GCC_ARM
Target NOT found in TARGET_MAP
Traceback (most recent call last):
  File "test\mbed-os-example-blinky\mbed-os\tools\test_api.py", line 2195, in build_test_worker
    bin_file = build_project(*args, **kwargs)
  File "test\mbed-os-example-blinky\mbed-os\tools\build_api.py", line 554, in build_project
    app_config=app_config, build_profile=build_profile, ignore=ignore)
  File "test\mbed-os-example-blinky\mbed-os\tools\build_api.py", line 328, in prepare_toolchain
    config = config or Config(target, src_paths, app_config=app_config)
  File "test\mbed-os-example-blinky\mbed-os\tools\config\__init__.py", line 469, in __init__
    self.app_config_data.get("custom_targets", {}), tgt)
  File "test\mbed-os-example-blinky\mbed-os\tools\targets\__init__.py", line 122, in generate_py_target
    return target(name, total_data)
  File "test\mbed-os-example-blinky\mbed-os\tools\targets\__init__.py", line 101, in target
    resolution_order = get_resolution_order(json_data, name, [])
  File "test\mbed-os-example-blinky\mbed-os\tools\targets\__init__.py", line 93, in get_resolution_order
    parents = json_data[target_name].get("inherits", [])
KeyError: u'MY_TARGET'

So I guess when scanning the GCC_ARM properties the target map TARGET_MAP is generated without the custom targets (it is generated in the correct way before) and so the script tries to find them in the mbed_app.json config. I think that is the error, i don't know if this also happens when just using the ARM and not the GCC_ARMcompiler

EmbedEngineer picture EmbedEngineer on 6 Jul 2018

@EmbedEngineer why did you come to the conclusion that it only affects GCC_ARM? I don't follow.

theotherjimmy picture theotherjimmy on 6 Jul 2018

Thanks for spending time to track down what's going wrong. Sorry this has been very difficult for me to reproduce.

theotherjimmy picture theotherjimmy on 6 Jul 2018

@theotherjimmy Well yeah no problem.
I tracked the problem a little bit deeper. It seems that the program looses its TARGET_MAPwhen the build_test_worker(*args, **kwargs) in test_api.py from build_tests results.append(p.apply_async(build_test_worker, args, kwargs)) in the same file in Line 2277 is called. I noticed that after this call the update_target_data() function in targets/__init__.py is called and, since the extract_mcus(parser, options) function from options.py isn't called prior to it, afterwards the TARGET_MAP does not hold the custom targets any more. Why this problem occurs, I unfortunatly don't know, as I am not a real python specialist.

EmbedEngineer picture EmbedEngineer on 9 Jul 2018

@EmbedEngineer Wow! Thanks for digging in that far, it's really helpful. As a workaround, does -j 1 solve the issue? (I'm skeptical that it would).

It's looking like we have to pass in the target as a target object instead.

theotherjimmy picture theotherjimmy on 9 Jul 2018

Further, this implies that the library build succeeds. Could you confirm that @EmbedEngineer?

theotherjimmy picture theotherjimmy on 9 Jul 2018

@theotherjimmy
To your first post: no unfortunatly I already tried the -j 1 option and it doesn't change a thing.
Concerning your second post: yes the library seems to be built correctly.

EmbedEngineer picture EmbedEngineer on 9 Jul 2018

That's a good start. ~It's very odd that update_target_data is called.~ OH! it's the next python process importing the same module (I'm on linux, so I don't actually see that bit). The weird bit is then that it's bothering to lookup the target in TARGET_MAP upon build. It should not be doing that.

theotherjimmy picture theotherjimmy on 9 Jul 2018

This patch:

diff --git a/tools/test_api.py b/tools/test_api.py
index 8f46cdac9..5b569f2f3 100644
--- a/tools/test_api.py
+++ b/tools/test_api.py
@@ -2225,7 +2225,11 @@ def build_tests(tests, base_source_paths, build_path, target, toolchain_name,
     execution_directory = "."
     base_path = norm_relative_path(build_path, execution_directory)

-    target_name = target.name if isinstance(target, Target) else target
+    if isinstance(target, Target):
+        target_name
+    else:
+        target_name = target
+        target = TARGET_MAP[target_name]
     cfg, _, _ = get_config(base_source_paths, target_name, toolchain_name, app_config=app_config)

     baud_rate = 9600
@@ -2255,7 +2259,7 @@ def build_tests(tests, base_source_paths, build_path, target, toolchain_name,
         bin_file = None
         test_case_folder_name = os.path.basename(test_paths[0])

-        args = (src_paths, test_build_path, target, toolchain_name)
+        args = (src_paths, test_build_path, deepcopy(target), toolchain_name)
         kwargs = {
             'jobs': 1,
             'clean': clean,

does not break it for me, and may fix it for you.

theotherjimmy picture theotherjimmy on 9 Jul 2018

This should prevent the TARGET_MAP lookup inside each process.

theotherjimmy picture theotherjimmy on 9 Jul 2018

I also put that patch up on a branch here: https://github.com/theotherjimmy/mbed/tree/fix-customtgt-tests

theotherjimmy picture theotherjimmy on 9 Jul 2018

@cmonr Since I suspect that this is a windows-only issue, I would appreciate it if you could reproduce it.

theotherjimmy picture theotherjimmy on 10 Jul 2018

@theotherjimmy Thank you so much. It is now working fine for me.

EmbedEngineer picture EmbedEngineer on 11 Jul 2018
🎉1

Thanks for testing that patch. I'll make the PR now.

theotherjimmy picture theotherjimmy on 11 Jul 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

MarceloSalazar picture MarceloSalazar  路  3Comments
bulislaw picture bulislaw  路  3Comments
1domen1 picture 1domen1  路  3Comments
drahnr picture drahnr  路  4Comments
hasnainvirk picture hasnainvirk  路  3Comments