Mbed-os: Bug: host_tests are not picked up if the tests build location is outside the project tree

[screamerbg]

Description

While trying to shorten the build path and avoid the GCC-Windows path length limitations, I've tried to use a build path outside the project tree and stumbled upon a bug that impacts any test that uses (python) host test located in it's own dir. I've managed to narrow it down to mbedhtrun being executed without the -e flag, which is used to tell mbedhtrun where it should look for python host test plugins.

Steps to reproduce:

1. mbed import https://github.com/ARMmbed/smcc-tests - accessible to Arm and Mbed Partners (sorry)
2. cd smcc-tests
3. (works) mbed test -t ARM -m NUCLEO_F429ZI -n simple-mbed-cloud-client-tests-simple* -v which executes mbedgt --test-spec .\BUILD\tests\NUCLEO_F429ZI\ARM\test_spec.json -n simple-mbed-cloud-client-tests-simple* -V" in "D:Worksmcc-tests and produces:

mbedgt: running 1 test for platform 'NUCLEO_F429ZI' and toolchain 'ARM'
        use 1 instance of execution threads for testing
mbedgt: checking for 'host_tests' directory above image directory structure
        found 'host_tests' directory in: 'simple-mbed-cloud-client\TESTS\host_tests'
mbedgt: selecting test case observer...
        calling mbedhtrun: mbedhtrun -m NUCLEO_F429ZI -p COM13:115200 -f "BUILD/tests/NUCLEO_F429ZI/ARM/simple-mbed-cloud-client/TESTS/simple/connect/connect.bin" -e "simple-mbed-cloud-client\TESTS\host_tests" -d E: -c default -t 07960221004F660A3F0BF0A1 -r default -C 4 --sync 5

1. (fails) mbed test -t ARM -m NUCLEO_F429ZI -n simple-mbed-cloud-client-tests-simple* -v --build /build-arm/ which executes mbedgt --test-spec /build-arm/test_spec.json -n simple-mbed-cloud-client-tests-simple* -V" in D:Worksmcc-tests and produces the following output:

mbedgt: running 1 test for platform 'NUCLEO_F429ZI' and toolchain 'ARM'
        use 1 instance of execution threads for testing
mbedgt: checking for 'host_tests' directory above image directory structure
mbedgt: there was a problem while looking for host_tests directory
        level 2, path: ../../build-arm/simple-mbed-cloud-client/TESTS/simple/connect/connect.bin
        The following directories were not in the path: BUILD, .build
mbedgt: there was a problem while looking for host_tests directory
        level 3, path: ../../build-arm/simple-mbed-cloud-client/TESTS/simple/connect/connect.bin
        The following directories were not in the path: BUILD, .build
        'host_tests' directory not found: two directory levels above image path checked
mbedgt: selecting test case observer...
        calling mbedhtrun: mbedhtrun -m NUCLEO_F429ZI -p COM13:115200 -f "../../build-arm/simple-mbed-cloud-client/TESTS/simple/connect/connect.bin" -d E: -c default -t 07960221004F660A3F0BF0A1 -r default -C 4 --sync 5

Notice that the "calling" line is missing the -e flag to mbedhtrun and also that there a bunch of complaints by mbedgt that host_tests aren't found.

I'm suspecting that the problem is in Greentea, but perhaps the test_api logic inside Mbed OS should give a hint to Greentea where to look for tests rather than Greentea having to guess these based on the project structure.

CC @bridadan

Impacted toolchains

[x] ARM Compiler 5
[x] IAR 7.8
[x] GCC_ARM

Issue request type

[ ] Question
[ ] Enhancement
[x] Bug

[screamerbg]

Also note that this is preventing Pelion Ready validation for GCC ARM. CC @MarceloSalazar @janjongboom @bridadan @dhwalters423

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-94

[bridadan]

Hi @screamerbg, thanks for raising the issue and for providing the steps to reproduce. Just a heads up, the tip of smcc-tests is broken due to in correct simple-mbed-cloud-client.lib. I think you meant the contents of the file to be:

https://github.com/armmbed/simple-mbed-cloud-client/#ced91b98487bcdb646e871c412c592919b37cf00

instead of:

https://github.com/screamerbg/simple-mbed-cloud-client/#ced91b98487bcdb646e871c412c592919b37cf00

I was able to confirm the bug after this modification. You're suspicions were correct that this is an issue with greentea. Would you mind moving this issue to https://github.com/ARMmbed/greentea?

[screamerbg]

@bridadan Thanks for the heads up. I've just fixed it.

As for moving the issue to Greentea, as I wrote in my original post before @ciarmcom crippled it, "I'm suspecting that the problem is in Greentea, but perhaps the test_api logic inside Mbed OS should give a hint to Greentea where to look for tests rather than Greentea having to guess these based on the project structure."

[adbridge]

As for moving the issue to Greentea, as I wrote in my original post before @ciarmcom crippled it,

What exactly has ciarmcom supposed to have done? It will only correct formatting errors in the issue header such as when someone messes with the template when they are explicitly asked not to....If you stray outside of the template then all bets are off...

[0xc0170]

I reverted the edit, should be as it was. Sorry about the unintentional edit, it was fixed.