Mbed-os: FUTURE_SEQUANA: failing make nightly exporters

[0xc0170]

Description

Logs here http://mbed-os-ci.s3-website-eu-west-1.amazonaws.com/?prefix=jenkins-ci/artifacts/master/107/exporter/FAIL/

Check make_ exporters. srcat is failing in the last step.

One of these failures:

srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500000 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500001 value
    (0x05)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500002 value
    (0xE2)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500003 value
    (0x07)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500004 value
    (0x21)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500005 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500006 value
    (0x21)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500007 value
    (0x01)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500008 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500009 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: contradictory 0x9050000A value
    (previous = 0x45, this one = 0x00)
make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2

or

Finished: 0 information, 9 warning and 0 error messages.
srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory

@orenc17 @alzix Who could help fixing these?

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

[orenc17]

@0xc0170 i believe @lrusinowicz could help

[0xc0170]

This is affecting our nightly, it would be good to have it fixed soon

[lrusinowicz]

@0xc0170
Martin, is this with build on the master/HEAD?
I believe I have a potential fix, but how can I test it helps?

[0xc0170]

Correct, master HEAD is being tested. you should be able to access the logs. Use mbed-os-example-blinky and export for failing exporters, to reproduce the errors

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-832

[0xc0170]

Fixed via referenced PR

[0xc0170]

@lrusinowicz I've fetched latest nightly results and we can still seeing this failing for ARMC5 on master

Finished: 0 information, 9 warning and 0 error messages.
srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory
make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
make: *** [all] Error 2

This is the failing example Exporting mbed-os-example-blinky FUTURE_SEQUANA make_armc5. Can you please review and reproduce?

[0xc0170]

The log should be accessible here http://mbed-os-ci.s3-eu-west-1.amazonaws.com/jenkins-ci/ARMmbed/mbed-os/mbed-os-ci-nightly/artifacts/master/126/exporter/FAIL/make_armc5/FUTURE_SEQUANA/exporter_build_log_FUTURE_SEQUANA_make_armc5.tar.gz

Failures are for targets FUTURE_SEQUANA and FUTURE_SEQUANA_PSA

[lrusinowicz]

@0xc0170
The log content is not giving any useful hints on why this might be failing.
Does this export test succeed on any other PSOC6 target, i.e. is a failure specific to FUTURE_SEQUANA target?
What is the environment (operating system, make version) used on your cloud servers?
I actually can't reproduce the exact issue at the moment. I don't have access to ARM compiler on my Linux system and the exported makefile doesn't work on Windows 7 with GNU make versions 4.1/4.2 because of different issue (command line being evidently truncated on generation of .link_options.txt

[0xc0170]

What is the environment (operating system, make version) used on your cloud servers?

The job run on linux node. I cant find make being printed in the logs.
The command python -u ../mbed-os/tools/test/examples/examples.py export make_armc5 --mcu FUTURE_SEQUANA_PSA .

From the log what I can read is that srec_cat fails, linking is fine

03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] Finished: 0 information, 9 warning and 0 error messages.
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] make: *** [all] Error 2
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] FAILURE
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] FAILURE building mbed-os-example-blinky FUTURE_SEQUANA_PSA make_armc5
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] # Examples compilation summary
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] #
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] # Failed build combinations
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] # mbed-os-example-blinky FUTURE_SEQUANA_PSA make_armc5
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] #
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] Number of failures = 1

Does this export test succeed on any other PSOC6 target, i.e. is a failure specific to FUTURE_SEQUANA target?

Only FUTURE_SEQUANA and FUTURE_SEQUANA_PSA are affected.

[0xc0170]

@ARMmbed/mbed-os-test Need your help to get more details for this nightly failures

[cmonr]

exporter_build_log_FUTURE_SEQUANA_make_armc5.log

The log, but extracted.

[cmonr]

Finished: 0 information, 9 warning and 0 error messages.
srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory
make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
make: *** [all] Error 2
FAILURE
FAILURE building mbed-os-example-blinky FUTURE_SEQUANA make_armc5

I recall a specific version of srec_cat being needed. @lrusinowicz does that sound accurate?
@studavekar Does that sound famililar?

@ARMmbed/mbed-os-test Fyi

All: Something that appears related: https://github.com/ARMmbed/mbed-os/pull/8491#issuecomment-435430183

[studavekar]

I recall a specific version of srec_cat being needed. @lrusinowicz does that sound accurate?
@studavekar Does that sound famililar?

Yeah, NRF devices are passing nightly even they use scre_cat. maybe its something else.

[cmonr]

@ARMmbed/mbed-os-test Are we able to tell what versions of python modules are in use in nightly?

[cmonr]

Whoops, looks like this is a native shell utility.

@ARMmbed/mbed-os-test-team Any way we can run srec_cat --version in the nightly VM?

[lrusinowicz]

Do you also test exporter for make_gcc_arm? If yes, then does it work?

I was testing make_gcc_arm exporter on Linux locally (don't have ARMCC on Linux) and the resulting makefile is working correctly, including building a combined image with srec_cat.
The combined image section (and resulting srec_cat command) seems to be identical for both, GCC and ARMCC makefiles.
Yes, I also recall we had some issue with srec_cat version before, but I cannot find any details.
I have srec_cat version 1.58.D001 and GNU make 3.81 on my Linux system.

[0xc0170]

We do also make gcc arm, also failing, latest nightly on 5.12 branch can be viewed here http://mbed-os-ci.s3-website-eu-west-1.amazonaws.com/?prefix=jenkins-ci/ARMmbed/mbed-os/mbed-os-ci/mbed-os-5.12/artifacts/mbed-os-5.12/2/exporter/FAIL/

[0xc0170]

@ARMmbed/mbed-os-test-team Any way we can run srec_cat --version in the nightly VM?

@alekla01 @timurh01 Please help us why is srec_cat failing in the CI.

This looks like specific to FUTURE_SEQUANA boards as other targets that use srec_cat are working.

[lrusinowicz]

@0xc0170
I see two, separate issues in the build logs:
_1. gcc/FUTURE_SEQUANA_PSA_
srec_cat fails, because re-generated PSA M0 images contain special addresses starting with 0x9**, conflicting with the same addresses generated for M4 image. These can be just manually cut-out from M0 hex files, but I will submit a change, that should stop generation of these addresses in the first place.
_2. armc5/FUTURE_SEQUANA (original issue)_
This one is failing, because somehow BUILD directory structure is non-consistent with makefile srec_cat command. I will need to do more analysis here. But this is somewhat difficult, as I do not have possibility to reproduce this on Linux and on Windows this fails for different reasons (too long paths, if I remember correctly).

[0xc0170]

Thanks for the update and looking at these logs. The fixes should target 5.12rc2 (reference this issue there so we are aware of the proper release target).

[lrusinowicz]

@0xc0170
Strange build failure in armc5 builds was caused by specific behaviour of ARM fromelf utility and the option used in exported makefile --i32. With this option, when sparse sections are present in binary, the tool generates multiple hex files within the directory named according to the output option, hence srec_cat was complaining that it was given directory instead of a file name.
Python build scripts use different option --i32combined to prevent this from happening.
Anyway, with cymetadata section removed with #10011, sparse sections will not be present and either option will work. Removal of metadata should also resolve FUTURE_SEQUANA_PSA failures even without re-build of PSA binaries, because there will be no conflicting data in the main program.

[0xc0170]

Thanks for fixing this

[MarceloSalazar]

@0xc0170 can we close this?