Mbed-os: 5.11.0 - RC1 breaks PDMC on some Nuvoton targets.

[ashok-rao]

Description

The following targets are affected by this issue:
1. NUMAKER_PFM_M487
2. NUMAKER_IOT_M487

For NUMAKER_PFM_M487: The following tests are passing.
NUMAKER_PFM_M487_ARM_output.txt

For NUMAKER_IOT_M487: The following tests are passing.
NUMAKER_IOT_M487_ARM_output.txt

With 5.10.4 the PDMC tests on these exact boards are passing fine. Results below:

[2018-11-30 10:27:38] INFO: TEST RUNNER RESULTS:

+-------------------------+---------------------+-------------------------------------------------+--------+--------+
| target                  | platform_name       | test suite                                      | result |   time |
+-------------------------+---------------------+-------------------------------------------------+--------+--------+
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |   69.3 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 139.86 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  92.49 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 176.27 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  83.55 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 172.03 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  88.64 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 196.17 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     | 175.92 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 492.89 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  88.77 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 176.32 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  74.32 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 154.13 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     | 110.85 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 204.66 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     | 161.46 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 979.02 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  94.79 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     |  181.9 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK     |  99.82 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK     | 231.66 |
+-------------------------+---------------------+-------------------------------------------------+--------+--------+


[2018-11-30 10:27:38] INFO: TEST FINISHED

However, updating ONLY the Mbed OS to 5.11.0-RC1 is breaking the same tests on the same boards.
For NUMAKER_PFM_M487: The following tests are failing:
NUMAKER_PFM_M487_ARM_output.txt

For NUMAKER_IOT_M487: The followign tests are failing:
NUMAKER_IOT_M487_ARM_output.txt

With 5.11.0-RC1 the PDMC tests on these exact boards are failing. Results below:

[2018-11-30 13:16:25] INFO: TEST RUNNER RESULTS:

+-------------------------+---------------------+-------------------------------------------------+---------+---------+
| target                  | platform_name       | test suite                                      | result  |    time |
+-------------------------+---------------------+-------------------------------------------------+---------+---------+
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-dev_mgmt-connect | TIMEOUT |  276.71 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-dev_mgmt-update  | TIMEOUT |   828.2 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |   92.66 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |   193.0 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |   83.24 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |  174.48 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |  103.72 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |   185.7 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-dev_mgmt-connect | TIMEOUT |  278.22 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-dev_mgmt-update  | TIMEOUT |    78.5 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |   84.58 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |  180.72 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |   74.27 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |  171.83 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-dev_mgmt-connect | TIMEOUT |   97.06 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-dev_mgmt-update  | FAIL    |   48.46 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-dev_mgmt-connect | TIMEOUT |  267.24 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-dev_mgmt-update  | TIMEOUT | 1109.38 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |   95.54 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |  183.65 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-dev_mgmt-connect | OK      |  189.74 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-dev_mgmt-update  | OK      |  259.39 |
+-------------------------+---------------------+-------------------------------------------------+---------+---------+


[2018-11-30 13:16:25] INFO: TEST FINISHED

This indicates there is something breaking these targets between the 2 versions.

Steps to reproduce: (_please note that this repo is private to Arm & partners only_)

1. mbed import pelion-enablement
2. cd pelion-enablement
3. mbed test -t ARM -n simple-mbed-cloud-client-tests-dev_mgmt* -m <targets_mentioned_above>

Expected:
All tests for connect + update should pass.

Actual:
Connect + update tests fails with either TIMEOUT or ERROR for the targets mentioned above.

This also impacts OoB for 5.11.0.
cc: @MarceloSalazar @screamerbg @0xc0170 @cmonr .

Issue request type

[ ] Question
[ ] Enhancement
[x] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-233

[samchuarm]

@ARMmbed/team-nuvoton

[0xc0170]

I am reviewing the changes between the versions. First thing t hat came to my mind: MPU enablement.

@ashok-rao Can you quickly test one of the failing tests with 5.11rc1, remove MPU from device_has from MCU_M480 target and add MBED_MPU_CUSTOM to macros. If not mpu related, we shall investigate further what has changed between the two versions

[0xc0170]

@c1728p9 Do you have a device to test this ? @ashok-rao is away for most of this week.

[c1728p9]

Hi @0xc0170 I don't have either of the Nuvoton boards to test with.

[0xc0170]

@ashok-rao @ARMmbed/team-nuvoton If anyone can help us narrow down the problem would be great

[cyliangtw]

@0xc0170 , I can't find any 5.11.x tag in mbed-os master branch, so just pull the last mbed-os. M487-iot log file shows network connect successfully but failed in "Initialize Simple PDMC", the root cause seems TLS init HW ECC twice and we are in modifying HW ECC init implementation to fulfill this use case.

m487-iot-ouput.txt

[0xc0170]

Use https://github.com/ARMmbed/mbed-os/releases/tag/tools-release-5.11.0-rc1 tag, or can just fetch master as it should be also there (hasn't been yet addressed).

[cmonr]

@cyliangtw Since we're in the middle of a feature release, the tag won't be added until the release is published.

For now, you can use either the mbed-os-5.11 branch, or master. Once https://github.com/ARMmbed/mbed-os/pull/8974 comes in, you should only use mbed-os-5.11 for testing.

[c1728p9]

I created #8994 to turn off the MPU for the MCU_M480 family of devices. The root cause of this problem still needs to be determined.

[Patater]

We can probably turn the MPU back on on this target, if the ECC issue is the only issue.

We have root caused the ECC issue and have a fix available at https://github.com/ARMmbed/mbed-os/pull/9005

[adbridge]

@c1728p9 See Jaeden's comment

[cmonr]

@adbridge @Patater It'll probably remain off for the rest of OOB since that can be brought in during a patch release.

@c1728p9 to confirm.

[c1728p9]

Sounds good to me.

[0xc0170]

@ashok-rao What is remaining here (for this issue to be closed) ?

[ashok-rao]

@0xc0170 .. This -> https://github.com/ARMmbed/mbed-os/pull/9020

[cmonr]

Closing since #9020 is now merged.