Mbed-os: NUCLEO_F767ZI failed with features-storage-tests-kvstore-general_tests_phase_1&2
Description of defect
With the PR of https://github.com/ARMmbed/mbed-os/pull/12638, kvstore tests been enabled on a number of targets.
One of the issue been noticed is
features-storage-tests-kvstore-general_tests_phase_1&2
is failing on NUCLEO_F767ZI with nightly tests:
Target(s) affected by this defect ?
NUCLEO_F767ZI
Toolchain(s) (name and version) displaying this defect ?
GCC_ARM and ARMC6
What version of Mbed-os are you using (tag or sha) ?
latest master or 098c72a31277e0b292dd6cdf1abaf9ca9d487032
What version(s) of tools are you using. List all that apply (E.g. mbed-cli)
latest mbed-cli
How is this defect reproduced ?
mbed test -t GCC_ARM -m NUCLEO_F767ZI -n features-storage-tests-kvstore-general_tests_phase_1
jamesbeyond
All 15 comments
@tymoteuszblochmobica @SeppoTakalo
jamesbeyond
on 14 Apr 2020
@jamesbeyond thank you for raising this issue.Please take a look at the following comments:
It would help if you could also specify the versions of any tools you are using?
NOTE: If there are fields which are not applicable then please just add 'n/a' or 'None'.This indicates to us that at least all the fields have been considered.
Please update the issue header with the missing information, the issue will not be mirroredto our internal defect tracking system or investigated until this has been fully resolved.
ciarmcom
on 14 Apr 2020
Issue:
[1586830838.28][CONN][RXD] >>> Running case #1: 'TDB_kvstore_init'...
[1586830839.38][CONN][RXD] ++ MbedOS Error Info ++
[1586830839.38][CONN][RXD] Error Status: 0x80FF0144 Code: 324 Module: 255
[1586830839.48][CONN][RXD] Error Message: Assertion failed: _area_params[0].size == _area_params[1].size
[1586830839.48][CONN][RXD] Location: 0x8014F8F
[1586830839.58][CONN][RXD] File: ./features/storage/kvstore/tdbstore/TDBStore.cpp+215
[1586830839.58][CONN][RXD] Error Value: 0x0
[1586830839.68][CONN][RXD] Current Thread: main Id: 0x20006094 Entry: 0x8015C9B StackSize: 0x1000 StackMem: 0x20004B00 SP: 0x200058B0
[1586830839.78][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80FF0144&tgt=NUCLEO_F767ZI
[1586830839.78][CONN][RXD] -- MbedOS Error Info --
jeromecoutant
on 15 Apr 2020
Internal Jira reference: https://jira.arm.com/browse/MBOTRIAGE-2632
ciarmcom
on 15 Apr 2020
cc @ARMmbed/mbed-os-storage
0xc0170
on 15 Apr 2020
Any update from @ARMmbed/mbed-os-storage ?
Thx
jeromecoutant
on 29 Apr 2020
I've seen this in the recent nightlies, going to talk to the team about it
0xc0170
on 6 May 2020
My first guess would be that the board is using non-uniform erase sector sizes. Other option is that there is so little space(required number of erase sectors for KVStore which is at minimum two) it doesn't fit.
// The size calculations are a bit complex because we need to make sure we're
// always aligned to an erase block boundary (whose size may not be uniform
// across the address space), and we also need to make sure that the first
// area never goes over half of the total size even if bd_size is an odd
// number of sectors.
while (true) {
bd_size_t erase_unit_size = _bd->get_erase_size(area_0_size);
if (area_0_size + erase_unit_size <= (bd_size / 2)) {
area_0_size += erase_unit_size;
} else {
break;
}
}
while (true) {
bd_size_t erase_unit_size = _bd->get_erase_size(area_0_size + area_1_size);
if (area_1_size + erase_unit_size <= (bd_size / 2)) {
area_1_size += erase_unit_size;
} else {
break;
}
}
_area_params[0].address = 0;
_area_params[0].size = area_0_size;
_area_params[1].address = area_0_size;
_area_params[1].size = area_1_size;
// The areas must be of same size
MBED_ASSERT(_area_params[0].size == _area_params[1].size);
VeijoPesonen
on 7 May 2020
I'm not seeing the same issue if I try to test in RAAS with latest Mbed OS. Not saying that I don't see any issues as there seems to be some. I would say that with debug build the application takes so much space there is no more left enough for KVStore. Situation might vary depending how much application happens to take space.
In case of the Mbed OS version 029109a2f020fb54813e616e35a735df73b540b1 the issue doesn't become visible - even with debug build. If I switch to same version as has been mentioned in the Mbed OS issue ticket the problem becomes visible with debug- but not with release-build. But that doesn't tell that the KVStore regions are same size, assertions are just disabled.
[1588835387.37][CONN][RXD] >>> Running case #44: 'FS_set_same_key_several_time'...
[1588835387.47][CONN][INF] found KV pair in stream: {{__testcase_start;FS_set_same_key_several_time}}, queued...
[1588835390.49][CONN][RXD] lfs warn:313: No more free space 7
[1588835390.59][CONN][RXD] lfs warn:313: No more free space 7
[1588835391.59][CONN][RXD] :259::FAIL: Expected 0 Was 271
[1588835395.82][CONN][RXD] >>> 'FS_set_same_key_several_time': 0 passed, 1 failed with reason 'Test Cases Failed'
[1588835395.92][CONN][RXD] >>> Running case #45: 'FS_set_several_keys_multithreaded'...
[1588835396.02][CONN][INF] found KV pair in stream: {{__testcase_start;FS_set_several_keys_multithreaded}}, queued...
[1588835400.06][CONN][RXD] lfs warn:313: No more free space 7
[1588835400.06][CONN][RXD] lfs warn:313: No more free space 7
[1588835402.15][CONN][RXD] :272:lfs w:FAILarn:3: Ex13: Npecteo mord 0 We fras 27ee sp1
[1588835402.15][CONN][RXD] ace 7
[1588835402.15][CONN][RXD] lfs warn:313: No more free space 7
[1588835403.15][CONN][RXD] :272::FAIL: Expected 0 Was 271
[1588835403.26][CONN][RXD] :297::FAIL: Expected 0 Was 263
[1588835403.26][CONN][RXD] :297::FAIL: Expected 0 Was 263
[1588835404.27][CONN][RXD] :303::FAIL: Expected 0 Was 263
[1588835404.36][CONN][RXD] :303::FAIL: Expected 0 Was 263
[1588835404.36][CONN][INF] found KV pair in stream: {{__testcase_finish;FS_set_several_keys_multithreaded;0;6}}, queued...
[1588835404.47][CONN][RXD] >>> 'FS_set_several_keys_multithreaded': 0 passed, 6 failed with reason 'Test Cases Failed'
[1588835434.32][CONN][RXD] >>> Running case #53: 'FS_set_several_key_value_sizes'...
[1588835434.42][CONN][INF] found KV pair in stream: {{__testcase_start;FS_set_several_key_value_sizes}}, queued...
[1588835438.44][CONN][RXD] lfs warn:313: No more free space 7
[1588835438.44][CONN][RXD] lfs warn:313: No more free space 7
[1588835439.55][CONN][RXD] :462::FAIL: Expected 0 Was 271
[1588835440.55][CONN][RXD] lfs warn:313: No more free space 7
[1588835440.55][CONN][RXD] lfs warn:313: No more free space 7
[1588835441.68][CONN][RXD] :462::FAIL: Expected 0 Was 271
[1588835442.67][CONN][RXD] lfs warn:313: No more free space 7
[1588835442.67][CONN][RXD] lfs warn:313: No more free space 7
[1588835443.77][CONN][RXD] :462::FAIL: Expected 0 Was 271
[1588835444.78][CONN][RXD] lfs warn:313: No more free space 7
[1588835444.78][CONN][RXD] lfs warn:313: No more free space 7
[1588835445.88][CONN][RXD] :462::FAIL: Expected 0 Was 271
[1588835446.90][CONN][RXD] :469::FAIL: Expected 0 Was 263
[1588835446.99][CONN][RXD] :470::FAIL: Expected 'name_b' Was ''
[1588835446.99][CONN][INF] found KV pair in stream: {{__testcase_finish;FS_set_several_key_value_sizes;0;6}}, queued...
[1588835447.09][CONN][RXD] >>> 'FS_set_several_key_value_sizes': 0 passed, 6 failed with reason 'Test Cases Failed'
I would say there need alter the build system in a way that KVStore is treated in a similar manner as bootloader. In other words certain space(region) is reserved for it and error is reported if there is an overlap. I don't know is our build system that clever but I would hope so.
Using ROM regions bootloader, header, application, post_application in this build.
Region bootloader: size 0x7000, offset 0x0
Region header: size 0x70, offset 0x8000
Region application: size 0x77c00, offset 0x8400
Region post_application: size 0x80000, offset 0x80000
_post_application_ would be something what is reserved for KVStore. The default KVStore configuration leaves nowadays two spare sectors(erase sectors) for the application to grow. Scratch that, that was old behavior. Nowadays KVStore tries to take two sectors from the end of the flash.
If bootloader is used this issue can be avoided by placing the KVStore between the bootloader and the application.
VeijoPesonen
on 7 May 2020
0xc0170 added the devices: st label on 15 Apr
Are you sure ?
Issue seems to be in features/storage/kvstore/tdbstore/TDBStore.cpp file
@ARMmbed/mbed-os-storage
@MarceloSalazar
jeromecoutant
on 8 Jun 2020
I can reproduce this issue 100% of the time no matter which toolchain, profile and revision I use.
LDong-Arm
on 17 Aug 2020
~Fix created: https://github.com/ARMmbed/mbed-os/pull/13466~
In short, the STM32F7 flash has only a few very large sectors of variable sizes, and we had alignment issues with FlashIAPBlockDevice and littlefs v1 lacks support when sectors are few.
Update: the fix is superseded - the solution is not generic enough and can't handle dual banks, etc.
LDong-Arm
on 20 Aug 2020
Issue:
[1586830838.28][CONN][RXD] >>> Running case #1: 'TDB_kvstore_init'... [1586830839.38][CONN][RXD] ++ MbedOS Error Info ++ [1586830839.38][CONN][RXD] Error Status: 0x80FF0144 Code: 324 Module: 255 [1586830839.48][CONN][RXD] Error Message: Assertion failed: _area_params[0].size == _area_params[1].size [1586830839.48][CONN][RXD] Location: 0x8014F8F [1586830839.58][CONN][RXD] File: ./features/storage/kvstore/tdbstore/TDBStore.cpp+215 [1586830839.58][CONN][RXD] Error Value: 0x0 [1586830839.68][CONN][RXD] Current Thread: main Id: 0x20006094 Entry: 0x8015C9B StackSize: 0x1000 StackMem: 0x20004B00 SP: 0x200058B0 [1586830839.78][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80FF0144&tgt=NUCLEO_F767ZI [1586830839.78][CONN][RXD] -- MbedOS Error Info --
This is harder to reproduce, but also caused by non-uniform erase size. TDBStore requires two block of the same size and uses the first two blocks, whose sizes are not guaranteed.
LDong-Arm
on 10 Sep 2020
Just to make it clear, we agreed with ST to skip LittleFileSystem on STM32F2/4/7 (#13593) and add to our documentation how to enable and properly align that manually.
LDong-Arm
on 14 Sep 2020
Related issues
0xc0170
路
3Comments
hasnainvirk
路
3Comments
cesarvandevelde
路
4Comments
1domen1
路
3Comments
bulislaw
路
3Comments