Zephyr: Can only run the flash_simulator test once on native_posix

[sjg20]

Describe the bug
The first time you run this it works. The second time, the flash.bin file is still around from last time and the test fails.

To Reproduce
Steps to reproduce the behavior:
1.west build --force -b native_posix tests/drivers/flash_simulator

1. rm flash.bin
2. ./build/zephyr/zephyr.elf - all good
3. ./build/zephyr/zephyr.elf - fails
4. go to 2

Expected behavior
Should always pass

Impact
annoyance

Screenshots or console output

$  ./build/zephyr/zephyr.elf 
*** Booting Zephyr OS build zephyr-v2.2.0-2692-gff856e01f5d8  ***
Running test suite flash_sim_api
===================================================================
starting test - test_int
PASS - test_int
===================================================================
starting test - test_write_read
PASS - test_write_read
===================================================================
starting test - test_erase
Incremental pattern expected
Erased area expected
Incremental pattern expected
PASS - test_erase
===================================================================
starting test - test_access
PASS - test_access
===================================================================
starting test - test_out_of_bounds
PASS - test_out_of_bounds
===================================================================
starting test - test_align
PASS - test_align
===================================================================
starting test - test_double_write
PASS - test_double_write
===================================================================
Test suite flash_sim_api succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL
ellesmere:/scratch/sglass/cosarm/zephyrproject/zephyr$ ./build/zephyr/zephyr.elf 
*** Booting Zephyr OS build zephyr-v2.2.0-2692-gff856e01f5d8  ***
Running test suite flash_sim_api
===================================================================
starting test - test_int

    Assertion failed at tests/drivers/flash_simulator/src/main.c:84: test_int: (0xff not equal to test_read_buf[i])
sim flash byte at offset 0x0 has value 0x002
FAIL - test_int
===================================================================
starting test - test_write_read

    Assertion failed at tests/drivers/flash_simulator/src/main.c:102: test_write_read: (0 not equal to rc)
flash_write (-5) should succedd at off 0x0
FAIL - test_write_read
===================================================================
starting test - test_erase
Incremental pattern expected

    Assertion failed at tests/drivers/flash_simulator/src/main.c:62: test_check_pattern32: (val32 not equal to r_val32)
flash word at offset 0x4 has value 0xffffffff02
FAIL - test_erase
===================================================================
starting test - test_access
PASS - test_access
===================================================================
starting test - test_out_of_bounds
PASS - test_out_of_bounds
===================================================================
starting test - test_align
PASS - test_align
===================================================================
starting test - test_double_write
PASS - test_double_write
===================================================================
Test suite flash_sim_api failed.
===================================================================
PROJECT EXECUTION FAILED

Environment (please complete the following information):

• OS: Linux
• Toolchain Zephyr SDK 0.11.2
• 1fed229f6e

Additional context
Add any other context about the problem here.

[nvlsianpu]

Creating the file only once is expected behaviour.
If recreating a file each time is needed please make this optional behaviour.

[sjg20]

So should we change the test to write first and then read, rather than assuming that the flash is erased?

[carlescufi]

@vanwinkeljan @nvlsianpu @fnde-ot What is the correct course of action here? Either document that flash.bin needs to be erased in between test runs, or change the test sequence so it erases the flash prior to writing it?

[vanwinkeljan]

I would say erase the flash, on the other hand is it not assumed that tests run in a pristine environment, eg. if you use sanitycheck instead of west (+ run manually) to run the test the issue is not encountered?

[nvlsianpu]

cc @de-nordic

[nvlsianpu]

Proper solution would be introduce configuration (ERASE_ON_EACH_STARTUP maybe) which determine how thread existing flash.bin.
Probably in test environment erasing the file should be the default behavior.

I found useful for some samples/application that them might store some data between resets.
I know that some users utilizes this fact as well.

@vanwinkeljan saniticheck indeed create pristine environment if it comes to the file existence as it creates new directory on each run.

[de-nordic]

I was thinking about generating random file name in /tmp for that.

[vanwinkeljan]

Proper solution would be introduce configuration (ERASE_ON_EACH_STARTUP maybe) which determine how thread existing flash.bin.
Probably in test environment erasing the file should be the default behavior.

Dont agree on this one, on real HW the flash will not be erased each time you reset/power-up your board. If the code assumes that the flash is erased at startup it should take care of this

[de-nordic]

There is advantage of leaving the file for debugging purposes.
Though I do not think that one test should depend on an output of another test, so when flash needs to be present and set prior to running test cases (power on case), operation of filling data into test should be part of test initialization, because it also ensures that test will be given predictable data.
I do not know how to test down/up case scenario with such approach, though I think that you should usually just test subsystem, that is use subsystem to write into flash the test data and then use the same subsystem read and check if you get what you expect.

[sjg20]

So shall we change the test to write whatever data it relies on? That's what I have in the 'flash read' test:

https://github.com/zephyrproject-rtos/zephyr/commit/6cc3f8f111074effaaef2198ac5d2821bed93be4

[de-nordic]

So shall we change the test to write whatever data it relies on? That's what I have in the 'flash read' test:

6cc3f8f

Yes, I believe so. It makes it easier to modify or expand a test without impacting other tests. It will not break tests if the order of tests get changed.

[sjg20]

OK I added another commit here to fix that:
https://github.com/zephyrproject-rtos/zephyr/pull/25048

[github-actions[bot]]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.