Mbed-os: LittleFS assert with "Assertion failed: block < lfs->cfg->block_count"

[jshen3]

Original question in mbed forum

https://os.mbed.com/questions/87274/LittleFS-assert-with-Assertion-failed-bl/

We have several instance of file corruption when power cycle thedevice and during normal operation. After i clone the SD card to new SD card, the new SD card will have the same assert when trying to read the corrupted file or when calling statvfs() or when trying to create new files.
Reformat the SD card, or remove all the corrupted file can fix the issue.

the error message

++ MbedOS Error Info ++
Error Status: 0x80FF0144 Code: 324 Module: 255
Error Message: Assertion failed: block < lfs->cfg->block_count
Location: 0x0
File: .mbed-osfeaturesstoragefilesystemlittlefslittlefslfs.c+18
Error Value: 0x0
Current Thread: main Id: 0x2000D430 Entry: 0x8023E09 StackSize: 0x1000 StackMem: 0x2000D490 SP: 0x2000DF18
For more info, visit: https://armmbed.github.io/mbedos-error/?error=0x80FF0144

$ cat temp1909280234.tmp h
lfs info:413: file_open(0, "00170d3314aa/00170d5acf2a/temp1909280234.tmp", 0x0)
lfs info:415: file_open -> 0
lfs info:441: file_read(20008520, 20008778, 512)
C:softwaregitconmet_test_automationmbed-osfeaturesstoragefilesystemlittlefslittlefslfs.c:1144 test -- assertion failed

++ MbedOS Fault Handler ++
FaultType: HardFault

Context:
R0 : 00000018
R1 : 00020026
R2 : 20002AF0
R3 : 00000000
R4 : 20008544
R5 : 20008548
R6 : 00000001
R7 : 00000000
R8 : 200033A8
R9 : 00000000
R10 : 0800282C
R11 : 00000001
R12 : 08008047
SP : 20002AF8
LR : 080015B5
PC : 08007B3C
xPSR : 01000000
PSP : 20002A90
MSP : 20005E00
CPUID: 410FC241
HFSR : 80000000
MMFSR: 00000000
BFSR : 00000000
UFSR : 00000000
DFSR : 0000000A
AFSR : 00000000
Mode : Thread
Priv : Privileged
Stack: PSP
++ MbedOS Error Info ++
Error Status: 0x80FF013D Code: 317 Module: 255
Error Message: Fault exception
Location: 0x0
Error Value: 0x8007B3C
Current Thread: main Id: 0x20001C58 Entry: 0x800C671 StackSize: 0x1000 StackMem: 0x20001CB8 SP: 0x20005DA0

And if turn on LFS info and disable assert, see infinity message of this

lfs info:443: file_read -> -5003 ffffffff lfs info:441: file_read(20008520, 20008778, 512) lfs info:443: file_read -> -5003 ffffffff lfs info:441: file_read(20008520, 20008778, 512) lfs info:443: file_read -> -5003 ffffffff lfs info:441: file_read(20008520, 20008778, 512)

[jshen3]

would it be possible to return error instead of assert which cause the device reboot?

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1935

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1936

[pilotak]

I have the same problem when calling statvfs() to find out free space on DataFlash, as said it would be nice to return error code instead rising an error. The problem is i don't know what is causing it if it's a problem in my code or in a lib.

[0xc0170]

cc @ARMmbed/mbed-os-storage @geky

[dbacher-linear]

Hello, we've seen this problem in systems where the power is cycled unpredictably. For our planning, is this issue being addressed and if so, is there an update on when a fix might be available?

[bmbove]

I'm possibly having this same issue (when power is cycled unpredictably). I can potentially help debug if needed.

[geky]

This looks like the same issue: https://github.com/ARMmbed/littlefs/issues/295

would it be possible to return error instead of assert which cause the device reboot?

This is reasonable, it will just take some effort to work the error in such that it doesn't trigger the bad-block handling logic.

Note it likely won't make the filesystem _recoverable_. If the error is hit in mount you won't be able to progress.

[dbacher-linear]

@geky based on your comments on ARMmbed/littlefs#295, it sounds like fixing the underlying cause will be a long-term effort involving the development of a test setup.

If there is a shorter-term fix that could allow the application to handle this error on file access or mount instead of asserting, that would be helpful.

[bmbove]

@dbacher-linear as this is a piece of community-developed free and open source software, it would be more helpful if you had resources you could put towards fixing this bug rather than hounding the maintainers for a fix

[dbacher-linear]

@bmbove My apologies, I didn't intend to come across as hounding.

[geky]

Hi, sorry, had to be away from GitHub for a bit.

If there is a shorter-term fix that could allow the application to handle this error on file access or mount instead of asserting, that would be helpful.

I did look into a short term fix (sorry for not updating earlier), but it wasn't going to be as straightforward as it first appears. We could have changed the assert to return LFS_ERR_CORRUPT, but littlefs tries to recover from LFS_ERR_CORRUPT errors in the block device, which would just lead to more corruption of the filesystem tree. We could have added a new error code, but this would create something else to maintain in the future.

I may have been wrong, but I think the fastest route to fixing this was just fixing the bug.

I have been convinced that littlefs needs to handle bad block addresses better, but I think this should be handled when we load the block addresses from disk. This avoid having the error caught and also moves the error to the source of the problem. I'm going to add this to https://github.com/ARMmbed/littlefs/pull/372 time permitting.

---

Also I believe this issue is now fixed by https://github.com/ARMmbed/littlefs/pull/372 (https://github.com/ARMmbed/littlefs/pull/372/commits/f4b6a6b3284d8b721ba3db0bba93ee45fde7ead9 specifically), though it will be a bit longer before it makes its way into the Mbed OS version.

[jobshen]

@geky thanks for the update.

[AriParkkila]

@geky Do you know when the fix is landing? What is the correct way to recover from this assert?

[geky]

https://github.com/ARMmbed/littlefs/pull/372 is ready to go at the moment, but it still needs to go through CI, have v2.2 merge conflicts sorted out, release notes, and then get brought into littlefs-fuse and mbed-littlefs. So it may still be a day or two before it's available.

Ah, I should also make it clear this is only for v2, which is not merged into Mbed OS yet and only available in this external repo https://github.com/ARMmbed/mbed-littlefs. If/when v2 is merged into Mbed OS is out of my hands.

[AriParkkila]

@geky Thanks, what would be good way to recover from this assert on Mbed OS 5.15?

[geky]

The fixes related to this PR should now be on https://github.com/armmbed/mbed-littlefs v2.2.

@geky Thanks, what would be good way to recover from this assert on Mbed OS 5.15?

I don't believe it's possible to recover from this state without modifying Mbed OS.

There was a discussion here about possible workarounds, but there's not an easy workaround for the assert in the case of writing (littlefs tries to recover from write errors at that stage).
https://github.com/ARMmbed/littlefs/issues/390#issuecomment-601745860

Unfortunately, I don't personally have enough free time to do anything more than accept other's patches into v1.

[VeijoPesonen]

LittleFSv2.2 has been merged to the Mbed OS master. PR #12783.