Arduino: SPIFFS doesn't read Files when loaded with the tool .

[ashiroji]

Basic Infos

Hardware

Hardware: ESP-12
Core Version: Commit : Update spiffs 0.3.7 to and mkspiffs 0.1.3 (6ac48124bd10151cdbc5c1003e41e4483f8f1f11 )

Description

Updating my esp8266 repo to the latest git version causes the following issues :+1:
-Ability to read/write SPIFFS through software (verified with ConfigFile example )
-Inability to read SPIFFS when files loaded via ESP8266 Sketch Data Upload tool (veirfied with the FS Browser example )

Settings in IDE

Module: Generic ESP8266 Module
Flash Size: 4MB/1MB
CPU Frequency: 80Mhz
Flash Mode: dio
Flash Frequency: 40Mhz
Upload Using: SERIAL
Reset Method: ck

Sketch

ConfigFile Example

Messages :
Config saved
Config loaded

FSBrowser

Messages :
File Not Found

[me-no-dev]

did you run get.py to update mkspiffs as well? If you are using old mkspiffs and new spiffs it will not work

[igrr]

@ashiroji which OS are you running mkspiffs on?

[ashiroji]

I'm using Ubuntu 16.04

[igrr]

Linking related mkspiffs ticket: https://github.com/igrr/mkspiffs/issues/26

[ashiroji]

@me-no-dev No i didn't run get.py.

If I understand this , the new lib is in a way incompatible with the old tool ?
so I need to update both lib and tool right ?

Thank you all for the quick answer

[universam1]

I tried the latest tool 1.3 and did not create a readable image!

[reaper7]

I have similar problem.
I use current github version + updated mkspiffs,
for test I uploaded ESP8266WebServer->FSBrowser example with additional binary file (277 552bytes) inside data folder.
WebFileEditor in this example shows all files (with my bin too) but downloaded bin is different than uploaded!
Downloaded file size is 276 667bytes and simple compare in WinMerge shows 9 differences in separate places.

[me-no-dev]

@reaper7 can you please show the differences in hex format? and more than one if possible? maybe the bytes can make some sense of this.

[me-no-dev]

their offsets can also point to something... seems to happen at equal-ish intervals

[reaper7]

can you recommend some tool for this?
You can download binaries: https://files.fm/u/pz7u5tee

[me-no-dev]

you can see a pattern there :) I have downloaded the files and will have a look ;) surely we will figure it out!

[reaper7]

except first diff, next diffs cut 109 bytes every 0x7A01

[me-no-dev]

can you try to upload the same files but add one that will be put before the bin, so we can see if the first offset will change. If so that would mean that something wrong is read from the file system. Also please point which config you are using (4M/3MB SPIFFS or something else) You could try different config maybe so it would change the block size.

[reaper7]

nodemcu1.0 4M/3SPIFFS

the same binary as second file (after jpg img 468 909bytes):

[reaper7]

nodemcu1.0 4M/3SPIFFS
and binary as a single file

looks identical

[reaper7]

nodemcu1.0 4M/1SPIFFS
and binary as a single file

so...spiffs size and file count/location does not affect the result

[reaper7]

and finally, a completely different file (single) 4M/1M
upload file size: 468 909
downloaded file size: 467 370

the same patern

[Lan-Hekary]

I am having the same problem here
but I used a different approach to check and found that the problem is in mkspiffs.exe tool ..

I used mkspiffs.exe version 0.1.3 in command line on windows to pack the files into spiffs.bin file ..
mkspiffs.exe -c original_data -b 8192 -p 256 -s 1028096 spiffs.bin
the block,page and fs sizes are from the ld script for 1m spiffs on a 4m flash

then I used this line to unpack:
mkspiffs.exe -u extracted_data -b 8192 -p 256 -s 1028096 spiffs.bin

the two folder's are not identical ...
some file are identical only

I can upload the content of the data if that would help ..
but I think it can be reproduced easily ..

I also tested it with version 0.1.2 but I found that the extracted files are all identical ..
so the bug is in the tool for sure ..

[me-no-dev]

This is a great clue!!! I might have an idea! See this: https://github.com/pellepl/spiffs/commit/f5e26c4e933189593a71c6b82cda381a7b21e41c and this: https://github.com/pellepl/spiffs/issues/165

this fix was released after we changed the version

[Lan-Hekary]

yup .. that's true ..
the corrupted files are over 25k ..
I think the fix is easy .. merge the new fix into the tool and core of the esp8266

[me-no-dev]

will try to do all that tomorrow :)

[Lan-Hekary]

I am trying to fix it now and test the result
but I am not sure how to fix the problem across all these repositories :D
after all I am new to this kind of integration ..

I will override the SPIFFS src files with the new one in the esp8266 core
but how can I change the compiled version of the tool ??
all I see is the source code from
https://github.com/igrr/mkspiffs

I can edit that and compile it on my PC but I have no Idea how to export it ..

[me-no-dev]

just copy the binary to your esp8266-arduino folder :) we will do the release (@igrr)

[me-no-dev]

Oh, make sure you also update the spiffs files in the Arduino as well (cores/esp8266/spiffs/...)

[Lan-Hekary]

I am doing it right now ..
if I can make a pull request I will make one ..
for the esp core and the mkspiffs tool ..

[Lan-Hekary]

I tested it multiple times with multiple files and I believe the fix works perfectly now ..
I also made pull requests to both the mkspiffs tool and the Arduino core with the new modifications ..
I guess you'll have to make a new release for the mkspiffs tool after all too ..

[reaper7]

someone can test ESP8266WebServer->FSBrowser example after latest changes on spiffs and mkspiffs tool (current master branch) ?
Seems like new mkspiffs 2.0 create propertly image but this example do not show any files
tested on nodemcu1.0 4M/3M & 4M/1M

[Pablo2048]

Same problem here - after update my IDE, upload new sketch and then upload spiffs image, SPIFFS.begin() return true, but FSBrowser (SPIFFSEditor form AsyncWebServer actually) show empty volume.

[igrr]

Not all updates are in, folks, sorry, i got distracted for a meeting :)
One thing remaining is to set SPIFFS_USE_MAGIC_LENGTH to 0 in spiffs_config in arduino. Will push the commit soon.

[Pablo2048]

Ok, no problem with that :-) I'll wait...

[Lan-Hekary]

I tested it originally with SPIFFS_USE_MAGIC_LENGTH set to 0 so it's going to be OK

[igrr]

Tested with 35d5fab, looks okay on my side.

[reaper7]

yes :)
tnx!

[Pablo2048]

LGTM! (4/1M, 4/3M). Thx.