Esp-idf: SPI SRAM memory test fail (IDFGH-1044)

[hutualive]

Environment

• Development Kit: ESP32-Wrover-Kit
• Kit version (for WroverKit/PicoKit/DevKitC): v3
• Core (if using chip or module): ESP32-Wrover
• Development Env: other
• Operating System: MacOS
• Power Supply: USB

Problem Description

Hi, new bought official ESP-WROVER-KIT flashed with the latest prebuilt firmware MicroPython_LoBo_esp32_psram_all from(https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo/wiki/firmwares), after reboot, can't be connected through rshell, conncted with a serial terminal, after reboot, the boot message show that cpu halted, seems due to SPI RAM check failed:

any hints on this ?

#

-->
ets Jun 8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x3e (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5904
ho 0 tail 12 room 4
load:0x40078000,len:0
load:0x40078000,len:15060
entry 0x40078630
.[0;32mI (30) boot: ESP-IDF v3.1-dev-961-ga2556229 2nd stage bootloader.[0m
.[0;32mI (30) boot: compile time 17:37:38.[0m
.[0;32mI (32) boot: Enabling RNG early entropy source....[0m
.[0;32mI (36) boot: SPI Speed : 40MHz.[0m
.[0;32mI (40) boot: SPI Mode : DIO.[0m
.[0;32mI (44) boot: SPI Flash Size : 4MB.[0m
.[0;32mI (49) boot: Partition Table:.[0m
.[0;32mI (52) boot: ## Label Usage Type ST Offset Length.[0m
.[0;32mI (59) boot: 0 nvs WiFi data 01 02 00009000 00006000.[0m
.[0;32mI (67) boot: 1 phy_init RF data 01 01 0000f000 00001000.[0m
.[0;32mI (74) boot: 2 MicroPython factory app 00 00 00010000 00200000.[0m
.[0;32mI (82) boot: 3 internalfs Unknown data 01 82 00210000 00100000.[0m
.[0;32mI (89) boot: End of partition table.[0m
.[0;32mI (93) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x71a84 (465540) map.[0m
.[0;32mI (265) esp_image: segment 1: paddr=0x00081aac vaddr=0x3ffb0000 size=0x059a8 ( 22952) load.[0m
.[0;32mI (275) esp_image: segment 2: paddr=0x0008745c vaddr=0x40080000 size=0x00400 ( 1024) load.[0m
.[0;32mI (275) esp_image: segment 3: paddr=0x00087864 vaddr=0x40080400 size=0x087ac ( 34732) load.[0m
.[0;32mI (298) esp_image: segment 4: paddr=0x00090018 vaddr=0x400d0018 size=0x15c060 (1425504) map.[0m
.[0;32mI (797) esp_image: segment 5: paddr=0x001ec080 vaddr=0x40088bac size=0x1576c ( 87916) load.[0m
.[0;32mI (834) esp_image: segment 6: paddr=0x002017f4 vaddr=0x400c0000 size=0x00714 ( 1812) load.[0m
.[0;32mI (835) esp_image: segment 7: paddr=0x00201f10 vaddr=0x50000000 size=0x0098c ( 2444) load.[0m
.[0;32mI (860) boot: Loaded app from partition at offset 0x10000.[0m
.[0;32mI (860) boot: Disabling RNG early entropy source....[0m
.[0;32mI (863) spiram: SPI RAM mode: flash 40m sram 40m.[0m
.[0;32mI (866) spiram: PSRAM initialized, cache is in low/high (2-core) mode..[0m
.[0;32mI (873) cpu_start: Pro cpu up..[0m
.[0;32mI (877) cpu_start: Starting app cpu, entry point is 0x4008180c.[0m
.[0;32mI (0) cpu_start: App cpu up..[0m
.[0;31mE (1779) spiram: SPI SRAM memory test fail. 131072/131072 writes failed, first @ 3F800000
.[0m
.[0;31mE (1779) cpu_start: External RAM failed memory test!.[0m
abort() was called at PC 0x400819e7 on core 0

Backtrace: 0x40096ca0:0x3ffe3cc0 0x40096e67:0x3ffe3ce0 0x400819e7:0x3ffe3d00 0x40078d52:0x3ffe3d20 0x40078e05:0x3ffe3d50 0x4007917b:0x3ffe3d90 0x40078643:0x3ffe3db0 0x40007c31:0x3ffe3eb0 0x4000073d:0x3ffe3f20

CPU halted.

#

per loboris(the creater and maintainer of micropython-esp32-lobo porting), seems it's similar as issue #1959, which related to spi psram operation. the board is bought from official channel in apr'18 and should be the latest version.

[projectgus]

It looks like the onboard PSRAM is not responding at all.

Is anything connected to the WROVER-KIT board? GPIOs 16, 17, 7, 8, 9 & 10 are connected to PSRAM inside the WROVER module, so anything connected to these (especially 16 & 17) would cause the chip to malfunction.

[hutualive]

hi @projectgus , nothing connected to those pins externally, the jumper setting is default per the getting started guide of espressif, per the doc, GPIO 16 & 17 are not broken out too.

[projectgus]

Thanks for confirming.

Most likely this is either faulty hardware or something related to the built micropython project or config (or an interaction between the config & some hardware problem).

To show it's definitely hardware, are you able to follow the Getting Started guide to build/flash an ESP-IDF example project? You can enable PSRAM in any example under "make menuconfig", the option is here. PSRAM self-test on startup is enabled by default.

[hutualive]

hi @projectgus , i'm using esp-idf to enable psram support and test with hello-world example, the result as below :

#

rst:0xc (SW_CPU_RESET),boot:0x3e (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5736
load:0x40078000,len:0
ho 12 tail 0 room 4
load:0x40078000,len:15036
entry 0x40078630
I (30) boot: ESP-IDF v3.1-dev-961-ga2556229 2nd stage bootloader
I (30) boot: compile time 12:07:00
I (32) boot: Enabling RNG early entropy source...
I (37) boot: SPI Speed : 40MHz
I (41) boot: SPI Mode : DIO
I (45) boot: SPI Flash Size : 4MB
I (49) boot: Partition Table:
I (52) boot: ## Label Usage Type ST Offset Length
I (60) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (67) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (75) boot: 2 factory factory app 00 00 00010000 00100000
I (82) boot: End of partition table
I (86) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x0893c ( 35132) map
I (107) esp_image: segment 1: paddr=0x00018964 vaddr=0x3ffb0000 size=0x022c0 ( 8896) load
I (111) esp_image: segment 2: paddr=0x0001ac2c vaddr=0x40080000 size=0x00400 ( 1024) load
0x40080000: _iram_start at /Users/dp/esp32/esp-idf/components/freertos/xtensa_vectors.S:1685

I (116) esp_image: segment 3: paddr=0x0001b034 vaddr=0x40080400 size=0x04fdc ( 20444) load
I (133) esp_image: segment 4: paddr=0x00020018 vaddr=0x400d0018 size=0x15cdc ( 89308) map
0x400d0018: _flash_cache_start at ??:?

I (164) esp_image: segment 5: paddr=0x00035cfc vaddr=0x400853dc size=0x0815c ( 33116) load
0x400853dc: esp_cpu_unstall at /Users/dp/esp32/esp-idf/components/soc/esp32/cpu_util.c:42

I (178) esp_image: segment 6: paddr=0x0003de60 vaddr=0x400c0000 size=0x00000 ( 0) load
I (186) boot: Loaded app from partition at offset 0x10000
I (186) boot: Disabling RNG early entropy source...
E (189) spiram: SPI RAM enabled but initialization failed. Bailing out.
I (195) cpu_start: Failed to init external RAM; continuing without it.
I (202) cpu_start: Pro cpu up.
I (206) cpu_start: Starting app cpu, entry point is 0x4008110c
0x4008110c: call_start_cpu1 at /Users/dp/esp32/esp-idf/components/esp32/cpu_start.c:224

I (197) cpu_start: App cpu up.
I (217) heap_init: Initializing. RAM available for dynamic allocation:
I (223) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (229) heap_init: At 3FFB2AE8 len 0002D518 (181 KiB): DRAM
I (236) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (242) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (248) heap_init: At 4008D538 len 00012AC8 (74 KiB): IRAM
I (255) cpu_start: Pro cpu start user code
I (49) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
Hello world!
This is ESP32 chip with 2 CPU cores, WiFi/BT/BLE, silicon revision 1, 4MB external flash
Restarting in 10 seconds...
Restarting in 9 seconds...
Restarting in 8 seconds...
Restarting in 7 seconds...
Restarting in 6 seconds...
Restarting in 5 seconds...
Restarting in 4 seconds...
Restarting in 3 seconds...

#

as you could see from the log, SPI RAM is not working properly:

E (189) spiram: SPI RAM enabled but initialization failed. Bailing out.
I (195) cpu_start: Failed to init external RAM; continuing without it.

so it's quite sure now the issue is due to faulty spi psram inside the module ?

thks.

[projectgus]

so it's quite sure now the issue is due to faulty spi psram inside the module ?

I think so, unfortunately. (Possibly the PSRAM itself is OK and this is a soldering issue inside the module, or on the dev board. But either way the hardware appears faulty.)

[hutualive]

@projectgus thanks. I will try with a new hardware to double check. with new findings will let you know(hope not...)

[talss89]

Not sure if this is useful, but I have just had (today) an ESP32-WROVER-KIT-V3 go bad in exactly the same manner as the OP. Had been left in an antistatic bag for 2 weeks, was working prior, and now no SPI PSRAM. All pins disconnected, default jumper position and hello-world example.

If this is a manufacturing fault, is there a place we can log these issues, and get an idea of failure rate?

[hutualive]

@projectgus I got a few new wrover modules and cross-check with the same firmware, it's working pretty well. it's quite clear now that the issue is due to the faulty hardware on wrover-kit.

@talss89 I sent back to the seller to test and validate, I think it will be a long process...not to say to identify the root cause.

[projectgus]

@hutualive Good to hear that you've confirmed this. Hopefully you're back up and running now.

@talss89 There isn't any central repository that I know of. However please let the vendor you bought from know about this failure (hopefully they can provide you with a replacement). For my part (as an IDF software developer) I don't recall any mentions of DOA (or suddenly dying) PSRAM apart from these two reports.

[formus14]

@hutualive i don't know if your error is fixed by now or not .. but one thing to consider trying out is the dual-core configuration...

You simply have to un-select "Run FreeRTOS only on first core" under "FreeRTOS" from the menuconfig.

[clduan]

Hi, I am running camera_web_server example on ESP-EYE dev board and come across the same issue. That example was run on dev board successfully, but after two weeks, the memory test never pass.

[mahavirj]

@clduan (Bit surprising) Can you please try with default sdk configuration (e.g. flash mode dual, frequency 40M), just want to double check that its really some sort of hardware issue, or issue tied with sdk configuration? In addition, do you have spare esp-eye kit to try same firmware image?

[clduan]

@mahavirj Hi there, thanks for reply.
I have tried to change Set RAM clock speed to 40MHz, and have the same error. I got another esp-eye dev board and ran the same firmware image successfully, so I think it's basically hardware issue. What interested me mostly is that why the hardware was okay previously and after a short while it became an RAM error, as I haven't treat it in an ugly manner.

[b2cbd]

@clduan Any update later? I'm facing the same issue now, it worked for a couple of days and started to show this error.

[clduan]

@b2cbd Not any luck here. Maybe hardware design issue since so many people came across the same weird issue...