Riot: _NVIC_SystemReset stuck in infinite loop when calling pm_reboot through shell after flashing with J-Link

Created on 8 Jan 2020 · 19Comments · Source: RIOT-OS/RIOT

For some curious reason, after flashing several openmote-b nodes with a J-Link programmer, trying to reboot said nodes via the shell "reboot" command causes those nodes to hang forever. I verified that the infinite for loop in cpu/cortexm_common/include/vendor/core_cm3.h _is_ reached as it's supposed to while awaiting a reset. However, this reset actually never occurs, causing the node to hang forever. I also tested this with openmote-b nodes that where only ever flashed via USB and they work just fine. This issue persists even after disconnecting the J-Link programmer and, worse, after reflashing the node via USB. As such I suspect that flashing over J-Link writes to certain registers that are otherwise untouched when flashing over USB.

__NO_RETURN __STATIC_INLINE void __NVIC_SystemReset(void)
{
  __DSB();                                                          /* Ensure all outstanding memory accesses included
                                                                       buffered write are completed before reset */
  SCB->AIRCR  = (uint32_t)((0x5FAUL << SCB_AIRCR_VECTKEY_Pos)    |
                           (SCB->AIRCR & SCB_AIRCR_PRIGROUP_Msk) |
                            SCB_AIRCR_SYSRESETREQ_Msk    );         /* Keep priority group unchanged */
  __DSB();                                                          /* Ensure completion of memory access */

  for(;;)                                                           /* wait until reset */
  {
    /* this loop is called infinitely until reset, which never 
       happens and thus causes the node to hang forever */
    __NOP();
  }
}

boards bug

Source

tinstructor

All 19 comments

This post might be worth looking into. Is the nRST pin somehow being pulled high (permanently)?

tinstructor on 9 Jan 2020

@benpicco I know you've got extensive knowledge on this platform. Any idea as to what might cause this issue?

tinstructor on 10 Jan 2020

Final try: @haukepetersen and @MrKevinWeiss do you have any idea what may cause this issue?

tinstructor on 13 Jan 2020

@tinstructor can you give more details on how this issue surfaced to try to reproduce. Do I need to flash multiple times for it to show up? I have flashed a couple without experiencing your issue. My Jlink version is

SEGGER J-Link Commander V6.56d (Compiled Dec 12 2019 13:05:00)
DLL version V6.56d, compiled Dec 12 2019 13:04:51

fjmolinas on 13 Jan 2020

My Jlink version is

SEGGER J-Link Commander V6.60c (Compiled Dec 23 2019 16:18:17)
DLL version V6.60c, compiled Dec 23 2019 16:17:59

Ok so what I did is, I flashed the openmote-b "successfully" via J-link with the following command:

PROGRAMMER=jlink make flash BOARD=openmote-b

and when I subsequently login to the shell (after disconnecting the Jlink flash tool) via make term PORT=/dev/ttyUSB1 BOARD=openmote-b, this happens:

Twisted not available, please install it if you want to use pyterm's JSON capabilities
2020-01-13 13:28:09,106 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
help
2020-01-13 13:28:12,776 # help
2020-01-13 13:28:12,777 # Command              Description
2020-01-13 13:28:12,777 # ---------------------------------------
2020-01-13 13:28:12,778 # reboot               Reboot the node
2020-01-13 13:28:12,792 # ps                   Prints information about running threads.
2020-01-13 13:28:12,793 # random_init          initializes the PRNG
2020-01-13 13:28:12,793 # random_get           returns 32 bit of pseudo randomness
2020-01-13 13:28:12,794 # ifconfig             Configure network interfaces
> reboot
2020-01-13 13:28:19,992 #  reboot

The node then hangs forever until I manually perform a reset with the hardware reset button.

tinstructor on 13 Jan 2020

What application are you uploading? I still can't reproduce, I will try your JLkin version as well:

PROGRAMMER=jlink BOARD=openmote-b make -C examples/default/ flash-only

/home/francisco/workspace/RIOT/dist/tools/jlink/jlink.sh flash /home/francisco/workspace/RIOT/examples/default/bin/openmote-b/default.bin
### Flashing Target ###
### Flashing at base address 0x200000 with offset 0 ###
SEGGER J-Link Commander V6.56d (Compiled Dec 12 2019 13:05:00)
DLL version V6.56d, compiled Dec 12 2019 13:04:51

J-Link Commander will now exit on Error

J-Link Command File read successfully.
Processing script file...

J-Link connection not established yet but required for command.
Connecting to J-Link via USB...O.K.
Firmware: J-Link EDU Mini V1 compiled Dec 13 2019 11:16:57
Hardware version: V1.00
S/N: 801014916
License(s): FlashBP, GDB
VTref=2.472V
Target connection not established yet but required for command.
Device "CC2538SF53" selected.


Connecting to target via JTAG
InitTarget: Found ICE-Pick with ID: 0x8B96402F
InitTarget: Found CPU TAP 0x4BA00477
Scanning AP map to find all available APs
AP[1]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x412FC230. Implementer code: 0x41 (ARM)
Found Cortex-M3 r2p0, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 002BB000 SCS
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 002BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 002BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 002BB923 TPIU-Lite
Cortex-M3 identified.
Halting CPU for downloading file.
Downloading file [/home/francisco/workspace/RIOT/examples/default/bin/openmote-b/default.bin]...
Comparing flash   [100%] Done.
J-Link: Flash download: Bank 0 @ 0x00200000: Skipped. Contents already match
O.K.


Sleep(100)



Script processing completed.

BOARD=openmote-b make -C examples/default/ term

make: Entering directory '/home/francisco/workspace/RIOT/examples/default'
/home/francisco/workspace/RIOT/dist/tools/pyterm/pyterm -p "/dev/riot/tty-openmote-b" -b "115200" 
2020-01-13 13:46:51,404 # Connect to serial port /dev/riot/tty-openmote-b
helpWelcome to pyterm!
Type '/exit' to exit.
help
2020-01-13 13:46:52,619 # help
2020-01-13 13:46:52,621 # Command              Description
2020-01-13 13:46:52,621 # ---------------------------------------
2020-01-13 13:46:52,622 # reboot               Reboot the node
2020-01-13 13:46:52,636 # ps                   Prints information about running threads.
2020-01-13 13:46:52,638 # random_init          initializes the PRNG
2020-01-13 13:46:52,639 # random_get           returns 32 bit of pseudo randomness
2020-01-13 13:46:52,641 # ifconfig             Configure network interfaces
2020-01-13 13:46:52,653 # txtsnd               Sends a custom string as is over the link layer
2020-01-13 13:46:52,655 # saul                 interact with sensors and actuators using SAUL
> reboot
2020-01-13 13:46:53,707 #  reboot
2020-01-13 13:46:53,723 # main(): This is RIOT! (Version: 2020.01-devel-1726-g19774-HEAD)
2020-01-13 13:46:53,724 # Welcome to RIOT!
> reboot
2020-01-13 13:46:54,731 #  reboot
2020-01-13 13:46:54,732 # main(): This is RIOT! (Version: 2020.01-devel-1726-g19774-HEAD)
2020-01-13 13:46:54,747 # Welcome to RIOT!
>

fjmolinas on 13 Jan 2020

@fjmolinas This is the output of the flash command:

### Flashing Target ###
### Flashing at base address 0x200000 with offset 0 ###
SEGGER J-Link Commander V6.60c (Compiled Dec 23 2019 16:18:17)
DLL version V6.60c, compiled Dec 23 2019 16:17:59

J-Link Commander will now exit on Error

J-Link Command File read successfully.
Processing script file...

J-Link connection not established yet but required for command.
Connecting to J-Link via USB...O.K.
Firmware: J-Link EDU Mini V1 compiled Dec 13 2019 11:16:57
Hardware version: V1.00
S/N: 801009716
License(s): FlashBP, GDB
VTref=2.475V
Target connection not established yet but required for command.
Device "CC2538SF53" selected.


Connecting to target via JTAG
InitTarget: Found ICE-Pick with ID: 0x8B96402F
InitTarget: Found CPU TAP 0x4BA00477
Scanning AP map to find all available APs
AP[1]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x412FC230. Implementer code: 0x41 (ARM)
Found Cortex-M3 r2p0, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 002BB000 SCS
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 002BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 002BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 002BB923 TPIU-Lite
Cortex-M3 identified.
Halting CPU for downloading file.
Downloading file [/home/relsas/RIOT-benpicco/examples/interference/bin/openmote-b/interference.bin]...
Comparing flash   [100%] Done.
Erasing flash     [100%] Done.
Programming flash [100%] Done.
Verifying flash   [100%] Done.
J-Link: Flash download: Bank 0 @ 0x00200000: 2 ranges affected (481280 bytes)
J-Link: Flash download: Total time needed: 5.891s (Prepare: 0.061s, Compare: 0.281s, Erase: 0.085s, Program: 5.370s, Verify: 0.075s, Restore: 0.017s)
O.K.


Sleep(100)



Script processing completed.

The example I'm flashing is something I made myself: https://github.com/tinstructor/RIOT/tree/interfere/examples/interference

tinstructor on 13 Jan 2020

You'll need to modify the flash command a little bit though:

PROGRAMMER=jlink make flash BOARD=openmote-b GNRC_NETIF_NUMOF=1

tinstructor on 13 Jan 2020

@tinstructor is your issue reproducible with master?

fjmolinas on 13 Jan 2020

@fjmolinas which application do you suggest I try?

tinstructor on 13 Jan 2020

@fjmolinas which application do you suggest I try?

examples/default or tests/shell since they have reboot

fjmolinas on 13 Jan 2020

@tinstructor I check-outed your branch, still cant reproduce.

last commit: 2479524e589c604b3b15138c48002b745e0cf924

Could you put the output of ./dist/tools/ci/print_toolchain_versions.sh. Also can you use BUILD_IN_DOCKER=1?

fjmolinas on 13 Jan 2020

@fjmolinas I tried examples/default and I still encounter the same issue:

relsas@relsas-VirtualBox:~/RIOT-benpicco/examples/default$ PROGRAMMER=jlink make flash BOARD=openmote-b
Building application "default" for "openmote-b" with MCU "cc2538".

"make" -C /home/relsas/RIOT-benpicco/boards/openmote-b
"make" -C /home/relsas/RIOT-benpicco/core
"make" -C /home/relsas/RIOT-benpicco/cpu/cc2538
"make" -C /home/relsas/RIOT-benpicco/cpu/cc2538/periph
"make" -C /home/relsas/RIOT-benpicco/cpu/cortexm_common
"make" -C /home/relsas/RIOT-benpicco/cpu/cortexm_common/periph
"make" -C /home/relsas/RIOT-benpicco/drivers
"make" -C /home/relsas/RIOT-benpicco/drivers/periph_common
"make" -C /home/relsas/RIOT-benpicco/drivers/saul
"make" -C /home/relsas/RIOT-benpicco/sys
"make" -C /home/relsas/RIOT-benpicco/sys/auto_init
"make" -C /home/relsas/RIOT-benpicco/sys/auto_init/saul
"make" -C /home/relsas/RIOT-benpicco/sys/fmt
"make" -C /home/relsas/RIOT-benpicco/sys/isrpipe
"make" -C /home/relsas/RIOT-benpicco/sys/newlib_syscalls_default
"make" -C /home/relsas/RIOT-benpicco/sys/phydat
"make" -C /home/relsas/RIOT-benpicco/sys/ps
"make" -C /home/relsas/RIOT-benpicco/sys/saul_reg
"make" -C /home/relsas/RIOT-benpicco/sys/shell
"make" -C /home/relsas/RIOT-benpicco/sys/shell/commands
"make" -C /home/relsas/RIOT-benpicco/sys/stdio_uart
"make" -C /home/relsas/RIOT-benpicco/sys/tsrb
   text    data     bss     dec     hex filename
  14936     136    2620   17692    451c /home/relsas/RIOT-benpicco/examples/default/bin/openmote-b/default.elf
/home/relsas/RIOT-benpicco/dist/tools/jlink/jlink.sh flash /home/relsas/RIOT-benpicco/examples/default/bin/openmote-b/default.bin
### Flashing Target ###
### Flashing at base address 0x200000 with offset 0 ###
SEGGER J-Link Commander V6.60c (Compiled Dec 23 2019 16:18:17)
DLL version V6.60c, compiled Dec 23 2019 16:17:59

J-Link Commander will now exit on Error

J-Link Command File read successfully.
Processing script file...

J-Link connection not established yet but required for command.
Connecting to J-Link via USB...O.K.
Firmware: J-Link EDU Mini V1 compiled Dec 13 2019 11:16:57
Hardware version: V1.00
S/N: 801009716
License(s): FlashBP, GDB
VTref=2.476V
Target connection not established yet but required for command.
Device "CC2538SF53" selected.


Connecting to target via JTAG
InitTarget: Found ICE-Pick with ID: 0x8B96402F
InitTarget: Found CPU TAP 0x4BA00477
Scanning AP map to find all available APs
AP[1]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x412FC230. Implementer code: 0x41 (ARM)
Found Cortex-M3 r2p0, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 002BB000 SCS
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 002BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 002BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 002BB923 TPIU-Lite
Cortex-M3 identified.
Halting CPU for downloading file.
Downloading file [/home/relsas/RIOT-benpicco/examples/default/bin/openmote-b/default.bin]...
Comparing flash   [100%] Done.
Erasing flash     [100%] Done.
Programming flash [100%] Done.
Verifying flash   [100%] Done.
J-Link: Flash download: Bank 0 @ 0x00200000: 1 range affected (47104 bytes)
J-Link: Flash download: Total time needed: 2.507s (Prepare: 0.060s, Compare: 1.367s, Erase: 0.513s, Program: 0.541s, Verify: 0.005s, Restore: 0.017s)
O.K.


Sleep(100)



Script processing completed.

relsas@relsas-VirtualBox:~/RIOT-benpicco/examples/default$ make term PORT=/dev/ttyUSB1 BOARD=openmote-b
/home/relsas/RIOT-benpicco/dist/tools/pyterm/pyterm -p "/dev/ttyUSB1" -b "115200" 
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2020-01-13 13:57:48,533 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
reboot
2020-01-13 13:57:51,581 # reboot

tinstructor on 13 Jan 2020

The output of ./dist/tools/ci/print_toolchain_versions.sh is:

relsas@relsas-VirtualBox:~/RIOT-benpicco$ ./dist/tools/ci/print_toolchain_versions.sh

Operating System Environment
-----------------------------
       Operating System: "Ubuntu" "18.04.3 LTS (Bionic Beaver)"
                 Kernel: Linux 5.0.0-37-generic x86_64 x86_64

Installed compiler toolchains
-----------------------------
             native gcc: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
      arm-none-eabi-gcc: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 8-2018-q4-major) 8.2.1 20181213 (release) [gcc-8-branch revision 267074]
                avr-gcc: missing
       mips-mti-elf-gcc: missing
             msp430-gcc: missing
   riscv-none-embed-gcc: missing
   xtensa-esp32-elf-gcc: missing
   xtensa-lx106-elf-gcc: missing
                  clang: missing

Installed compiler libs
-----------------------
   arm-none-eabi-newlib: "3.0.0"
    mips-mti-elf-newlib: missing
riscv-none-embed-newlib: missing
xtensa-esp32-elf-newlib: missing
xtensa-lx106-elf-newlib: missing
               avr-libc: missing (missing)

Installed development tools
---------------------------
                  cmake: cmake version 3.10.2
               cppcheck: missing
                doxygen: 1.8.13
                    git: git version 2.17.1
                   make: GNU Make 4.1
                openocd: Open On-Chip Debugger 0.10.0+dev-00932-g85a460d5 (2019-09-23-11:58)
                 python: Python 2.7.17
                python2: Python 2.7.17
                python3: Python 3.6.8
                 flake8: error: /usr/bin/python3: No module named flake8
             coccinelle: missing

tinstructor on 13 Jan 2020

@aabadie can you give this a try and reproduce?

fjmolinas on 13 Jan 2020

@fjmolinas I remember that I also had to put CCA_BACKDOOR_ENABLE to 0 (in boards/openmote-b/include/board.h) in order to unbrick the device after a previous person had used it. However, ever since I've put it back to 1. It might be that I've mistakenly left it unchanged for the other nodes (they are all reflashed with the parameter set to 1 as of today though) and so that might've affected something?

/**
 * @name    Flash Customer Configuration Area (CCA) parameters
 * @{
 */
#ifndef UPDATE_CCA
#define UPDATE_CCA                (1)
#endif

#define CCA_BACKDOOR_ENABLE       (1)
#define CCA_BACKDOOR_PORT_A_PIN   (6) /**< BSL_BOOT Pin */
#define CCA_BACKDOOR_ACTIVE_LEVEL (0) /**< Active low */

tinstructor on 13 Jan 2020

@fjmolinas The issue seems to be gone when I put CCA_BACKDOOR_ENABLE to 0 again. However that's not supposed to be how that works I think.

tinstructor on 13 Jan 2020

@fjmolinas The issue seems to be gone when I put CCA_BACKDOOR_ENABLE to 0 again. However that's not supposed to be how that works I think.

I'll try to look into this more.. I don't have many ideas right now.

fjmolinas on 13 Jan 2020

👍1

I have struggled with the boot and nrst pins for a while now. I am pretty busy but after Wednesday I can dive deeper into it.

I will note that lockup does occur on some platforms when the nrst pin is held high as the program tries to pull the nrst down when rebooting and waits until it reads low (forever).

The nrst pin is controlled by an additional msp430 chip that is intended to standardize the signals from the ftdi RTS and DTR pins used to set the boot0 and nrst for bootloading over usb.

It may be something that in RIOT that has it initial state a push-pull instead of an open drain output with a pullup (as it should be for nrst pins).

If you guys don't solve it by the time I am free I will jump on it.