Tock: Tracking: Allow Apps to Fault and Restart Successfully

[bradjc]

Having apps separate from the kernel and able to fail without crashing the system is a core design goal for Tock. Currently, however, faulty apps do crash the entire board. This tracking issue outlines what needs to be done to allow apps to fail and have the kernel restart them successfully.

• [x] Implement the logic for FaultResponse::Restart in the kernel. (#941)
  
  
  
  • [x] Update to match changes (#1529)
  
  
  • [x] Separate process array index from identifier (#1566)
  
  
  • [x] Implement a process identifier allocation scheme so that restarted processes get different identifiers. (#1588)
  
  
• [x] Add an app fault option that just stops the app on a fault. (#1238)
• [x] Create a test capsule that can artificially restart otherwise working apps. (#957)
  
  
  
  • [x] Implement capabilities so not just any capsule can restart apps. (#975)
  
  
• [x] Verify that all capsules respond correctly to an app being restarted (and having its grant region reset).
  
  
  
  • [x] Blink (timer + LED)
  
  
  • [x] hello_loop (timer + console)
  
  
  • [x] adc: fails (#1525, #1609)
  
  
  • [x] ble serialization: fails (#1612)
  
  
  • [ ] If an IPC service restarts, the client(s) do not know and do not reconfigure themselves.
  
  
• [x] Implement tracking and a policy for apps that fail too quickly or too often. (#1565)
• [x] Add a fault command to process console to specifically fault certain processes.

[bradjc]

I've had a branch https://github.com/tock/tock/tree/adc-grant-apps with a re-write of the ADC for a while now, but I got too ambitious and tried to refactor it as well, but all of the buffers and possibilities make it difficult without breaking backwards compatibility.

[bradjc]

If someone is willing to add grant region support to the ADC capsule driver that would really help with this effort.

[bradjc]

I think this one is getting pretty close, just needs some PRs to be merged, and some code in branches to become PRs.

Also, I'm planning on scoping this issue to enabling apps to restart without the kernel being in some odd state when they do. This does not mean that all apps will restart and behave exactly as they did when the board first booted. For example, the ADC capsule only supports one app at a time. So, if an app restarts and tries to use the ADC capsule, the restarted app will look the same as if a second app is trying to use the capsule, and the capsule will return an error. The kernel is in an expected state, and if the app waits and tries again to use the capsule it should be fine. The root cause is that the capsule is not virtualized properly, which is a separate issue.

[bradjc]

I've verified that with all of the active PRs merged, the following apps work on hail after being manually faulted with the process console:

• adc
• adc_continuous
• ambient_light
• button_print
• callback_remove_test01
• console
• console_recv_long
• console_recv_short
• console_timeout
• crash_dummy
• crc
• fxos8700cq
• gpio
• hail
• hello_loop
• malloc_test01
• mpu
• mpu_stack_growth
• mpu_walk_region
• multi_alarm_test
• number_guess_game
• printf_long
• rng
• stack_size_test01
• stack_size_test02
• whileone
• accel-leds
• blink
• buttons
• c_hello
• cxx_hello
• find_north
• lua-hello
• sensors

I managed to fix and open PRs for all that didn't work on the first try. The other test apps are not supported on hail.

[bradjc]

I'm calling this done! Tock now supports restarting applications!

But, it is up to capsule authors to use grants correctly to ensure that if applications restart or go away then their resources in the capsule are freed properly. There may still need to be fixes in the future to make sure this happens correctly.

Also, there is no provision in the current API for IPC to notify either a server or a client if one or the other disappears. This is a larger change that is outside of the scope of this tracking issue.

[alevy]

Also, there is no provision in the current API for IPC to notify either a server or a client if one or the

Yeah, IPC probably needs a redesign anyway. I agree this is out of scope for this issue.