Riot: RFC: lower network stack rework

[jia200x]

Description

I'm opening this issue to synchronize and keep track of the netif rework.

Motivation and related issues

Mixed transceiver, PHY and MAC logic in netdev implementations and hardware

From the documentation, netdev acts as an interface to "provide a uniform API for network stacks to interacts with network device drivers".
In practice, the network stack interacts with an interface (e.g gnrc_netif) and then there's mixed transceiver, PHY and MAC layer logic.

Some consequences of this:

• Some device drivers already include PHY layer indications. E.g see https://github.com/RIOT-OS/RIOT/blob/03c467af75c7257fa9597a9f0539055a8dc3044e/cpu/esp32/esp-wifi/esp_wifi_netdev.c#L169
  In this case, the device driver already calls a function with the packet. In order to pass it to the network stack, it's necessary to generate a fake ISR event and an extra copy.
• Without a clear distinction between transceiver logic (e.g interrupt from the radio and functions to access the framebuffer) and PHY layer (send and receive data), we need to tell radios how to "read_pkt_size" or "drop" packets. Or sometime "preload" before sending. In practice all radios "preload" and then "trigger send", but we need to add the "send" semantics in the radios. Besides being more complex, it's sometimes inefficient (see https://github.com/RIOT-OS/RIOT/issues/11652)
• Network device drivers have PHY state changes that should be handled by the MAC layers. E.g at86rf2xx radios go back to the idle_state after sending. This behavior is not valid e.g in TSCH
• We cannot process ACK packets if the radio doesn't support retransmissions (see https://github.com/RIOT-OS/RIOT/issues/7304#issuecomment-314375839)
• And we have to implement ACK handling or software CSMA on each radio, although the logic is the same (see https://github.com/RIOT-OS/RIOT/pull/11150 or https://github.com/RIOT-OS/RIOT/pull/8332)
• Upper layers expect radios running on Extended Mode (auto ACK, retransmissions, etc). Radios running in Basic Mode won't work properly unless they implement Extended Mode features (see e.g https://github.com/RIOT-OS/RIOT/issues/8213)

Related work:
https://github.com/RIOT-OS/RIOT/issues/7736

There is one thread per (gnrc_)netif

It's needed to allocate one gnrc_netif_t thread per network interface. Besides allocating more resources than needed, this is sometimes problematic for platforms that have several transceivers.

See https://github.com/RIOT-OS/RIOT/issues/10496

Netif relays on IPC messages for handling ISR events, send and receive.

• Race conditions might occur if a single queue process ISR and send events (see #11256 )
• Interrupts can be lost using this mechanism (see https://github.com/RIOT-OS/RIOT/issues/5486)

Related work:

9326

Network stacks don't share initialization logic, ISR processing and network interfaces

Each stack needs to handle radio events, auto_init logic and MAC layers.

Some consequences of this:

• We cannot use GNRC on top of LoRaWAN (at least without #11022), or use gnrc_netif_ieee802154 code for LWIP or emb6. It's also not posible to use the OpenThread lower layers (IEEE802.15.4 with security, plus joiner and commissioner roles) with GNRC.
• Some stacks are network device dependent. E.g it's not possible to use other radios than at86rf2xx in OpenThread. It's similar in LWIP.

Outcome

Separate transceiver, PHY and MAC logic

If we can provide interfaces between this components we could benefit of:

• A lot of code re utilization.
• Simpler network device drivers (e.g that only write transceiver logic instead of transceiver+PHY+MAC logic).
• A better testing infrastructure for network devices and their layers.
• A better interface with external code that already include a PHY or MAC layer

For IEEE802.15.4, the PHY layer can be implemented as a SubMAC layer (see #13376 ) in order to take advantage of the interface of most device drivers (MAC hw accelerations already included in the device).

Move auto-initialization and transceiver ISR logic out of the network interface

Moving the initialization and transceiver ISR logic out of the interface allows as to reuse code and immediately extend the support for more radios in LWIP, OpenThread, etc.

With this we could e.g get rid of these lines since the device initialization doesn't depend of the stack.

Remove the need of allocating one thread per interface

Interfaces can be represented with pointers. All events can be handled by only one thread or OS mechanism (e.g see https://github.com/RIOT-OS/RIOT/pull/12474).

Improve support of software MAC layers

We shouldn't worry if a radio doesn't support Auto-ACK, retransmissions, etc. Also, we could have more powerful MAC layers (IEEE802.15.4 with security, pan coordinators, etc).

Write network stack independent network interfaces (see https://github.com/RIOT-OS/RIOT/issues/12688#issuecomment-587082089)

This means the network interface is handled by the OS and not by the network stack. Network stacks can then reuse link layer logic (MAC, upper PHY). With this we can have a more uniform experience between different network stacks.

---

Proposed roadmap

This whole rework can be done in the following phases

1. Improve Link Layer support

• [ ] IEEE802.15.4 SubMAC
  
  
  
  • [ ] Add support for radio caps (https://github.com/RIOT-OS/RIOT/pull/11473)
  
  
  • [ ] Add support for common link layer features (Auto ACK, retransmissions, etc)
  
  
• [ ] IEEE802.15.4 MAC
  
  
  
  • [ ] L2 security
  
  

2. Add required interfaces PHY and HAL interfaces

• [ ] Extensibility of netdev (#12469)
  TBD

3. Rework and extend the netif API (TO BE REVISED, see https://github.com/RIOT-OS/RIOT/issues/12688#issuecomment-587082089)

This step is required in order to provide a network stack independent netif.

• [x] Make netif_t a pointer instead of a stack defined type (#11879)
• [ ] Define an interface for stack independent packet allocation, data handling and passing data up to the stack
  
  
  
  • [ ] Netbuf: #12691
  
  
• [ ] Extend the netif API to add send/recv operations (analog to sock, but intended to be used from network stacks or applications that send data via an interface)
• [ ] Migrate common code from gnrc_netif to netif
  
  
  
  • [ ] Refactor MAC layers to make them independent of GNRC (e.g gnrc_pktbuf dependencies in gnrc_netif_xxx functions)
  
  
  • [ ] Move gnrc_netif events to external event handlers + callbacks.
  
  
  • [ ] Move gnrc_netif_ops_t ops into netif_ops_t
    
    I will open a Github project to be able synchronize better.
    
    
    

    Useful links

  https://github.com/RIOT-OS/RIOT/issues/11483

  11879

  https://github.com/RIOT-OS/RIOT/pull/11473

  This is intended to be addressed to: https://github.com/RIOT-OS/RIOT/issues/4876

[jia200x]

@RIOT-OS/maintainers there have been some offline conversation with some maintainers (@haukepetersen, @bergzand, @kaspar030, @miri64, @leandrolanzieri, @PeterKietzmann).

Unfortunately we didn't have time to discuss this during the RIOT summit, so I open this issue for syncing.

All kind of feedback on the fundamentals or roadmap is more than welcome!

[maribu]

Also related to this: https://github.com/RIOT-OS/RIOT/issues/12469

[jia200x]

@maribu added to the road map

[miri64]

Define an interface for stack independent packet allocation, data handling and passing data up to the stack
[…]

• [ ] Remove GNRC dependencies in MAC layers (e.g gnrc_pktbuf dependencies in gnrc_netif_xxx functions)

I thought we figured last week, that this makes things more complicated than it need be…

[jia200x]

I thought we figured last week, that this makes things more complicated than it need be…

What I meant with that was: make the MAC layer component independent of GNRC

[jia200x]

@miri64 rephrased :)

[roberthartung]

12128 would probably benefit from this as well.

[jia200x]

I renamed it to "lower network stack" rework, since the network interface are only one of the components involved

[jia200x]

I would like to share some thoughts about the "Write network stack independent network interfaces" part:

After some time, I think it's different to distinguish these 2 elements:

1. The OS representation of a network interface
2. The glue code between a network stack and the OS.

For 1., it's clear that the representation of a network interface should be agnostic to the network stack (e.g running ifconfig should work with all network stacks, same as sock). We already have most of this functions (' netif_iter,netif_get_opt`, etc).

However, 2. doesn't necessarily need a unique entry point (e.g netdev_recv and netif_send function) or a global allocator (netbuf). Here are some of the reasons:

1. Some network stacks require access to the radio (OpenWSN), some other to the PHY/SubMAC layer (OpenThread, LWIP) or even on top of the Link Layer. So, not all network stacks would make use of the netif_recv and netif_send function. Only access to a dedicated layer is needed.
2. Some MAC layers and drivers already have Framebuffers. Those who don't, it's always possible to design MAC layer allocators (e.g for Ethernet) or simply write one function per stack to allocate a packet (in the RECV function of the HAL).
3. Device ISR handling can be done by an (optional) event_thread module (#12474 ), and this logic can be used by all Network Stacks. This would simplify much more the integration of network stacks

And last but not least, solving the netif problem first doesn't solve the problems of radios with different caps and lack of MAC layers, so I would probably give more priority to the devices/link layer rework.

Any comments?

[jia200x]

To make this easier to follow, I will split this rework in 2 independent reworks:

1. Improve GNRC Netif
2. Network stack independent link layers (this also includes HALs and mechanisms to process IRQs)

[jia200x]

this is reorganized in #13771 and #13763 , so I will close the issue.