Openthread: [mle] device can't merge for extended period of time

[LuDuda]

Hello,

During some intensive stress tests, I observed very strange behaviour, where two devices for some reason created two partitions and couldn't merge (i believe someone reported similar issue on Google Groups some time ago). After some more investigation, it seems I found the root cause. I hit this on some previous revision, but it can be reproduced on the master branch as well.

Unexpected behaviour

Two FTD devices seem to ignore Parent Requests or Link Requests and create two partitions, after some time they merge.

Root cause

In case the Router A decides to remove a neighbor Router X due to not responding 4 times for some application data (see MeshForwarder::UpdateNeighborOnSentFrame(), and this removed Router happens to be the Leader, the Router A stops responding on Parent Requests (for all devices) and Link Requests (for Leader) till realize there is no Leader in the network (Network ID timeout).

How to reproduce

Leader

panid 0
ifconfig up
thread start

Router

panid 0
ifconfig up
thread start
state router
ping LEADER_ML-EID

Leader

^Ctrl + C

Router

ping LEADER_ML-EID 10 5

Leader

ifconfig up
thread start

Two partitions will be created and merged after less than 120 seconds. In this time, no MTD devices could attach to the Router as well.

Merge of partition can't happen probably because of this check in HandleParentRequest.
Link Requests are not processed probably because of this check in HandleLinkRequest.

Logs from the Router:

[INFO]-MLE-----: Receive Parent Request (fe80:0:0:0:4c1f:89e8:1ba2:1146)
[WARN]-MLE-----: Failed to process Parent Request: Drop

@jwhui @abtink could you help to resolve this problem or point on a possible solution?

[jwhui]

@LuDuda, thank you for raising this issue and narrowing it down!

I agree with your analysis.

The check in HandleParentRequest() follows the Thread Specification:

• _"A Router or REED MUST NOT send an MLE Parent Response if ... its current routing path cost to the Network Partition Leader is infinite."_

However, I think the check in HandleLinkRequest() is too strict. The Thread Specification simply states:

• _"Neighboring Routers respond only if they have a valid Router ID assignment for the sender."_

As a result, the router need only check that the Router ID is valid - not that the neighbor is a valid neighbor as it currently does.

I'll work to propose a fix.

[LuDuda]

Jonathan, thank you for taking care of it!

Certainly, the resolution of the problem with the MLE Link Request is important, it would not entirely solve all the cases.

When I found the problem, from logs I do see that Router device returned NoBufs error in MeshForwarded layer when receiving the Link Request. This is due to the fact that Host flooded NCP with application packets to the Leader. Packets landed in the EID-to-RLOC resolving queue. This is another problem we may try to address (always have some buffers to receive frames).

Even if above scenario, didn't happen, still Router could simply miss Link Request, which would mean that in my scenario network is not operable for up to 120 seconds.

Thus I'm more concerned about Parent Requests. Do you think this is the limitation of the Thread Specification, which we can't address in the code?

[jwhui]

I just submitted #4753 to address the immediate problem of rejecting MLE Link Request messages.

Certainly, the resolution of the problem with the MLE Link Request is important, it would not entirely solve all the cases.

When I found the problem, from logs I do see that Router device returned NoBufs error in MeshForwarded layer when receiving the Link Request. This is due to the fact that Host flooded NCP with application packets to the Leader. Packets landed in the EID-to-RLOC resolving queue. This is another problem we may try to address (always have some buffers to receive frames).

I agree. We already have a way to evict queued messages in MeshForwarder based on message priority (see MeshForwarder::EvictMessage()). However, we need to extend similar functionality to include other queues. I can take a look at this as well.

Even if above scenario, didn't happen, still Router could simply miss Link Request, which would mean that in my scenario network is not operable for up to 120 seconds.

Thus I'm more concerned about Parent Requests. Do you think this is the limitation of the Thread Specification, which we can't address in the code?

The Parent Request constraint is a limitation of the Thread Specification. Maybe we should at least consider increasing the number of attempts on sending the initial Link Request message after reset. Part of the challenge is when a router-capable device should give up and form its own partition.

[jwhui]

@LuDuda, just submitted #4776 to support evicting messages from the address resolver queue. Please review.

[LuDuda]

@jwhui Thank you for your time on helping with the issue! This is so great! I reviewed both PRs.

Regarding your last paragraph about retrying the MLE Link Request, we think about increasing the robustness of multicast MLE control messages in general (i.e. Link Request or Advertisements) by introduce blind retransmissions for them, in scenario where packet losts are more frequent. I know that OT has a feature to retransmit all broadcast messages, but that seems to be to much overhead. We may propose some solution for that soon. Though I'm not sure Thread Certification Harness will like it ;)

Anyway, I believe we can close this issue now as resolved! Thanks!

[jwhui]

@jwhui Thank you for your time on helping with the issue! This is so great! I reviewed both PRs.

Thanks again for raising this issue with detailed description and your reviews on the PRs!

Regarding your last paragraph about retrying the MLE Link Request, we think about increasing the robustness of multicast MLE control messages in general (i.e. Link Request or Advertisements) by introduce blind retransmissions for them, in scenario where packet losts are more frequent. I know that OT has a feature to retransmit all broadcast messages, but that seems to be to much overhead. We may propose some solution for that soon. Though I'm not sure Thread Certification Harness will like it ;)

Sounds good!

Anyway, I believe we can close this issue now as resolved! Thanks!

Thanks.