Jormungandr: Error: deadline has elapsed

[Rugbertl]

Describe the bug
Node not responding on ./jcli rest v0 node stats get -h "http://127.0.0.1:3100/api"

Mandatory Information

1. jcli 0.8.3 (HEAD-8f276c0, release, linux [x86_64]) - [rustc 1.39.0 (4560ea788 2019-11-04)]
2. jormungandr 0.8.3 (HEAD-8f276c0, release, linux [x86_64]) - [rustc 1.39.0 (4560ea788 2019-11-04)]

To Reproduce
Start node and wait couple of hours.

Expected behavior
Respond with node status.

Additional context
Dec 18 11:31:55.732 ERRO cannot process block announcement, reason: deadline has elapsed, from_node_id: 1a491553b176cec46b88679e2a5acc8d5f5c96f5b6546378, date: 4.27542, parent: 6b67ae68149833b689d6334f09ebb5b73322b8c02b53d35ffb9538f690fcb9f8, hash: afe8d290fae7015e36af7961d82d78536611fe60e8725fe344aae490b9e35641, task: block
Dec 18 11:32:02.311 ERRO cannot process block announcement, reason: deadline has elapsed, from_node_id: 731215fc3f5f48e218a839f0a221acaa8fbe781c959399f6, date: 4.27550, parent: afe8d290fae7015e36af7961d82d78536611fe60e8725fe344aae490b9e35641, hash: ef515b12d08b0bd331028961e76b3bfbb7cd0136df24f7f45027176255093718, task: block

[rinor]

Do you have also the explorer enabled?

[Rugbertl]

Explorer is not enabled.

[arussell29]

I am seeing this error many times as well. I do not have the explorer enabled either.

[rinor]

Ok, so the situation is the following:

• cannot process block announcement, reason: deadline has elapsed ..
  
  
  
  • present
  
  
• cannot process network headers, reason: deadline has elapsed, task: block
  
  
  
  • may or may not be present (but usually is)
  
  
• blockchain is not moving up...stuck_notifier
  
  
  
  • present
  
  
• cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier
  
  
  
  • may or may not be present (but usually is and depends on the state of the chain before this issue kicks in)
  
  
• jcli rest v0 node stats get
  
  
  
  • ~timeout (curl as well)~
  
  
• jcli rest v0 block XXXX get
  
  
  
  • ~timeout (curl as well)~
  
  

At the same time other rest commands get replied, ex:

• jcli rest v0 tip get - works
• jcli rest v0 network stats get - works
• jcli rest v0 settings get - works
• jcli rest v0 account get - works
• ....

I will save the logs once done, since I'm keeping the node up in this state in case I can get further details.

Update 2020-01-03: Rest timeouts are fixed and were related to storage locks issue fixed in https://github.com/input-output-hk/jormungandr/pull/1412 and another one will be merged https://github.com/input-output-hk/jormungandr/pull/1441

[rcmorano]

Not sure if related, but I never got this until I published my peer and a lot of people started bootstrapping from it.

[Rugbertl]

Just found on v0.8.4

Dec 19 21:31:22.442 WARN response canceled by the processing task, request: PushHeaders, sub_task: server, task: network
Dec 19 21:31:58.346 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 19 21:31:58.355 WARN response canceled by the processing task, request: PushHeaders, sub_task: server, task: network

[rinor]

similar/related to https://github.com/input-output-hk/jormungandr/issues/1413 and https://github.com/input-output-hk/jormungandr/issues/1320.