Jormungandr: Bootstrap failed

Created on 17 Jan 2020 · 7Comments · Source: input-output-hk/jormungandr

Get this below bootstrap error , and also sometimes bootstraps stucks and have to restart jormungandr many times for successful bootstrap.

Jan 17 09:58:29.675 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 52.8.15.52:3000, task: bootstrap
Jan 17 09:58:33.135 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 3.125.75.156:3000, task: bootstrap
Jan 17 09:59:48.507 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 13.114.196.228:3000, task: bootstrap
Jan 17 09:59:49.364 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.9.132.248:3000, task: bootstrap

Mandatory Information

0.8.6 release

config file
{
"log": [
{
"format": "plain",
"level": "warn",
"output": "stderr"
}
],
"storage": "./storage/",
"p2p": {
"listen_address": "/ip4/0.0.0.0/tcp/3000",
"public_address": "**/tcp/3000",
"public_id": "****",
"max_connections": 300 ,
"max_unreachable_nodes_to_connect_per_event": 10,

"topics_of_interest": {
  "blocks": "high",
  "messages": "high"
},
"trusted_peers": [

  {
    "address": "/ip4/13.114.196.228/tcp/3000",
    "id": "7e1020c2e2107a849a8353876d047085f475c9bc646e42e9"
  },
  {
    "address": "/ip4/13.112.181.42/tcp/3000",
    "id": "52762c49a84699d43c96fdfe6de18079fb2512077d6aa5bc"
  }
  ,
  {
    "address": "/ip4/3.124.116.145/tcp/3000",
    "id": "99cb10f53185fbef110472d45a36082905ee12df8a049b74"
  }
  ,
  {
    "address": "/ip4/3.125.75.156/tcp/3000",
    "id": "22fb117f9f72f38b21bca5c0f069766c0d4327925d967791"
  }
  ,
  {
    "address": "/ip4/52.28.91.178/tcp/3000",
    "id": "23b3ca09c644fe8098f64c24d75d9f79c8e058642e63a28c"
  }
  ,
  {
    "address": "/ip4/52.8.15.52/tcp/3000",
    "id": "18bf81a75e5b15a49b843a66f61602e14d4261fb5595b5f5"
  } ,
  {
    "address": "/ip4/52.9.132.248/tcp/3000",
    "id": "671a9e7a5c739532668511bea823f0f5c5557c99b813456c"
  }
]

},
"rest": {
"listen": "127.0.0.1:3100"
}

}

Source

BitcoinMicroPaymentNetwork

👍2

Most helpful comment

Same in 0.8.18-690eb547.

Steps:

Bootstrapped once successfully with storage.
Set the VM to reboot and immediately start jormungandr on boot (as a service, ran by the same user - root).

Problem:

Check syslog

WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }

It looks to me like it's trying to pull data from previously connected nodes, but all of them are disconnected or something. If I can't reconnect after a reboot, this is a serious blocker.

milosmns on 26 Apr 2020

👍2 😕1

All 7 comments

I also agree with the above difficulties to complete a successful bootstrap in under 10 minutes. I update the node-config.yaml peers at every start of jormungandr from the latest available node-config.yaml example online. Yet, it seems impossible to guarantee a swift bootstrap phase, leading to multiple reboots/restarts (because what other options do we have?).

dmindfuse on 17 Jan 2020

Fortunately now that it does not crash anymore we don't need to bootstrap every 30 minutes after automatically restarting my nodes... but yeah it is an issue, an official list of the trusted servers should be provided by the Cardano foundation and jormungandr should download these at start.

ghost on 17 Jan 2020

👎1

See #1582 and #1612

michaeljfazio on 21 Jan 2020

👍1

still happening on 0.8.9

without storage - bootstrapping is imposible

Feb 03 08:52:07.731 INFO Starting jormungandr 0.8.9 (HEAD-30d20d2e, release, windows [x86_64]) - [rustc 1.40.0 (73528e339 2019-12-16)], task: init
Feb 03 08:52:07.745 WARN Node started without path to the stored secret keys (not a stake pool or a BFT leader), task: init
jormungandr ---config node_config.yaml --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676
Feb 03 08:49:07.469 INFO Starting jormungandr 0.8.9 (HEAD-30d20d2e, release, windows [x86_64]) - [rustc 1.40.0 (73528e339 2019-12-16)], task: init
Feb 03 08:49:07.475 WARN Node started without path to the stored secret keys (not a stake pool or a BFT leader), task: init
Feb 03 08:49:07.492 INFO storing blockchain in '"./storage\\blocks.sqlite"', task: init
Feb 03 08:49:07.503 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 52.9.132.248:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:28.513 WARN unable to reach peer for block download, reason: HTTP/2.0 connection error, peer_address: 52.9.132.248:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:28.533 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 3.125.75.156:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:28.589 WARN failed to download block, error: GetBlocks { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_address: 3.125.75.156:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:28.596 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 52.8.15.52:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:29.031 WARN failed to download block, error: GetBlocks { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_address: 52.8.15.52:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:29.039 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 13.114.196.228:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:30.289 WARN failed to download block, error: GetBlocks { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_address: 13.114.196.228:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:30.314 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 3.124.116.145:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:51.295 WARN unable to reach peer for block download, reason: HTTP/2.0 connection error, peer_address: 3.124.116.145:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:51.314 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 52.28.91.178:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:51.381 WARN failed to download block, error: GetBlocks { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_address: 52.28.91.178:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:51.387 INFO fetching block 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, peer_address: 13.112.181.42:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Feb 03 08:49:51.998 WARN failed to download block, error: GetBlocks { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_address: 13.112.181.42:3000, block: 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676, task: init
Error fetching the genesis block from the network
 |-> could not download block hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676

with storage, it can take hours to bootstrap

Feb 03 08:44:45.429 INFO connecting to bootstrap peer 3.124.116.145:3000, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:45:06.433 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:45:06.442 INFO connecting to bootstrap peer 13.114.196.228:3000, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:45:09.513 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:45:09.520 INFO connecting to bootstrap peer 13.112.181.42:3000, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:45:10.311 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:45:10.320 INFO connecting to bootstrap peer 52.28.91.178:3000, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:45:10.420 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:45:10.428 INFO connecting to bootstrap peer 3.125.75.156:3000, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:45:10.494 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:45:10.500 INFO connecting to bootstrap peer 52.8.15.52:3000, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:45:10.933 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:45:10.938 INFO connecting to bootstrap peer 52.9.132.248:3000, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:45:31.937 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:45:31.947 INFO bootstrap attempt #1 failed, trying again in 5 seconds..., task: bootstrap
Feb 03 08:45:36.937 INFO connecting to bootstrap peer 13.112.181.42:3000, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:45:38.014 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:45:38.020 INFO connecting to bootstrap peer 52.28.91.178:3000, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:45:38.168 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:45:38.174 INFO connecting to bootstrap peer 13.114.196.228:3000, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:45:39.335 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:45:39.343 INFO connecting to bootstrap peer 3.124.116.145:3000, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:46:00.344 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:46:00.355 INFO connecting to bootstrap peer 52.8.15.52:3000, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:46:00.850 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:46:00.857 INFO connecting to bootstrap peer 52.9.132.248:3000, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:46:21.857 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:46:21.876 INFO connecting to bootstrap peer 3.125.75.156:3000, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:46:21.932 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:46:21.938 INFO bootstrap attempt #2 failed, trying again in 5 seconds..., task: bootstrap
Feb 03 08:46:26.933 INFO connecting to bootstrap peer 3.124.116.145:3000, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:46:47.940 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:46:47.951 INFO connecting to bootstrap peer 13.114.196.228:3000, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:46:50.372 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:46:50.376 INFO connecting to bootstrap peer 52.9.132.248:3000, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:47:11.377 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:47:11.385 INFO connecting to bootstrap peer 13.112.181.42:3000, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:47:12.213 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:47:12.220 INFO connecting to bootstrap peer 52.8.15.52:3000, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:47:12.580 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:47:12.586 INFO connecting to bootstrap peer 52.28.91.178:3000, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:47:12.662 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:47:12.667 INFO connecting to bootstrap peer 3.125.75.156:3000, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:47:12.739 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:47:12.750 INFO bootstrap attempt #3 failed, trying again in 5 seconds..., task: bootstrap
Feb 03 08:47:17.739 INFO connecting to bootstrap peer 52.9.132.248:3000, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:47:38.743 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 52.9.132.248:3000, task: bootstrap
Feb 03 08:47:38.760 INFO connecting to bootstrap peer 3.125.75.156:3000, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:47:38.938 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:47:38.945 INFO connecting to bootstrap peer 52.28.91.178:3000, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:47:39.215 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:47:39.223 INFO connecting to bootstrap peer 13.114.196.228:3000, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:47:40.004 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:47:40.012 INFO connecting to bootstrap peer 13.112.181.42:3000, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:47:41.021 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:47:41.034 INFO connecting to bootstrap peer 3.124.116.145:3000, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:48:02.028 WARN unable to reach peer for initial bootstrap, reason: HTTP/2.0 connection error, peer_addr: 3.124.116.145:3000, task: bootstrap
Feb 03 08:48:02.032 INFO connecting to bootstrap peer 52.8.15.52:3000, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:48:02.396 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:48:02.402 INFO bootstrap attempt #4 failed, trying again in 5 seconds..., task: bootstrap
Feb 03 08:48:07.396 INFO connecting to bootstrap peer 13.114.196.228:3000, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:48:08.984 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.114.196.228:3000, task: bootstrap
Feb 03 08:48:08.989 INFO connecting to bootstrap peer 52.28.91.178:3000, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:48:09.072 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.28.91.178:3000, task: bootstrap
Feb 03 08:48:09.080 INFO connecting to bootstrap peer 52.8.15.52:3000, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:48:09.580 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 52.8.15.52:3000, task: bootstrap
Feb 03 08:48:09.587 INFO connecting to bootstrap peer 13.112.181.42:3000, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:48:10.313 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 13.112.181.42:3000, task: bootstrap
Feb 03 08:48:10.322 INFO connecting to bootstrap peer 3.125.75.156:3000, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:48:10.441 WARN initial bootstrap failed, error: PullRequestFailed { source: Error { code: Unknown, source: Status { code: Unknown, message: "connection error: broken pipe" } } }, peer_addr: 3.125.75.156:3000, task: bootstrap
Feb 03 08:48:10.446 INFO connecting to bootstrap peer 52.9.132.248:3000, peer_addr: 52.9.132.248:3000, task: bootstrap

logs debug
node_logs.log

dorin100 on 3 Feb 2020

👍1

Same problem here for 0.8.11

mach-diamond on 22 Feb 2020

👍1