Jormungandr: Node not receiving updates until...

Created on 29 Sep 2019  ·  38Comments  ·  Source: input-output-hk/jormungandr

Describe the bug
Creating a new node, setting the listen/public tcp port to 3100 will initially download the chain history but then fail to receive updates. If you change the listen/public tcp port to any other valid tcp port (3111) and restart the node starts to receive updates.

Mandatory Information
jcli 0.5.2 (master-23a9fe1c, release, linux [x86_64]) - [rustc 1.36.0 (a53f9df32 2019-07-03)]\
jormungandr 0.5.2 (master-23a9fe1c, release, linux [x86_64]) - [rustc 1.36.0 (a53f9df32 2019-07-03)]

To Reproduce
Steps to reproduce the behavior:

  • run node with config like this:
log:
  format: "plain"
  level: "info"
  output: "stderr"
p2p:
  listen_address: "/ip4/0.0.0.0/tcp/3100"
  public_address: "/ip4/x.x.x.x/tcp/3100"
  topics_of_interest:
    blocks: "high"
    messages: "high"
  trusted_peers:
    - "/ip4/136.244.106.95/tcp/3000"
    - "/ip4/91.121.85.221/tcp/3100"
    - "/ip4/68.183.155.37/tcp/3100"
    - "/ip4/3.123.177.192/tcp/3000"
    - "/ip4/3.123.155.47/tcp/3000"
    - "/ip4/3.115.57.216/tcp/3000"
    - "/ip4/3.112.185.217/tcp/3000"
    - "/ip4/18.139.40.4/tcp/3000"
    - "/ip4/18.140.134.230/tcp/3000"
rest:
  listen: "127.0.0.1:3101"
storage: "/path/to/storage"
  • stop node, change config to this:
log:
  format: "plain"
  level: "info"
  output: "stderr"
p2p:
  listen_address: "/ip4/0.0.0.0/tcp/3333"
  public_address: "/ip4/x.x.x.x/tcp/3333"
  topics_of_interest:
    blocks: "high"
    messages: "high"
  trusted_peers:
    - "/ip4/136.244.106.95/tcp/3000"
    - "/ip4/91.121.85.221/tcp/3100"
    - "/ip4/68.183.155.37/tcp/3100"
    - "/ip4/3.123.177.192/tcp/3000"
    - "/ip4/3.123.155.47/tcp/3000"
    - "/ip4/3.115.57.216/tcp/3000"
    - "/ip4/3.112.185.217/tcp/3000"
    - "/ip4/18.139.40.4/tcp/3000"
    - "/ip4/18.140.134.230/tcp/3000"
rest:
  listen: "127.0.0.1:3101"
storage: "/path/to/storage"
  • restart node with new config

Expected behavior
node syncs new blocks as expected but it doesn

Additional context
weird problem eh?

bug subsys-network Priority - High
Source
picture onicrom

Most helpful comment

Thanks a lot everyone for the testing and bearing with us while we fixed the low handing fruits.

What's happening here

There is multiple reasons that can explain why a node is no longer syncing. The more obvious one at the moment is that the node had an issue and entered in an invalid state, preventing it from updating. We believe most of the nodes will have this problem fixed as we fix bugs.
The second obvious cause will be in the p2p topology. It is possible a school of nodes (yes, using fish metaphor here) is heading toward a fork, not updating the state and ignoring every new updates. This can happen for different reasons: the nodes producing blocks are not reachable because of too many connections, the p2p topology parameters are not well tuned and need to be updated.

What now?

This is one of most difficult issue we will have to address and the plan to address it is going to be as follow:

  1. provide more way for the users to check their connectivity with other nodes (i.e. add an end points to query the network status) see #871
  2. from there we will be able to collect more details regarding how well connected each nodes that cannot sync are.
  3. see if this is a parameter of the p2p topology that needs to be changed or if the problem is elsewhere

There is also a possibility that most of us will have this issue addressed as we fix bugs.

This might take some time, so please, bear with us a little longer.

picture NicolasDP on 1 Oct 2019
👍26 ❤6 🎉2

All 38 comments

This may be the wrong solution to the problem... I just spun up a new node and tried changing the port to receive updates. It did not work.
What did work was adding my new node's ip/port to my existing node's trusted_peers list. Once I did that the new node started receiving blocks.

onicrom picture onicrom on 29 Sep 2019

I just played around with the config and there is definietly some issue,
the config below for example is running on a node for 14 hours now.

You can connect to it by adding it to your trusted peers, all important portsare opend
( closed net-test env )

storage: "/home/username/rel_052/storage"

p2p:
  public_address: "/ip4/79.143.188.85/tcp/3000"
  trusted_peers: [/ip4/3.123.177.192/tcp/3000,
                  /ip4/3.123.155.47/tcp/3000,
                  /ip4/52.57.157.167/tcp/3000,
                  /ip4/3.112.185.217/tcp/3000,
                  /ip4/18.140.134.230/tcp/3000,
                  /ip4/18.139.40.4/tcp/3000,
                  /ip4/3.115.57.216/tcp/3000]
  topics_of_interest:
    messages: high
    blocks: high

rest:
  listen: "127.0.0.1:8443"

Edit but I have an issue on this node with creating accounts

metastrix picture metastrix on 29 Sep 2019

metastrix, i didnt expect it to work, but following your reformatting for the p2p section fixed the problem for me.
i had to also add listen_address before public_address.

log:
  format: "plain"
  level: "debug"
  output: "stderr"
p2p:
  listen_address: "/ip4/0.0.0.0/tcp/14509"
  public_address: "/ip4/MYPUBLICIP/tcp/14509"
  trusted_peers: [/ip4/3.123.177.192/tcp/3000,
                  /ip4/3.123.155.47/tcp/3000,
                  /ip4/52.57.157.167/tcp/3000,
                  /ip4/3.112.185.217/tcp/3000,
                  /ip4/18.140.134.230/tcp/3000,
                  /ip4/18.139.40.4/tcp/3000,
                  /ip4/3.115.57.216/tcp/3000]
  topics_of_interest:
    messages: high
    blocks: high
rest:
  listen: "127.0.0.1:3101"
storage: "/mnt/f/jormungandr/testnet"
lazrfocus picture lazrfocus on 29 Sep 2019
👍1

Same issue, couldn't fix it. Tried different ports and formats.

noneymous picture noneymous on 29 Sep 2019

Do not bother changing ports or config. Just ask the community in telegram for some stable/working peers.

rinor picture rinor on 29 Sep 2019
👍2

Also experiencing this issue. Asking and using stable/working peers didn't seem to help the matters.

hodlonaut picture hodlonaut on 30 Sep 2019

I am also experiencing this from time to time. There are incoming connections but the node does not receive blocks.

C:\Users\Dorin>jcli rest v0 node stats get --host "http://127.0.0.1:9000/api"
---
blockRecvCnt: 0
lastBlockDate: "219.42663"
lastBlockFees: 0
lastBlockHash: 441f82e72d9dc6aa71a9feae9d39292f74b671385d2f3c823b6911a4e2c58d44
lastBlockHeight: "28230"
lastBlockSum: 0
lastBlockTime: ~
lastBlockTx: 0
txRecvCnt: 0
uptime: 246
dorin100 picture dorin100 on 30 Sep 2019

Having the same issue

codefly picture codefly on 30 Sep 2019

Same issue after updating to version 0.5.4. I updated by simply replacing the jormungandr and jcli files of the old version with the new one and restarting the node with /jormungandr --config ./node-config.yaml --genesis-block-hash adbdd5ede31637f6c9bad5c271eec0bc3d0cb9efb86a5b913bb55cba549d0770 --log-level=info

Output:

Oct 01 13:23:52.988 INFO Starting jormungandr 0.5.4 (HEAD-a9ec5ce, release, linux [x86_64]) - [rustc 1.38.0 (625451e37 2019-09-23)], task: init
Oct 01 13:23:52.988 WARN Node started without path to the stored secret keys (not a stake pool or a BFT leader), task: init
Oct 01 13:23:52.988 INFO storing blockchain in '"/tmp/jormungandr/blocks.sqlite"', task: init
Oct 01 13:24:04.819 INFO connecting to bootstrap peer 18.140.134.230:3000, peer_addr: 18.140.134.230:3000, task: bootstrap
Oct 01 13:24:05.362 WARN with initial bootstrap, reason: bootstrap pull request failed, peer_addr: 18.140.134.230:3000, task: bootstrap
Oct 01 13:24:05.363 INFO connecting to bootstrap peer 3.123.155.47:3000, peer_addr: 3.123.155.47:3000, task: bootstrap
Oct 01 13:24:06.018 WARN with initial bootstrap, reason: bootstrap pull request failed, peer_addr: 3.123.155.47:3000, task: bootstrap
Oct 01 13:24:06.018 INFO connecting to bootstrap peer 3.112.185.217:3000, peer_addr: 3.112.185.217:3000, task: bootstrap
Oct 01 13:24:06.283 WARN with initial bootstrap, reason: bootstrap pull request failed, peer_addr: 3.112.185.217:3000, task: bootstrap
Oct 01 13:24:06.283 INFO connecting to bootstrap peer 3.123.177.192:3000, peer_addr: 3.123.177.192:3000, task: bootstrap
Oct 01 13:24:06.928 WARN with initial bootstrap, reason: bootstrap pull request failed, peer_addr: 3.123.177.192:3000, task: bootstrap
Oct 01 13:24:06.928 INFO connecting to bootstrap peer 18.139.40.4:3000, peer_addr: 18.139.40.4:3000, task: bootstrap
Oct 01 13:24:07.471 WARN with initial bootstrap, reason: bootstrap pull request failed, peer_addr: 18.139.40.4:3000, task: bootstrap
Oct 01 13:24:07.471 INFO connecting to bootstrap peer 52.57.157.167:3000, peer_addr: 52.57.157.167:3000, task: bootstrap
Oct 01 13:24:35.731 INFO initial bootstrap completed, peer_addr: 52.57.157.167:3000, task: bootstrap
Oct 01 13:24:35.732 INFO starting task, task: client-query
Oct 01 13:24:35.732 INFO starting task, task: network
Oct 01 13:24:35.732 INFO adding P2P Topology module: trusted-peers, task: network
Oct 01 13:24:35.732 INFO start listening and accepting gRPC connections on 0.0.0.0:3100, task: network
Oct 01 13:24:35.732 INFO Starting 1 workers, scope: global
Oct 01 13:24:35.732 INFO Starting server on 127.0.0.1:3101, scope: global
Oct 01 13:24:35.732 INFO connecting to initial gossip peer, peer_addr: 3.112.185.217:3000, task: network
Oct 01 13:24:35.732 INFO connecting to initial gossip peer, peer_addr: 3.115.57.216:3000, task: network
Oct 01 13:24:35.733 INFO connecting to initial gossip peer, peer_addr: 52.57.157.167:3000, task: network
Oct 01 13:24:35.733 INFO connecting to initial gossip peer, peer_addr: 18.139.40.4:3000, task: network
Oct 01 13:24:35.733 INFO connecting to initial gossip peer, peer_addr: 3.123.177.192:3000, task: network
Oct 01 13:24:35.733 INFO connecting to initial gossip peer, peer_addr: 3.123.155.47:3000, task: network
Oct 01 13:24:35.733 INFO connecting to initial gossip peer, peer_addr: 18.140.134.230:3000, task: network

I then removed the /tmp/jormungandr/blocks.sqlite file to see if it made any difference, but it still doesn't sync (i.e. stats still output blockRecvCnt: 0)

jbax picture jbax on 1 Oct 2019

Thanks a lot everyone for the testing and bearing with us while we fixed the low handing fruits.

What's happening here

There is multiple reasons that can explain why a node is no longer syncing. The more obvious one at the moment is that the node had an issue and entered in an invalid state, preventing it from updating. We believe most of the nodes will have this problem fixed as we fix bugs.
The second obvious cause will be in the p2p topology. It is possible a school of nodes (yes, using fish metaphor here) is heading toward a fork, not updating the state and ignoring every new updates. This can happen for different reasons: the nodes producing blocks are not reachable because of too many connections, the p2p topology parameters are not well tuned and need to be updated.

What now?

This is one of most difficult issue we will have to address and the plan to address it is going to be as follow:

  1. provide more way for the users to check their connectivity with other nodes (i.e. add an end points to query the network status) see #871
  2. from there we will be able to collect more details regarding how well connected each nodes that cannot sync are.
  3. see if this is a parameter of the p2p topology that needs to be changed or if the problem is elsewhere

There is also a possibility that most of us will have this issue addressed as we fix bugs.

This might take some time, so please, bear with us a little longer.

NicolasDP picture NicolasDP on 1 Oct 2019
👍26 ❤6 🎉2

@NicolasDP would it help if I sent you guys a zip with all files I got (especially the content of /tmp/jormungandr) so you can hopefully debug what is going on based off this inconsistent state you mentioned?

jbax picture jbax on 1 Oct 2019

I got it working now but cannot say what exactly caused the problem. I changed the values for the keys listen_address and public_address to my public ip, removed blank lines and changed the keys to the following order: 

log:
  format: "plain"
  level: "info"
  output: "stderr"
p2p:
  listen_address: "/ip4/95.217.XXX.YYY/tcp/3100"
  public_address: "/ip4/95.217.XXX.YYY/tcp/3100"
  trusted_peers:
   - "/ip4/3.123.177.192/tcp/3000"
   - "/ip4/3.123.155.47/tcp/3000"
   - "/ip4/52.57.157.167/tcp/3000"
   - "/ip4/3.112.185.217/tcp/3000"
   - "/ip4/18.140.134.230/tcp/3000"
   - "/ip4/18.139.40.4/tcp/3000"
   - "/ip4/3.115.57.216/tcp/3000"
  topics_of_interest:
    messages: high
    blocks: high
rest:
  listen: "127.0.0.1:8443"
storage: "/tmp/jormungandr-storage-test/"

here's the expected output

$ jcli rest v0 node stats get -h http://127.0.0.1:8443/api
blockRecvCnt: 1663
lastBlockDate: "221.5683"
lastBlockFees: 0
lastBlockHash: 9f3eef1c97e0a24b2d41f389203199ac76a7ee0fd6d084035b135a246f4c4a46
lastBlockHeight: "36066"
lastBlockSum: 0
lastBlockTime: "2019-10-01T11:03:05+00:00"
lastBlockTx: 0
txRecvCnt: 4
uptime: 7816
mescedia picture mescedia on 1 Oct 2019

Request:
Is it possible to _force_ the bootstrap peer to be the first update peer until a new peer successfully connects and starts feeds updates successfully?

it seems, without fail, there is always a bootstrap peer who can feed the latest blockchain on launch of jormungandr.
but, then the bootstrap peer disconnects, and we are left with no other peers to feed us updates.

lazrfocus picture lazrfocus on 2 Oct 2019

@NicolasDP - one more detail, hopefully helpful (v0.5.5):

I observed that my node stops to receive blocks after generating the below logs:

Oct 02 09:04:49.510 INFO gossip propagation to peer 2841633050629252853 failed: SubscriptionClosed, task: network
Oct 02 09:16:46.386 INFO gossip subscription stream failure: Error { code: Unknown, source: Status { code: Unknown, message: "h2 protocol error: broken pipe" } }, sub_task: server, task: network

$ jcli rest v0 node stats get --host "http://127.0.0.1:9000/api"
---
blockRecvCnt: 2241
lastBlockDate: "221.39387"
lastBlockFees: 0
lastBlockHash: 08e9c2b9f527ebe6546a979db412b42cf865b643e0684a79a5e345f267f987e6
lastBlockHeight: "39915"
lastBlockSum: 0
lastBlockTime: "2019-10-02T06:00:43+00:00"
lastBlockTx: 0
txRecvCnt: 16
uptime: 7958

And from around that time, it did not receive blocks anymore. The time from the logs is my local one that is +3h to the blockchain time (displayed in node stats - lastBlockTime)

dorin100 picture dorin100 on 2 Oct 2019
👍2

Seems like it's stuck in a ConnectionRefused loop:

error connecting to peer: ConnectError(Http(Connect(Os { code: 61, kind: ConnectionRefused, message: "Connection refused" }))), peer_addr: 163.172.195.51:4444, task: network

Adding full debug output on this issue below.
debug_out.txt

yodsv2 picture yodsv2 on 2 Oct 2019

Hi, I am having this problem and I am trying to run my node on windows using the PowerShell.
Is this relevant to windows users?
Is there something I can do to troubleshoot?

hugodeanda picture hugodeanda on 4 Oct 2019

Having issue on Windows 10 Pro. I'm using a windows 10 box running wireshark and I immediately see connection resets from all endpoints except for 3.115.57.216. Currently my p2p settings are

listen_address: /ip4/0.0.0.0/tcp/3100
public_address: /ip4/24.aaa.bbb.ccc/tcp/3100

Just masked out my public ip. It seems to be that the only p2p node responding is that 57.216.

1337alan picture 1337alan on 7 Oct 2019

Having an issue syncing with peer nodes

Running with this config file, I have opened port 3100 and 3101 on my router, and disabled the ufw firewall on my kubuntu machine.

node-config.yaml

log: format: "plain" level: "info" output: "stderr" p2p: listen_address: "/ip4/[myLocal_IPaddressHere]/tcp/3100" public_address: "/ip4/[myPublic_IPaddressHere]/tcp/3100" topics_of_interest: blocks: "high" messages: "high" trusted_peers: - "/ip4/3.123.177.192/tcp/3000" - "/ip4/3.123.155.47/tcp/3000" - "/ip4/52.57.157.167/tcp/3000" - "/ip4/3.112.185.217/tcp/3000" - "/ip4/18.140.134.230/tcp/3000" - "/ip4/18.139.40.4/tcp/3000" - "/ip4/3.115.57.216/tcp/3000" rest: listen: "127.0.0.1:3101" storage:"/tmp/jormungandr"`

This is what I am seeing

Oct 06 12:25:11.459 INFO starting task, task: client-query Oct 06 12:25:11.459 INFO starting task, task: network Oct 06 12:25:11.459 INFO adding P2P Topology module: trusted-peers, task: network Oct 06 12:25:11.459 INFO start listening and accepting gRPC connections on [myLocal_IPaddressHere]:3100, task: network Oct 06 12:25:11.460 INFO connecting to initial gossip peer, peer_addr: 3.112.185.217:3000, task: network Oct 06 12:25:11.460 INFO Starting 1 workers, scope: global Oct 06 12:25:11.460 INFO connecting to initial gossip peer, peer_addr: 3.115.57.216:3000, task: network Oct 06 12:25:11.460 INFO Starting server on 127.0.0.1:3101, scope: global Oct 06 12:25:11.461 INFO connecting to initial gossip peer, peer_addr: 52.57.157.167:3000, task: network Oct 06 12:25:11.461 INFO connecting to initial gossip peer, peer_addr: 18.139.40.4:3000, task: network Oct 06 12:25:11.461 INFO connecting to initial gossip peer, peer_addr: 3.123.177.192:3000, task: network Oct 06 12:25:11.461 INFO connecting to initial gossip peer, peer_addr: 3.123.155.47:3000, task: network Oct 06 12:25:11.461 INFO connecting to initial gossip peer, peer_addr: 18.140.134.230:3000, task: network Oct 06 12:25:20.985 INFO incoming P2P connection on [myLocal_IPaddressHere]:3100, peer_addr: 104.254.90.195:36272, task: network Oct 06 12:25:31.753 INFO peer responded with different node id: 8983095618647613224, node_id: 8983095618647613224, peer_addr: 18.219.55.201:3000, task: network Oct 06 12:25:41.779 INFO peer responded with different node id: 8983095618647613224, node_id: 8983095618647613224, peer_addr: 18.219.55.201:3000, task: network

any help would be much appreciated.

brandonlimpert picture brandonlimpert on 7 Oct 2019

Was having same issue, might be a coincidence, but this is what worked for me. Platform is Ubuntu running on Windows Services for Linux

Port forwarding on router for TCP 3100 -> 192.168.0.xxx (my local IP)

log:
format: "plain"
level: "info"
output: "stderr"
p2p:
listen_address: "/ip4/192.168.0.xxx/tcp/3100"
public_address: "/ip4/MyPublicAddress/tcp/3100"
topics_of_interest:
blocks: "high"
messages: "high"
trusted_peers:
- "/ip4/3.123.177.192/tcp/3000"
- "/ip4/3.123.155.47/tcp/3000"
- "/ip4/52.57.157.167/tcp/3000"
- "/ip4/3.112.185.217/tcp/3000"
- "/ip4/18.140.134.230/tcp/3000"
- "/ip4/18.139.40.4/tcp/3000"
- "/ip4/3.115.57.216/tcp/3000"
rest:
listen: "127.0.0.1:3101"
storage: "/tmp/jormungandr"

isonis picture isonis on 7 Oct 2019

Oct 7 21:49:54 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 176.31.122.83:3100, task: network, reason: Connection refused (os error 111)
Oct 7 21:49:54 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2419909587550422536, peer_addr: 176.31.122.83:3100, task: network
Oct 7 21:49:54 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 94.11.44.127:3100, task: network, reason: Connection refused (os error 111)
Oct 7 21:49:54 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 15964830016144729199, peer_addr: 94.11.44.127:3100, task: network
Oct 7 21:49:54 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 188.193.116.232:3000, task: network, reason: Connection refused (os error 111)
Oct 7 21:49:54 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5229285601346881136, peer_addr: 188.193.116.232:3000, task: network
Oct 7 21:49:55 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 68.183.189.215:3100, task: network, reason: Connection refused (os error 111)
Oct 7 21:49:55 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 11484448678615156572, peer_addr: 68.183.189.215:3100, task: network
Oct 7 21:50:02 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 34.84.102.29:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:02 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 16698054959422377585, peer_addr: 34.84.102.29:3100, task: network
Oct 7 21:50:02 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 14.201.165.168:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:02 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5583382374378854273, peer_addr: 14.201.165.168:3100, task: network
Oct 7 21:50:02 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 70.74.237.155:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:02 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2326356386753111771, peer_addr: 70.74.237.155:3100, task: network
Oct 7 21:50:04 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: [2001:19f0:ac01:20::200]:3100, task: network, reason: Network is unreachable (os error 101)
Oct 7 21:50:04 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 10127517464646818779, peer_addr: [2001:19f0:ac01:20::200]:3100, task: network
Oct 7 21:50:04 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 176.31.122.83:3100, task: network, reason: Connection refused (os error 111)
Oct 7 21:50:04 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2419909587550422536, peer_addr: 176.31.122.83:3100, task: network
Oct 7 21:50:04 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 188.193.116.232:3000, task: network, reason: Connection refused (os error 111)
Oct 7 21:50:04 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5229285601346881136, peer_addr: 188.193.116.232:3000, task: network
Oct 7 21:50:04 instance-1 jormungandr[27268]: peer responded with different node id: 18435507258811407394, node_id: 18435507258811407394, peer_addr: 40.86.191.78:3131, task: network
Oct 7 21:50:12 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 34.84.102.29:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:12 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 16698054959422377585, peer_addr: 34.84.102.29:3100, task: network
Oct 7 21:50:12 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 14.201.165.168:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:12 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5583382374378854273, peer_addr: 14.201.165.168:3100, task: network
Oct 7 21:50:12 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 70.74.237.155:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:12 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2326356386753111771, peer_addr: 70.74.237.155:3100, task: network
Oct 7 21:50:14 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: [2001:19f0:ac01:20::200]:3100, task: network, reason: Network is unreachable (os error 101)
Oct 7 21:50:14 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 10127517464646818779, peer_addr: [2001:19f0:ac01:20::200]:3100, task: network
Oct 7 21:50:14 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 176.31.122.83:3100, task: network, reason: Connection refused (os error 111)
Oct 7 21:50:14 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2419909587550422536, peer_addr: 176.31.122.83:3100, task: network
Oct 7 21:50:14 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 188.193.116.232:3000, task: network, reason: Connection refused (os error 111)
Oct 7 21:50:14 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5229285601346881136, peer_addr: 188.193.116.232:3000, task: network
Oct 7 21:50:14 instance-1 jormungandr[27268]: peer responded with different node id: 18435507258811407394, node_id: 18435507258811407394, peer_addr: 40.86.191.78:3131, task: network
Oct 7 21:50:14 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 7223119958431270766, peer_addr: 40.86.191.78:3131, task: network
Oct 7 21:50:22 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 34.84.102.29:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:22 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 16698054959422377585, peer_addr: 34.84.102.29:3100, task: network
Oct 7 21:50:22 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 14.201.165.168:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:22 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5583382374378854273, peer_addr: 14.201.165.168:3100, task: network
Oct 7 21:50:22 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 70.74.237.155:3100, task: network, reason: Connection timed out (os error 110)
Oct 7 21:50:22 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2326356386753111771, peer_addr: 70.74.237.155:3100, task: network
Oct 7 21:50:24 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: [2001:19f0:ac01:20::200]:3100, task: network, reason: Network is unreachable (os error 101)
Oct 7 21:50:24 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 10127517464646818779, peer_addr: [2001:19f0:ac01:20::200]:3100, task: network
Oct 7 21:50:24 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 176.31.122.83:3100, task: network, reason: Connection refused (os error 111)
Oct 7 21:50:24 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2419909587550422536, peer_addr: 176.31.122.83:3100, task: network
Oct 7 21:50:24 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 188.193.116.232:3000, task: network, reason: Connection refused (os error 111)
Oct 7 21:50:24 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 5229285601346881136, peer_addr: 188.193.116.232:3000, task: network
Oct 7 21:50:24 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 7223119958431270766, peer_addr: 40.86.191.78:3131, task: network
Oct 7 21:50:24 instance-1 jormungandr[27268]: peer responded with different node id: 18435507258811407394, node_id: 18435507258811407394, peer_addr: 40.86.191.78:3131, task: network
Oct 7 21:50:27 instance-1 jormungandr[27268]: error connecting to peer, peer_addr: 5.12.114.32:3001, task: network, reason: No route to host (os error 113)
Oct 7 21:50:27 instance-1 jormungandr[27268]: client P2P connection closed, node_id: 2230315542497794011, peer_addr: 5.12.114.32:3001, task: network

rgoerwit picture rgoerwit on 7 Oct 2019
👍1

Same here : os Error 110 and os Error 111

Proxiweb picture Proxiweb on 8 Oct 2019
👍2

The connection errors logged at the "info" level don't necessarily indicate a problem, these are just unsuccessful attempts to contact nodes advertised on the network. As long as the node has some connections and is able to receive messages via the subscription streams, propagation should occur.

mzabaluev picture mzabaluev on 10 Oct 2019

We're working to add live collection information available via REST and jcli: #846
This should make investigation easier.

mzabaluev picture mzabaluev on 10 Oct 2019
👍2

My node not sync

Oct 11 20:48:22.221 INFO client P2P connection closed, node_id: 3480640125560260922, peer_addr: 80.57.91.60:3333, task: network
Oct 11 20:48:22.519 INFO error connecting to peer, reason: Operation timed out (os error 60), peer_addr: 140.82.56.80:31000, task: network

timsahabutdinov picture timsahabutdinov on 11 Oct 2019

I noticed some people fixed the issue by forwarding a port and exposing a node externally. Is this a short term hack or are we expected to need to port forward, I was thinking that in theory blockchain system and node should work without having to port forward to a computer in your local network (for those behind firewall -> like I guess most of us)

matiwinnetou picture matiwinnetou on 11 Oct 2019

i'm directly attached to the internet and still have the problem. I have noticed, though, that the node ocasionally syncs... but then it doesn't do for a while (hours)

noneymous picture noneymous on 11 Oct 2019

One identified cause is that nodes that don't have a public_address do not receive items through p2p propagation mechanism. The "private" node feature was added recently and has not yet received sufficient amount of love. An interim solution is to set up a port forwarding with a known public IP address and make sure the address and port can be connected to from the public internet.

mzabaluev picture mzabaluev on 11 Oct 2019

Providing additional info if helpful to bug fixing...

I have static IP public_address configured in node-config.yaml and firewall's port 3100 forwarded to LAN host running jormungandr. I'm getting significant outbound connection failures (TCP SYN_SENT) but can successful ping most of these hosts...

netstat -n -t -a -p | grep jormungandr output:

tcp        0      0 10.212.3.212:3100       0.0.0.0:*               LISTEN      53494/jormungandr
tcp        0      0 127.0.0.1:3101          0.0.0.0:*               LISTEN      53494/jormungandr
tcp        0      1 10.212.3.212:39066      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:60606      91.158.25.32:3000       SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:39048      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:60630      91.158.25.32:3000       SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:53308      187.131.7.168:3200      SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:33828      3.112.185.217:3000      ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:39072      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:47928      3.123.177.192:3000      ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:51172      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:37044      76.184.240.151:3000     ESTABLISHED 53494/jormungandr
tcp        0      0 10.212.3.212:57892      18.139.40.4:3000        ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:39084      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:3100       193.105.188.10:50825    ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:51190      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:53332      187.131.7.168:3200      SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:53302      187.131.7.168:3200      SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:57532      3.122.125.100:3000      ESTABLISHED 53494/jormungandr
tcp        0      0 10.212.3.212:47854      3.123.155.47:3000       ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:37634      92.37.24.198:3100       SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51202      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51178      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51148      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51196      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:60624      91.158.25.32:3000       SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:59058      103.89.83.152:7777      SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:60618      91.158.25.32:3000       SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51124      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:59042      103.89.83.152:7777      SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51140      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:33132      18.140.134.230:3000     ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:51132      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:40634      3.115.57.216:3000       ESTABLISHED 53494/jormungandr
tcp        0      0 10.212.3.212:34730      52.57.157.167:3000      ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:53338      187.131.7.168:3200      SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:60614      91.158.25.32:3000       SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:39060      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:39078      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:47912      187.131.250.14:6000     SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:3100       85.218.89.33:35358      ESTABLISHED 53494/jormungandr
tcp        0      0 10.212.3.212:41136      206.189.10.179:3000     ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:53314      187.131.7.168:3200      SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:58130      142.163.140.47:3002     ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:39054      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:53320      187.131.7.168:3200      SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:59050      103.89.83.152:7777      SYN_SENT    53494/jormungandr
tcp        0      0 10.212.3.212:50716      3.123.252.31:3000       ESTABLISHED 53494/jormungandr
tcp        0      0 10.212.3.212:50352      172.104.129.129:3100    ESTABLISHED 53494/jormungandr
tcp        0      1 10.212.3.212:47928      187.131.250.14:6000     SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:39090      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51166      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:39096      91.66.10.14:3100        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51160      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:47920      187.131.250.14:6000     SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51154      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:51184      69.245.21.9:3333        SYN_SENT    53494/jormungandr
tcp        0      1 10.212.3.212:53326      187.131.7.168:3200      SYN_SENT    53494/jormungandr

jormungandr output:

Oct 11 14:59:39.924 INFO error connecting to peer, reason: Connection timed out (os error 110), peer_addr: 149.28.75.195:3000, task: network
Oct 11 14:59:39.925 INFO error connecting to peer, reason: Connection timed out (os error 110), peer_addr: 104.214.105.17:3000, task: network
Oct 11 14:59:39.930 INFO client P2P connection closed, node_id: 1054724132166253757, peer_addr: 104.214.105.17:3000, task: network
Oct 11 14:59:39.930 INFO error connecting to peer, reason: Connection timed out (os error 110), peer_addr: 52.57.16.140:3000, task: network
Oct 11 14:59:39.931 INFO client P2P connection closed, node_id: 15663431134006357362, peer_addr: 149.28.75.195:3000, task: network
Oct 11 14:59:39.932 INFO client P2P connection closed, node_id: 10341631933794349269, peer_addr: 52.57.16.140:3000, task: network
Oct 11 14:59:45.635 INFO error connecting to peer, reason: No route to host (os error 113), peer_addr: 80.57.91.60:60001, task: network
Oct 11 14:59:45.636 INFO client P2P connection closed, node_id: 2875490547356695070, peer_addr: 80.57.91.60:60001, task: network

"./jcli rest v0 node stats get --host "http://127.0.0.1:3101/api" output:

blockRecvCnt: 2791
lastBlockDate: "231.19055"
lastBlockFees: 0
lastBlockHash: 5fdf9b2f4d675b51fa6843ba66b5bdfb29b105183b87b6d9ac1e97638d7acbc6
lastBlockHeight: "98554"
lastBlockSum: 0
lastBlockTime: "2019-10-11T18:29:03+00:00"
lastBlockTx: 0
txRecvCnt: 3
uptime: 18415
CraniumDad picture CraniumDad on 11 Oct 2019
👀1

ver 5.6 running on ubuntu-18.04.3-live-server-amd64

Similar problem. Node starts ok and begins to download the blockchain. I can cd into my storage directory and see the blocks.sqlite file increasing in size. After a 10 to 15 minute period the file stops to increase and the node returns info that it cant connect to peer. It then goes into a loop of trying to connect to different peers and they all fail.

I am not totally clear on if it is a problem with my network topology and access. Can you give concise details about what IP addresses and ports should be used in the node-config.yaml file and also what needs to be done on the network firewall level e.g. local and public address and how they correlate to each other and how to configure this. Any help would be great. Thanks. Carl

node-config.yaml is below

log:
format: "plain"
level: "info"
output: "stderr"
p2p:
listen_address: "/ip4/0.0.0.0/tcp/3100"
public_address:
topics_of_interest:
blocks: "high"
messages: "high"
trusted_peers:
- "/ip4/3.123.177.192/tcp/3000"
- "/ip4/3.123.155.47/tcp/3000"
- "/ip4/52.57.157.167/tcp/3000"
- "/ip4/3.112.185.217/tcp/3000"
- "/ip4/18.140.134.230/tcp/3000"
- "/ip4/18.139.40.4/tcp/3000"
- "/ip4/3.115.57.216/tcp/3000"
rest:
listen: "127.0.0.1:3101"
storage: "/home/gareth1/jormungandr/storage"

carlwhiston picture carlwhiston on 13 Oct 2019
👍1

ver 5.6 running on ubuntu-18.04.3-live-server-amd64

Similar problem. Node starts ok and begins to download the blockchain. I can cd into my storage directory and see the blocks.sqlite file increasing in size. After a 10 to 15 minute period the file stops to increase and the node returns info that it cant connect to peer. It then goes into a loop of trying to connect to different peers and they all fail.

I am not totally clear on if it is a problem with my network topology and access. Can you give concise details about what IP addresses and ports should be used in the node-config.yaml file and also what needs to be done on the network firewall level e.g. local and public address and how they correlate to each other and how to configure this. Any help would be great. Thanks. Carl

node-config.yaml is below

log:
format: "plain"
level: "info"
output: "stderr"
p2p:
listen_address: "/ip4/0.0.0.0/tcp/3100"
public_address:
topics_of_interest:
blocks: "high"
messages: "high"
trusted_peers:

  • "/ip4/3.123.177.192/tcp/3000"
  • "/ip4/3.123.155.47/tcp/3000"
  • "/ip4/52.57.157.167/tcp/3000"
  • "/ip4/3.112.185.217/tcp/3000"
  • "/ip4/18.140.134.230/tcp/3000"
  • "/ip4/18.139.40.4/tcp/3000"
  • "/ip4/3.115.57.216/tcp/3000"
    rest:
    listen: "127.0.0.1:3101"
    storage: "/home/gareth1/jormungandr/storage"

same problem.

smurf123444 picture smurf123444 on 14 Oct 2019

jormungandr 0.6.1 (HEAD-a0de8d4, release, linux [x86_64])

Chain downloads but node not synchronising.

blockRecvCnt: 0
lastBlockDate: "238.1466"
lastBlockFees: 0
lastBlockHash: b3424ad6249bf0068f34187f129b5eb684f3ebd5a372051fa2c7de90a71784d2
lastBlockHeight: "145190"
lastBlockSum: 0
lastBlockTime: ~
lastBlockTx: 0
txRecvCnt: 0
uptime: 1736

Tried on three different devices, with many configuration variations.
Public address on given port is reachable. (changing ports and restarting node does not help)

Every 10 seconds there is same output:

Oct 18 08:45:44.048 DEBG scheduling Read for: 0, scope: global Oct 18 08:45:44.048 DEBG selecting gossips for 91672571fddfd0a3d95e0b955b8f2f62c818a32f35c1ab97667033c1d4fcaea3, task: network Oct 18 08:45:44.048 DEBG sending gossip to node 91672571fddfd0a3d95e0b955b8f2f62c818a32f35c1ab97667033c1d4fcaea3, task: network Oct 18 08:45:44.048 DEBG selecting gossips for 9861dfd6450967f4c1e986d32223a5e02d44621f27186268393186f946b23c47, task: network Oct 18 08:45:44.048 DEBG sending gossip to node 9861dfd6450967f4c1e986d32223a5e02d44621f27186268393186f946b23c47, task: network Oct 18 08:45:44.049 DEBG selecting gossips for a042da1be8d2c9feee16058d7a8c5cdab4e2f701b9d0241f4eab5a5e9ce00cb7, task: network Oct 18 08:45:44.049 DEBG sending gossip to node a042da1be8d2c9feee16058d7a8c5cdab4e2f701b9d0241f4eab5a5e9ce00cb7, task: network Oct 18 08:45:44.049 DEBG selecting gossips for d05ca82e8e41ab4a1f1a5e7723ac4c784188f800904948a60b6904e374816649, task: network Oct 18 08:45:44.049 DEBG sending gossip to node d05ca82e8e41ab4a1f1a5e7723ac4c784188f800904948a60b6904e374816649, task: network Oct 18 08:45:44.049 DEBG scheduling Read for: 2, scope: global Oct 18 08:45:44.049 DEBG send; frame=Data { stream_id: StreamId(7) }, scope: global Oct 18 08:45:44.050 DEBG scheduling Read for: 3, scope: global Oct 18 08:45:44.050 DEBG send; frame=Data { stream_id: StreamId(7) }, scope: global Oct 18 08:45:44.050 DEBG scheduling Read for: 4, scope: global Oct 18 08:45:44.050 DEBG send; frame=Data { stream_id: StreamId(7) }, scope: global Oct 18 08:45:44.050 DEBG scheduling Read for: 5, scope: global Oct 18 08:45:44.050 DEBG send; frame=Data { stream_id: StreamId(7) }, scope: global

storage: "/home/ubuntu/iohk/storage"
rest:
listen: "127.0.0.1:3101"
log:
level: debug
format: plain
output: stderr
p2p:
trusted_peers:
- address: "/ip4/3.115.194.22/tcp/3000"
id: ed25519_pk1npsal4j9p9nlfs0fsmfjyga9uqk5gcslyuvxy6pexxr0j34j83rsf98wl2
- address: "/ip4/13.113.10.64/tcp/3000"
id: ed25519_pk16pw2st5wgx4558c6temj8tzv0pqc37qqjpy53fstdyzwxaypveys3qcpfl
- address: "/ip4/52.57.214.174/tcp/3000"
id: ed25519_pk1v4cj0edgmp8f2m5gex85jglrs2ruvu4z7xgy8fvhr0ma2lmyhtyszxtejz
- address: "/ip4/3.120.96.93/tcp/3000"
id: ed25519_pk10gmg0zkxpuzkghxc39n3a646pdru6xc24rch987cgw7zq5pmytmszjdmvh
- address: "/ip4/52.28.134.8/tcp/3000"
id: ed25519_pk1unu66eej6h6uxv4j4e9crfarnm6jknmtx9eknvq5vzsqpq6a9vxqr78xrw
- address: "/ip4/13.52.208.132/tcp/3000"
id: ed25519_pk15ppd5xlg6tylamskqkxh4rzum26w9acph8gzg86w4dd9a88qpjms26g5q9
- address: "/ip4/54.153.19.202/tcp/3000"
id: ed25519_pk1j9nj2u0amlg28k27pw24hre0vtyp3ge0xhq6h9mxwqeur48u463s0crpfk
topics_of_interest:
messages: high
blocks: high
listen_address: "/ip4/0.0.0.0/tcp/9001"
public_address: "/ip4/x.x.x.x/tcp/9001"
private_id: "ed25519_sk1xxxxxxxxxxx..."

Jamanis picture Jamanis on 18 Oct 2019

For nodes behind a NAT, it makes sense to limit the number of their p2p connections to what their NAT gateway can realistically support:

p2p:
  max_connections: 10

Try this if you see too many outbound connection errors or timeouts to nodes that are actually available.

mzabaluev picture mzabaluev on 18 Oct 2019

https://github.com/input-output-hk/jormungandr/issues/860#issuecomment-543621453

This is exactly what I'm seeing too. Node running on EC2 with ports opened and reachable and node-config as below

log:
  format: "plain"
  level: "debug"
  output: "stderr"
p2p:
  listen_address: "/ip4/0.0.0.0/tcp/3000"
  public_address: "/ip4/1.2.3.4/tcp/3000"
  topics_of_interest:
    blocks: "high"
    messages: "high"
  trusted_peers:
    - address: "/ip4/3.115.194.22/tcp/3000"
      id: ed25519_pk1npsal4j9p9nlfs0fsmfjyga9uqk5gcslyuvxy6pexxr0j34j83rsf98wl2
    - address: "/ip4/13.113.10.64/tcp/3000"
      id: ed25519_pk16pw2st5wgx4558c6temj8tzv0pqc37qqjpy53fstdyzwxaypveys3qcpfl
    - address: "/ip4/52.57.214.174/tcp/3000"
      id: ed25519_pk1v4cj0edgmp8f2m5gex85jglrs2ruvu4z7xgy8fvhr0ma2lmyhtyszxtejz
    - address: "/ip4/3.120.96.93/tcp/3000"
      id: ed25519_pk10gmg0zkxpuzkghxc39n3a646pdru6xc24rch987cgw7zq5pmytmszjdmvh
    - address: "/ip4/52.28.134.8/tcp/3000"
      id: ed25519_pk1unu66eej6h6uxv4j4e9crfarnm6jknmtx9eknvq5vzsqpq6a9vxqr78xrw
    - address: "/ip4/13.52.208.132/tcp/3000"
      id: ed25519_pk15ppd5xlg6tylamskqkxh4rzum26w9acph8gzg86w4dd9a88qpjms26g5q9
    - address: "/ip4/54.153.19.202/tcp/3000"
      id: ed25519_pk1j9nj2u0amlg28k27pw24hre0vtyp3ge0xhq6h9mxwqeur48u463s0crpfk

rest:
  listen: "127.0.0.1:3101"
storage: "/tmp/jormungandr"
jwbrew picture jwbrew on 21 Oct 2019

Running v0.6.5

Exactly the same issue here, I've tried setting up locally via WSL as well as on a VPS and the node doesn't sync. I've tried forwarding the port on my router and opened ports on the firewall. Here is my local config:

```log:
format: plain
level: info
output: stderr

storage: "/home/scott/self-node/jormungandr-storage-test/"

rest:
listen: "127.0.0.1:8443"

p2p:
trusted_peers:
- address: "/ip4/3.115.194.22/tcp/3000"
id: ed25519_pk1npsal4j9p9nlfs0fsmfjyga9uqk5gcslyuvxy6pexxr0j34j83rsf98wl2
- address: "/ip4/13.113.10.64/tcp/3000"
id: ed25519_pk16pw2st5wgx4558c6temj8tzv0pqc37qqjpy53fstdyzwxaypveys3qcpfl
- address: "/ip4/52.57.214.174/tcp/3000"
id: ed25519_pk1v4cj0edgmp8f2m5gex85jglrs2ruvu4z7xgy8fvhr0ma2lmyhtyszxtejz
- address: "/ip4/3.120.96.93/tcp/3000"
id: ed25519_pk10gmg0zkxpuzkghxc39n3a646pdru6xc24rch987cgw7zq5pmytmszjdmvh
- address: "/ip4/52.28.134.8/tcp/3000"
id: ed25519_pk1unu66eej6h6uxv4j4e9crfarnm6jknmtx9eknvq5vzsqpq6a9vxqr78xrw
- address: "/ip4/13.52.208.132/tcp/3000"
id: ed25519_pk15ppd5xlg6tylamskqkxh4rzum26w9acph8gzg86w4dd9a88qpjms26g5q9
- address: "/ip4/54.153.19.202/tcp/3000"
id: ed25519_pk1j9nj2u0amlg28k27pw24hre0vtyp3ge0xhq6h9mxwqeur48u463s0crpfk
public_address: "/ip4/[PUBLIC IP]/tcp/3000"
listen_address: "/ip4/192.168.1.83/tcp/3000"
topics_of_interest:
messages: high
blocks: high

scottdarby picture scottdarby on 21 Oct 2019

Upgrade to v0.6.5 fixed my issue.

Jamanis picture Jamanis on 21 Oct 2019

Running v0.6.5

Exactly the same issue here, I've tried setting up locally via WSL as well as on a VPS and the node doesn't sync. I've tried forwarding the port on my router and opened ports on the firewall. Here is my local config:

  format: plain
  level: info
  output: stderr

storage: "/home/scott/self-node/jormungandr-storage-test/"

rest:
  listen: "127.0.0.1:8443"

p2p:
  trusted_peers:
    - address: "/ip4/3.115.194.22/tcp/3000"
      id: ed25519_pk1npsal4j9p9nlfs0fsmfjyga9uqk5gcslyuvxy6pexxr0j34j83rsf98wl2
    - address: "/ip4/13.113.10.64/tcp/3000"
      id: ed25519_pk16pw2st5wgx4558c6temj8tzv0pqc37qqjpy53fstdyzwxaypveys3qcpfl
    - address: "/ip4/52.57.214.174/tcp/3000"
      id: ed25519_pk1v4cj0edgmp8f2m5gex85jglrs2ruvu4z7xgy8fvhr0ma2lmyhtyszxtejz
    - address: "/ip4/3.120.96.93/tcp/3000"
      id: ed25519_pk10gmg0zkxpuzkghxc39n3a646pdru6xc24rch987cgw7zq5pmytmszjdmvh
    - address: "/ip4/52.28.134.8/tcp/3000"
      id: ed25519_pk1unu66eej6h6uxv4j4e9crfarnm6jknmtx9eknvq5vzsqpq6a9vxqr78xrw
    - address: "/ip4/13.52.208.132/tcp/3000"
      id: ed25519_pk15ppd5xlg6tylamskqkxh4rzum26w9acph8gzg86w4dd9a88qpjms26g5q9
    - address: "/ip4/54.153.19.202/tcp/3000"
      id: ed25519_pk1j9nj2u0amlg28k27pw24hre0vtyp3ge0xhq6h9mxwqeur48u463s0crpfk
  public_address: "/ip4/[PUBLIC IP]/tcp/3000"
  listen_address: "/ip4/192.168.1.83/tcp/3000"
  topics_of_interest:
    messages: high
    blocks: high

My node just started updating correctly following a restart.

scottdarby picture scottdarby on 21 Oct 2019

After dozens of restarts, all on 0.6.5, the thing that seemed to fix the selecting gossips loop appears to be setting the max_connections in the node config to less than the total number of trusted peers 🤷‍♂️

jwbrew picture jwbrew on 21 Oct 2019

Thanks everyone for the testing and sending logs and useful info.

What happened

A lot have been made since the last time I commented on this issue (https://github.com/input-output-hk/jormungandr/issues/860#issuecomment-536980736):

  1. we added more way to collect connectivity info, better logging and a new end point #871
  2. thanks to that and everyone sending us issue reports we found some issues and fixed them, I think 0.6.5 is one of the most stable network we have made so far: #946 #940 #981 #985 #994 (and so many more);
  3. we have changed things in our poldercast implementation to evict nodes that are not reachable faster #866 and to make rooms for nodes that are not publicly reachable #926

What now

The network is way more stable than before. So I think we can close this issue. However it is possible some will continue to experience connectivity issues: don't worry we are still working on the network. We are going to continue to fix bugs and to improve stability and scalability.

NicolasDP picture NicolasDP on 21 Oct 2019
❤4
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lovelypool picture lovelypool  ·  14Comments
Linicks picture Linicks  ·  13Comments
hemants1 picture hemants1  ·  14Comments
gufmar picture gufmar  ·  17Comments
rinor picture rinor  ·  11Comments