Substrate: Can't connect to local node without manually copying chain db

[jak-pan]

I tried to build private substrate (polkadot-kusama) network locally for testing. I've got three development machines running OSX on local network. I've used this substrate tutorial
my node version is 0.7.19-7c34a67 on all machines

my command to run bootstrap node was

 ./target/release/polkadot   --base-path /tmp/alice   --chain local   --alice   --port 30333   --ws-port 9944   --rpc-port 9933 --validator

As soon as I tried to connect another node, I got this on bootstrap node, seeing data is being sent and received.

Discovered new external address for our node: /ip4/192.168.0.94/tcp/30333/p2p/QmPUTDXgUonhf46aAXKufWJjnJXzxdGKVaAWEzFeao6wnv
socket error: decode error: i/o error: Connection reset by peer (os error 54)

command to run client node is

./target/release/polkadot   --base-path /tmp/bob   --chain local   --bob   --port 30333   --ws-port 9944   --rpc-port 9933   --validator --bootnodes /ip4/192.168.0.94/tcp/30333/p2p/QmPUTDXgUonhf46aAXKufWJjnJXzxdGKVaAWEzFeao6wnv

I noticed that genesis block was different on each node, I tried deleting chain data and retrying from 2 machines, each time generating different genesis block.

So I copied this folder from bootstrap node /tmp/alice/chains/local_testnet/db to /tmp/bob/chains/local_testnet/db of other validator node and the chain started and works perfectly. I needed to do the same for third (bob) validator machine for it to start working too, so I guess there is a problem with bootstrapping mechanism.

[bkchr]

The problem is that the wasm file is not build deterministic and that leads to a different genesis block hash on each machine. You need to use build-spec and distribute to your nodes.

[jak-pan]

That makes sense, it should probably be mentioned in the tutorial. I noticed different genesis blocks only because of this. If it was not present, I would have been clueless.

[bkchr]

Cc @shawntabrizi @JoshOrndorff would be great to include this info into the tutorial.

[JoshOrndorff]

Hi @jak-pan sorry you had trouble with our tutorial. Indeed creating a chain-spec and distributing it is how real-world networks should be created. That process is covered later in the tutorial along with a warning about nondeterministic compiles.

During the part of the tutorial you linked, a single compiled binary should be used on a single machine (perhaps that should be clarified). Did you compile two different polkadot binaries for Alice and Bob?

Also, while this would not be related to the mismatched Genesis blocks, I notice you've specified the exact same ports for both nodes. If the nodes are on the same machine, they need their own ports specified. This _may_ explain the socket error.

[jak-pan]

Hello, I have three separate machines running on a local network, each of them compiled from scratch.

I wanted to test polkadot/substrate BABE equivocation, but I wanted to have more real-life experience than running on a single machine, since I was also running our software (polkalert) we're developing for validator monitoring with each of the nodes. I was not sure about the data we got while running it on a single machine (which worked), so I tried to install it on separate machines.

New chain spec was overkill for me, because I could do that perfectly with just --chain local setting and Alice and Bob accounts. I made it work in the end, but it took a while.

[JoshOrndorff]

Glad you've got it working now! If you'd like to use the local option without making any edits, you can quickly create a distributable chainspec in a single command.

./target/release/polkadot build-spec --chain local --raw > localSpecRedistributable.json

I'll close this for now.

[jak-pan]

That I did not know and will be very helpful next time! Thank you. (should have read on in the tutorial)