Substrate: Problems creating staging or local chain

[avalverdero]

Hi!

I have problems starting a private network because the validators don't produce blocks. I tried following different Substrate and Parity documentations but doesn't work.

The validators can connect to other peers and no error are thrown.
I tested with versions:

• v2.0.0-alpha.2
• v2.0.0-alpha.1

and with the --chain staging and local.

followed these docs but nothing worked:

next version
pre-v2.0-3e65111

Thanks in advance

[bkchr]

So you started alice and bob?

Is there anything useful in logs?

[avalverdero]

So you started alice and bob?

No, I am starting a private chain with my own validators from scratch, creating first my spec.json.

This are the commands used to create my spec.json (tested with staging and local chain)

./target/release/substrate build-spec --chain staging > stagingSpec.json

After the creation of spec file, I modified it with my own root and validators keys.

./target/release/substrate build-spec --chain ./stagingSpec.json --raw > stagingSpecRaw.json
./target/release/substrate --base-path /tmp/validator1 --chain ./stagingSpecRaw.json --validator --name validator1

take in account that I modified the specFile to have a minimalValidators number setted to 2 and I started with 2 validators nodes (the minimun required)

Is there anything useful in logs?

Nothing more than the tipical log showing the connection between the peers and the number of blocks synced

[bkchr]

Did you add your keys? I would propose to test with alice and bob. If that works, try with your own chain spec.

[avalverdero]

Yes, I added my keys.

I tested with alice and bob but it doesn't work with chain staging( I changed minimumValidatorCount to 2). But with local chain works fine. Anyway I tested also using chain-spec-builder and the result is the same.

I propose several points/questions to clarify my doubts or find the problem that perhaps are causing this behavior:

• Can you provide a minimal procedure to create a private and decentralized network with substrate?

• I can see that the palletBabe and palletGrandpa are empty inside the Spec file template, this is correct?

• I only modified the keys of the template replacing it by my validators and sudo keys

• Maybe the keys are the problem, if you can clarify the correct keys to put into each line of the spec file template I will be thankful. This is the template that Substrate generates for a Staging chain:

"palletBabe": {
    "authorities": []
  },
  "palletIndices": {
    "indices": []
  },
  "palletBalances": {
    "balances": [
      [
        "5Ff3iXP75ruzroPWRP2FYBHWnmGGBSb63857BgnzCoXNxfPo",
        1000000000000000000000
      ],
      [
        "5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy",
        10000000000000000
      ],
      [
        "5ERawXCzCWkjVq3xz1W5KGNtVx2VdefvZ62Bw1FEuZW4Vny2",
        10000000000000000
      ],
      [
        "5DyVtKWPidondEu8iHZgi6Ffv9yrJJ1NDNLom3X9cTDi98qp",
        10000000000000000
      ],
      [
        "5HYZnKWe5FVZQ33ZRJK1rG3WaLMztxWrrNDb1JRwaHHVWyP9",
        10000000000000000
      ]
    ]
  },
  "palletStaking": {
    "validatorCount": 2,
    "minimumValidatorCount": 2,
    "invulnerables": [
      "5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy",
      "5ERawXCzCWkjVq3xz1W5KGNtVx2VdefvZ62Bw1FEuZW4Vny2",
      "5DyVtKWPidondEu8iHZgi6Ffv9yrJJ1NDNLom3X9cTDi98qp",
      "5HYZnKWe5FVZQ33ZRJK1rG3WaLMztxWrrNDb1JRwaHHVWyP9"
    ],
    "currentEra": 0,
    "forceEra": "NotForcing",
    "slashRewardFraction": 100000000,
    "canceledPayout": 0,
    "stakers": [
      [
        "5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy",
        "5EnCiV7wSHeNhjW3FSUwiJNkcc2SBkPLn5Nj93FmbLtBjQUq",
        10000000000000000,
        "Validator"
      ],
      [
        "5ERawXCzCWkjVq3xz1W5KGNtVx2VdefvZ62Bw1FEuZW4Vny2",
        "5Gc4vr42hH1uDZc93Nayk5G7i687bAQdHHc9unLuyeawHipF",
        10000000000000000,
        "Validator"
      ],
      [
        "5DyVtKWPidondEu8iHZgi6Ffv9yrJJ1NDNLom3X9cTDi98qp",
        "5FeD54vGVNpFX3PndHPXJ2MDakc462vBCD5mgtWRnWYCpZU9",
        10000000000000000,
        "Validator"
      ],
      [
        "5HYZnKWe5FVZQ33ZRJK1rG3WaLMztxWrrNDb1JRwaHHVWyP9",
        "5EPQdAQ39WQNLCRjWsCk5jErsCitHiY5ZmjfWzzbXDoAoYbn",
        10000000000000000,
        "Validator"
      ]
    ]
  },
  "palletSession": {
    "keys": [
      [
        "5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy",
        "5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy",
        {
          "grandpa": "5Fb9ayurnxnaXj56CjmyQLBiadfRCqUbL2VWNbbe1nZU6wiC",
          "babe": "5EZaeQ8djPcq9pheJUhgerXQZt9YaHnMJpiHMRhwQeinqUW8",
          "im_online": "5EZaeQ8djPcq9pheJUhgerXQZt9YaHnMJpiHMRhwQeinqUW8",
          "authority_discovery": "5EZaeQ8djPcq9pheJUhgerXQZt9YaHnMJpiHMRhwQeinqUW8"
        }
      ],
      [
        "5ERawXCzCWkjVq3xz1W5KGNtVx2VdefvZ62Bw1FEuZW4Vny2",
        "5ERawXCzCWkjVq3xz1W5KGNtVx2VdefvZ62Bw1FEuZW4Vny2",
        {
          "grandpa": "5EockCXN6YkiNCDjpqqnbcqd4ad35nU4RmA1ikM4YeRN4WcE",
          "babe": "5DhLtiaQd1L1LU9jaNeeu9HJkP6eyg3BwXA7iNMzKm7qqruQ",
          "im_online": "5DhLtiaQd1L1LU9jaNeeu9HJkP6eyg3BwXA7iNMzKm7qqruQ",
          "authority_discovery": "5DhLtiaQd1L1LU9jaNeeu9HJkP6eyg3BwXA7iNMzKm7qqruQ"
        }
      ],
      [
        "5DyVtKWPidondEu8iHZgi6Ffv9yrJJ1NDNLom3X9cTDi98qp",
        "5DyVtKWPidondEu8iHZgi6Ffv9yrJJ1NDNLom3X9cTDi98qp",
        {
          "grandpa": "5E1jLYfLdUQKrFrtqoKgFrRvxM3oQPMbf6DfcsrugZZ5Bn8d",
          "babe": "5DhKqkHRkndJu8vq7pi2Q5S3DfftWJHGxbEUNH43b46qNspH",
          "im_online": "5DhKqkHRkndJu8vq7pi2Q5S3DfftWJHGxbEUNH43b46qNspH",
          "authority_discovery": "5DhKqkHRkndJu8vq7pi2Q5S3DfftWJHGxbEUNH43b46qNspH"
        }
      ],
      [
        "5HYZnKWe5FVZQ33ZRJK1rG3WaLMztxWrrNDb1JRwaHHVWyP9",
        "5HYZnKWe5FVZQ33ZRJK1rG3WaLMztxWrrNDb1JRwaHHVWyP9",
        {
          "grandpa": "5DMa31Hd5u1dwoRKgC4uvqyrdK45RHv3CpwvpUC1EzuwDit4",
          "babe": "5C4vDQxA8LTck2xJEy4Yg1hM9qjDt4LvTQaMo4Y8ne43aU6x",
          "im_online": "5C4vDQxA8LTck2xJEy4Yg1hM9qjDt4LvTQaMo4Y8ne43aU6x",
          "authority_discovery": "5C4vDQxA8LTck2xJEy4Yg1hM9qjDt4LvTQaMo4Y8ne43aU6x"
        }
      ]
    ]
  },
  "palletDemocracy": {},
  "palletCollectiveInstance1": {
    "phantom": null,
    "members": [
      "5Ff3iXP75ruzroPWRP2FYBHWnmGGBSb63857BgnzCoXNxfPo"
    ]
  },
  "palletCollectiveInstance2": {
    "phantom": null,
    "members": [
      "5Ff3iXP75ruzroPWRP2FYBHWnmGGBSb63857BgnzCoXNxfPo"
    ]
  },
  "palletMembershipInstance1": {
    "members": [],
    "phantom": null
  },
  "palletGrandpa": {
    "authorities": []
  },
  "palletTreasury": {},
  "palletContracts": {
    "currentSchedule": {
      "version": 0,
      "put_code_per_byte_cost": 1,
      "grow_mem_cost": 1,
      "regular_op_cost": 1,
      "return_data_per_byte_cost": 1,
      "event_data_per_byte_cost": 1,
      "event_per_topic_cost": 1,
      "event_base_cost": 1,
      "call_base_cost": 135,
      "instantiate_base_cost": 175,
      "sandbox_data_read_cost": 1,
      "sandbox_data_write_cost": 1,
      "transfer_cost": 100,
      "max_event_topics": 4,
      "max_stack_height": 65536,
      "max_memory_pages": 16,
      "max_table_size": 16384,
      "enable_println": false,
      "max_subject_len": 32
    },
    "gasPrice": 1000000000
  },
  "palletSudo": {
    "key": "5Ff3iXP75ruzroPWRP2FYBHWnmGGBSb63857BgnzCoXNxfPo"
  },
  "palletImOnline": {
    "keys": []
  },
  "palletAuthorityDiscovery": {
    "keys": []
  },
  "palletSociety": {
    "pot": 0,
    "maxMembers": 999,
    "members": [
      "5Ff3iXP75ruzroPWRP2FYBHWnmGGBSb63857BgnzCoXNxfPo"
    ]
  },
  "palletVesting": {
    "vesting": []
  }
}

Thanks in advance

[bkchr]

The staging one does not works with Alice and Bob. This one only works with the keys, you don't have access too^^

It seems that you are not using the node-template? I recommend that you use the node-template and follow exactly the steps here: https://substrate.dev/docs/en/next/tutorials/start-a-private-network/customchain This tutorial directly explains how and which keys you need to replace and how to register the keys at your nodes.

[avalverdero]

The staging one does not works with Alice and Bob. This one only works with the keys, you don't have access too^^

Yes, I can imagine it because staging is the chain that you provide to deploy a ready to production environment chain.

Ok, using node-template works but don't have any module available is not a "full node".

I found this :

Use the Full Node
You can go through a very similar process using Substrate's full node that lives in the node directory rather than the node-template that we used. The process should be familiar to you, and you'll get to explore more runtime modules that are included in the full node

Ref: https://substrate.dev/docs/en/tutorials/start-a-private-network-with-substrate#use-the-full-node

How I can implement a network with all the modules up and running with my own validators? Is the thing I am trying to do using a staging chain with substrate runtime instead node-template.

[bkchr]

@JoshOrndorff can you help here?

[JoshOrndorff]

Sure I can try to help. Thanks for the ping. Let me start by answering a few questions that weren't directly answered yet.

I can see that the palletBabe and palletGrandpa are empty inside the Spec file template, this is correct?

This is correct when you are using the session pallet. The chainspec you pasted above is using the session pallet so it is correct to leave the Babe and Grandpa configs empty in that case. You've probably noticed that this process is different than it was with the node template. That's because the node template does _not_ use the session pallet. So in your case, you need to enter the custom keys in the session config rather than directly into the babe and grandpa configs.

I only modified the keys of the template replacing it by my validators and sudo keys

That is correct, but let's be sure you've modified all of the right parts of the spec. In the full node you will need to modify the session pallet config (as just discussed) and you will also want to modify the staking pallet config to include your same validators. If you fail to modify the staking pallet, then after the first session ends, you will not have the same validators you specified in genesis.

Looking in detail at the spec you posted it seems you want 5 accounts that will have funds at genesis. The last 4 of those accounts will be bonded as validators, and use the same account for the stash and controller (aside: that's dangerous in the wild, but fine for testing, and definitely not your current problem). Those same last 4 validators are all included in the session pallet config which is also correct. I'm not sure how you got the actual session keys that you've included there though. Maybe you generated them with subkey?

The one thing you haven't mentioned yet is inserting the keys into each node's keystore. You will need to make an RPC call to insert each session key into each node. Since you have 4 validator nodes and 4 session keys (gran, babe, imol, audi) you will need to make a total of 16 calls to author_insertKey. The process will be similar to what's described here https://substrate.dev/docs/en/tutorials/start-a-private-network/customchain#add-keys-to-keystore

Finally, I just noticed that the link you quoted above accidentally points to an outdated tutorial that targets Substrate v1.0. If you've been trying to use the --key commandline argument, that's your problem. The link I pasted just above goes to the current version of the tutorial.

Good luck, and I'm happy to help more if you need it.

[JoshOrndorff]

One more thought, A good sanity check is to ensure that each node is expecting the same genesis block. When your nodes start up you'll see a line similar to Initializing Genesis block/state (state: 0x4a75…ea43, header-hash: 0x05e7…019e) Make sure that each of your nodes has the same hashes there. If they have different hashes, let's diagnose that first.

[avalverdero]

Hi @JoshOrndorff , thanks for the reply

Maybe you generated them with subkey?

yes, I am creating all the keys with subkey.

Maybe this is the error, the documentation don't specify it and I only did it for granpa and babe keys.

Since you have 4 validator nodes and 4 session keys (gran, babe, imol, audi) you will need to make a total of 16 calls to author_insertKey

About all the info you shared with me, I have doubts: I would like to have an answer before to do all the procedure of start a new private chain:

• What have to use instead of "--key" after I will do the RPC calls? and what expects the "--node-key" parameter. We must use it passing the Session Key (ImOnline), right?

• You can confirm that the keys specified and type are correct please?

"keys": [
[
"5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy", # STASH KEY (SR25519)
"5Fbsd6WXDGiLTxunqeK5BATNiocfCqu9bS1yArVjCgeBLkVy", # CONTROLLER KEY (SR25519)
{
"grandpa": "5Fb9ayurnxnaXj56CjmyQLBiadfRCqUbL2VWNbbe1nZU6wiC", # GRANDPA KEY (ED25519)
"babe": "5EZaeQ8djPcq9pheJUhgerXQZt9YaHnMJpiHMRhwQeinqUW8", # BABE KEY (SR25519)
"im_online": "5EZaeQ8djPcq9pheJUhgerXQZt9YaHnMJpiHMRhwQeinqUW8", # IOM KEY (SR25519)
"authority_discovery": "5EZaeQ8djPcq9pheJUhgerXQZt9YaHnMJpiHMRhwQeinqUW8" # AUTH DISC. KEY (SR25519)
}
],`

• All the keys above should be generated from same secret seed?

Thanks in advance

[JoshOrndorff]

The --key flag was removed after v1.0. It is replaces with the author_insertKey RPC call.

The --node-key flag supplies your node with a libp2p key for communicating with other nodes. The format is described when running substrate --help, but if your nodes are peering, the --node-key flag is not a problem. That key is _not_ associated with I'm Online. Your I'm Online key should be inserted via author_insertKey just like the other session keys.

You may be able to get some block authoring without inserting all 4 session keys, I'm not sure. But if your node has 4 session keys, it will eventually expect them. For example, your validators will be slashed for being offline unless you give them keys for I'm online. That's not in the tutorial because the node template only has two session keys (aura and grandpa). The full node which you're currently using has four. It may be a useful experiment to see whether the nodes will author while missing some of their session keys. But for now, since you're having trouble, best to insert all of them (total of 16 rpc calls).

There isn't a single "right" set of keys, but here are some properties that are right about the snippet you posted.

• Each stash key has 4 associated session keys :heavy_check_mark:
• The babe, imol, and audi keys use the same sr25519 crypto :heavy_check_mark:
• The grandpa key uses different ed25519 crypto :heavy_check_mark:

It isn't _necessary_ that all the keys be generated from the same phrase, and in a production network it is best that they aren't. But for getting this testnet started, it's fine that they are, and it simplifies the process.

[Swader]

Wondering about something here - if the full node needs a different genesis (with session populated instead of gran/aura and with staking populated), does this mean that, contrary to a lot of the docs out there, it is not possible to pull in session and staking as a runtime upgrade of node-template?

[bkchr]

What is the full node? Genesis always needs to be the same across a network.

[avalverdero]

I only mentioned a full node because is the terminology used by this documentation Ref

But is true that as @bkchr said "Genesis always needs to be the same across a network.". I think that when the documentation (in this case) speak about full node, means a node with all the SRML modules included on the runtime ( basically this )

I hope I resolved your doubts @Swader

@JoshOrndorff thanks for all the usefull info you shared with me. I will try to start our network with this knowledge and come back to the ticket to inform about the results.

Many thanks to all of you.

[JoshOrndorff]

@Swader The following scenario is possible (and I think what you're asking about)

• Start a minimal node with Aura and Grandpa, but no session pallet.
• Upgrade the runtime to include the session pallet and something like staking or POA

The second runtime in that scenario will not need a genesis configuration because it did not start being used at genesis. Rather it will use the blockchain's existing state as the previous runtime left it.

I've occasionally used the term "full node" to refer to the node that lives in substrate/bin/node. Probably "main Substrate node" would be a better temr.

[avalverdero]

Hi! @JoshOrndorff @bkchr

finally works adding all the session keys of the nodes using author_insertKey :D

But before close the ticket, I have two more doubts:

Scenario: I have a network with 2 validator nodes, 2 sentry nodes, 1 bootnode and other full node (with a public websocket for our apps).

Question1: Do I insert the keys of sentry, bootnode and full node as I did it with validator nodes using author_insertKey? In case affirmative, how many keys need each node, I think only one with SR cryptography, this is right?

Question2: Do you have any documentation related with how to upgrade the runtime of this network type? I need do the upgrade first in bootnodes or validators?

Thanks in advance.

[JoshOrndorff]

Hooray! Glad to hear you're up and running :)

Answer 1: Only the validator nodes need session keys. You will not have to insert keys into non-validating nodes. But do realize that the roles you listed are not mutually exclusive. For example a validator node could, in some networks, also be a bootnode.

Answer 2: The beauty of on-chain runtime upgrades is that you only do it once. You submit the new runtime as part of a transaction and all the nodes sync it and upgrade automatically. The simplest way to submit the upgrade is to use sudo can call system::set_code.

There isn't a doc that walks you through exactly that, but here is some helpful material.
Conceptual docs on runtime upgrades: https://substrate.dev/docs/en/next/conceptual/core/executor#runtime-upgrades
Video of doing a runtime upgrade at Substrate Seminar: https://www.youtube.com/watch?v=yHodYke7nms&list=PLp0_ueXY_enXRfoaW7sTudeQH10yDvFOS&index=8&t=240s

[avalverdero]

Hi !@JoshOrndorff @bkchr

now we are working with the new version of Substrate as expected.

A lot of worth information to me in this ticket.

Many thanks for the help!

[rahul-soshte]

I followed the same instructions all the way but block are still not finalizing I mean I inserted the session keys using the curl command but I dont know why it is not finalizing. Blocks are being authored tho.

How do I add the stash and the controller keys into the keystore. Do blocks finalize only if you just add the session keys and not the stash and controller keys? How does the session keys they are associated with a stash or controller key.

[danforbes]

Stash and controller keys are typically something that is used for a staking system, I believe. In this case, the keys that are being added are for a proof-of-authority system. Stash and controller keys are an abstraction that help you separate funds from an account that controls them. Proof-of-authority keys are used for identification purposes.