Jormungandr: All the transactions remain in `Pending`

[dorin100]

Describe the bug
All the transactions(transfer funds, send certificate, delegate, etc) are remaining in Pending even multiple new blocks are created. Only the transaction from getting money from web faucet are included in a block.

Mandatory Information

jcli 0.5.2 (HEAD-e7a3efe7, release, windows [x86_64]) - [rustc 1.37.0 (eae3437df 2019-08-13)]
jormungandr 0.5.2 (HEAD-e7a3efe7, release, windows [x86_64]) - [rustc 1.37.0 (eae3437df 2019-08-13)]

To Reproduce
Steps to reproduce the behavior:

1. start the node using: jormungandr ---config node_config.yaml --genesis-block-hash adbdd5ede31637f6c9bad5c271eec0bc3d0cb9efb86a5b913bb55cba549d0770
2. extract the attached archive into the same folder with the node config
3. execute the create account and delegate script: bash create-account-and-delegate.sh
4. the same scripts worked until v0.5.0
5. I updated the genesis_hash, faucet private key (with personal account details with funds) and stakePoolID with valid values --> the transactions are correctly created but they remain in Pending
6. wait for at least 1 new block to be created and check node stats and logs --> all the transactions are remaining stuck in Pending

Expected behavior
All the transactions should be executed - be included in a block or rejected with a valid message/reason.

Additional context

• OS: win 10; local node has an open connection with the outside/internet
• node logs:
  node_logs.txt

• script execution logs:
  account_delegation_script_logs.txt

• scripts_used:
  scripts_used.zip

• node config

rest:
  listen: "127.0.0.1:9000"
p2p:
  public_address: /ip4/5.12.xx.xx/tcp/3000
  listen_address: /ip4/192.168.xx.xx/tcp/3000
  trusted_peers: [/ip4/3.123.177.192/tcp/3000,
                  /ip4/3.123.155.47/tcp/3000,
                  /ip4/52.57.157.167/tcp/3000,
                  /ip4/3.112.185.217/tcp/3000,
                  /ip4/18.140.134.230/tcp/3000,
                  /ip4/18.139.40.4/tcp/3000,
                  /ip4/3.115.57.216/tcp/3000]
  topics_of_interest:
    messages: high
    blocks: high

[rdlrt]

You've probably checked, but just for completion... do the fee/cert constants in the script match the results from jcli settings?

[dorin100]

Hi @rdlrt . You are right, the fees were updated and they were different from the ones from my scripts. Anyway, I updated the scripts with the nee fee values but the behavior is the same.

[dorin100]

Ok. this seems related to the fees. So if the correct fees are provided for each type of transaction, then everything is ok.

The user experience is still lacking when the transaction is Rejected because this info is not propagated back to the initiator that can only see a transaction with Status: Pending

[NicolasDP]

did you look at the configuration for the mempool ?

mempool:
    # this value sets for how long your fragment logs will stay
    log_ttl: 1h
    # this value sets for how long the fragment will remaining pending at most
    fragment_ttl: 20m
    # this will ask the node to perform garbage collection on the logs and fragment every...
    garbage_collection_interval: 20m

[dkijania]

I'm also experiencing mentioned issue on 0.5.2 on my local private testnet. I'm sure fragment is correct:

λ jcli rest v0 message logs -h http://127.0.0.1:8000/api
---
- fragment_id: d7df73a5959ac6dcce4765b42c4f483d13c77add23b70f94bb1a39111946fb50
  last_updated_at: "2019-09-28T13:07:48.714351600+00:00"
  received_at: "2019-09-28T13:07:48.714351600+00:00"
  received_from: Rest
  status: Pending

C:\Work\iohk\jormungandr\jormungandr-scenario-tests (jts -> origin)
λ jcli rest v0 message logs -h http://127.0.0.1:8001/api
---
- fragment_id: d7df73a5959ac6dcce4765b42c4f483d13c77add23b70f94bb1a39111946fb50
  last_updated_at: "2019-09-28T13:08:01.008176200+00:00"
  received_at: "2019-09-28T13:07:48.718354600+00:00"
  received_from: Network
  status:
    InABlock:
      date: "0.23"

The fragment is in pending state for more than 10 mins.
This issue was fix already in 0.3.x version i believe and looks like it reappeared?

[mmahut]

@dkijania do you mind sharing jcli transaction info of your TX?

[dorin100]

@NicolasDP - regarding mempool, it will not be any better if I don't see the transaction at all in node stats comparing to seeing it in Pending.

What would help me would be to see the real state of the transaction (to see the Reason why the transaction was Rejected) on the local/source/initiator node.

[NicolasDP]

No it is not possible to do that at the moment. We are looking into trying to provide early informations before propagations, but again this is not a good solutions anyway as what isnot valid for a node at a given point may be valid later for that node (or any other node).

[NicolasDP]

@dorin100 are we still having issues with fragment propagation?

[dorin100]

Hi @NicolasDP . Yes, when 1 transaction is Rejected on a stake pool node, the status is not propagated also on the original/source node for the transactions -> there the status is remaining in Pending until the tx is removed based on mempool settings.

ex:

• stake pool node (the tx was failed on purpose in this case)

$ jcli rest v0 message logs --host "http://127.0.0.1:9001/api"
---
- fragment_id: d047a824e35cf56ef8d06633c5aab490a90f7b0d5014e873cbb860d917dd48a3
  last_updated_at: "2019-11-11T11:41:20.001562200+00:00"
  received_at: "2019-11-11T11:41:18.896480800+00:00"
  received_from: Network
  status:
    Rejected:
      reason: "Failed to validate transaction balance: transaction value not balanced, has inputs sum 1110 and outputs sum 2100"

• on source node

$ jcli rest v0 message logs --host "http://127.0.0.1:9002/api"
---
- fragment_id: d047a824e35cf56ef8d06633c5aab490a90f7b0d5014e873cbb860d917dd48a3
  last_updated_at: "2019-11-11T11:41:18.896480800+00:00"
  received_at: "2019-11-11T11:41:18.896480800+00:00"
  received_from: Rest
  status: Pending

[rinor]

Is it ok to continue this discussion in here or should we file new ones? I'm asking because I think that we have 2 different case scenarios in here:

• @dkijania comment https://github.com/input-output-hk/jormungandr/issues/844#issuecomment-536192695 is still reproducible under some scenarios, but in my opinion, is different from:
• @dorin100 comment https://github.com/input-output-hk/jormungandr/issues/844#issuecomment-552411760

[NicolasDP]

@dorin100 , please don't change the subject of an issue, this is really misleading.

What you are after is a new feature: somehow have all the nodes omniscient about every other nodes' reason to reject a transaction. This is impossible. Especially that a transaction is broadcasted to as many nodes as possible in order to allow as much chance for the fragment (transaction) to be added in a block.

The fact that you managed to build an un-balanced transaction is an error of the wallet's settings.

As for the original issue: it was due to network issues and block/fragment propagation issues. As far as I can tell: this is now fixed.

[huwabee]

I am currently having all of my transactions created on pending states…
Also re-run my node over and over but nothing seems to change…