Cosmos-sdk: Node Tx Index unexpected behaviour

[fedekunze]

Summary of Bug

Ref: https://github.com/cosmos/voyager/issues/988#issuecomment-417002921

Example:
103.72.145.36:26657 shows tx-index=on but doesn't show txs.
195.201.141.141:26657 shows tx-index=on and shows txs.

Steps to Reproduce

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[ValarDragon]

Can you provide more info about the setups? What are the nodes configurations? Are they identical?

[jackzampolin]

I feel like I've also seen community reports of weird indexing behavior as well. Trying to find it right now.

[cwgoes]

I believe this is a Tendermint RPC endpoint which the SDK doesn't handle.

We could add additional info through ABCIQuery() if desired.

[faboweb]

Isn't Tendermint handling indexing? If not, adding an endpoint to the LCD is perfectly fine.

[faboweb]

From the awesome @xla of the Tendermint team:

http://103.72.145.36:26657/tx?hash=0x04C9AE23F7054F768376502813C9BAD30758805E
http://195.201.141.141:26657/tx?hash=0x04C9AE23F7054F768376502813C9BAD30758805E

both return the transactions. they would error if the indexer is not present. so this is probably an SDK issue.

[alexanderbez]

@faboweb or @fedekunze can you allude to the setup and versions of gaiad and gaiacli involved here?

[faboweb]

gaiacli version is 0.24.2-1c38c704
195.201.141.141 gaiad version is 0.24.2-1c38c70
103.72.145.36 gaiad version is 0.24.2-0bf061b -> not indexing

[alexanderbez]

Having trouble reproducing this. Running 0.24.2-a6120686,

I have a local testnet running (4 nodes), I verified all TM nodes have "tx_index=on", I sent some txs, I ran an LCD service through all 4 nodes and was able to query for all txs that I made via the LCD services.

Is there an easier way to reproduce this? @faboweb I noticed the node that doesn't return txs is running a different gaiad/gaiacli version.

[faboweb]

This is on the testnet gaia-8001. I don't know more then you. We can blacklist a certain version of the SDK, but I am not sure this is the desired way to go.

[alexanderbez]

Yeah but I'm not sure the version is the culprit. I can try that specific version 👍

EDIT: Was able to reproduce by running an LCD service using 0bf061b but found it was due to the underlying query being different (diff bech32 prefixing/encoding).

[fedekunze]

@alexanderbez I also just reproduced the error on Voyager with version gaiacli: 0.24.2-1c38c704

electron  tx_index=on
electron  Node localhost IS indexing transactions
          Picked node: localhost

How to reproduce error on Voyager

1. Install prerequisites
2. Build local-node automatically and copy the seed phrase
3. Go to the $HOME/.cosmos-voyager-dev folder and change the indexing values in config.toml within the specified tesnet folder
4. On the Terminal run:
   ```
   ./builds/Gaia/{OS}/gaiad start --home ~/.gaiad-testnet

5. On a separate tab run the local testnet:

yarn start local-testnet
```

[faboweb]

Go to the $HOME/.cosmos-voyager-dev folder and change the indexing values in config.toml within the specified tesnet folder

He meant go to ~/.gaiad-testnet/config/config.toml and change the indexing behavior there.

[alexanderbez]

@fedekunze @faboweb I ran gaiacli: 0.24.2-1c38c704 locally with 4 nodes and created some txs and was not able to reproduce this (I ran an LCD service on all 4 nodes). I also ran this Voyager (only a single node) and also was not able to reproduce this.

Note, this is my config:

indexer = "kv"
index_tags = ""
index_all_tags = true

[faboweb]

Your configuration is expected to produce an indexing node. The problem is, that a not indexing node is telling us, that it is indexing. The configuration @fedekunze used was:

indexer = "kv"
index_tags = "off"
index_all_tags = false

[alexanderbez]

@faboweb that wasn't mentioned anywhere. I'll look into this again later today 👍

[faboweb]

Another example of a by us controlled environment:
fabo.interblock.io is not indexing
http://fabo.interblock.io:26657/status shows tx_index=on
http://fabo.interblock.io:1317/stake/delegators/cosmosaccaddr1cqz8e6ahj44fpnfmm4uvm04uxc4hytx3ugffpr/txs returns [] It should return 2 transactions

Pulling in @greg-szabo

[alexanderbez]

@faboweb just to clarify, fabo.interblock.io is NOT indexing and yet it states it is AND querying for a delegation's tx should return a result but it doesn't?

Does the latter depend on indexing?

[faboweb]

fabo.interblock.io is NOT indexing and yet it states it is

yes

querying for a delegation's tx should return a result but it doesn't

this is a test case to show that the node is really not indexing. Using the node 8.9.36.157 via the LCD I see the transactions.

[ebuchman]

indexer = "kv"
index_tags = "off"
index_all_tags = false

This config technically has indexing enabled. It's using the kv indexer and will index any tx that returns a tag with key = "off".

To turn off indexing properly, you need to set indexer = "null".

Does that explain the issue here?

[faboweb]

@greg-szabo what is the config used in our node?

[faboweb]

index_tags = "off"

oha, thanks for pointing this out.

[greg-szabo]

I just turned on indexing on the fabo node.

[faboweb]

http://fabo.interblock.io:1317/stake/delegators/cosmosaccaddr1cqz8e6ahj44fpnfmm4uvm04uxc4hytx3ugffpr/txs still produces []

[ebuchman]

Indexing needs to be on from the start otherwise it won't have indexed old txs. If you send a new tx for the relevant account, can you then query for it?

[faboweb]

This is a strange behavior. How about: If the node detects, that it is indexing now, it resyncs and fills up the index?

[faboweb]

Indexing works now for the node.

[jackzampolin]

Can we close this issue now @faboweb @greg-szabo, or is there some some documentation you would like to see around this behavior?

[ebuchman]

This is a strange behavior. How about: If the node detects, that it is indexing now, it resyncs and fills up the index?

This could take a long time and may not actually be desired (eg. maybe you want to change what things you're indexing over time, and don't want to reindex the whole history). I think we just need to be wary of making sure indexing is on from when we want it.

[faboweb]

I think this is very confusing but I understand the technical difficulties. I created a follow up proposal: https://github.com/cosmos/cosmos-sdk/issues/2465

Thank you for your help you all.