Tendermint: Latest versions fill up temporary storage rapidly to the point of crashing

[bijlar]

BUG REPORT

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):
0.13.0 and 0.14.0

Environment:
Windows

What happened:
I recently updated from 0.10.3 to 0.13.0 and later to 0.14.0 when that was available, but both of these experience the same behavior. After I set up a single node and enter a bunch of data, my temporary storage drive starts filling up rapidly (3GB+ for around 20k tx's, which have never taken up more than 130MB of data when looking at the .tendermint folder). Since I'm running this on a basic VM, I only have a bit of space for my temporary storage drive and after it fills up entirely Tendermint crashes with the following output.

What you expected to happen:
When working with 0.10.3, I never experienced this. I even went back to check and it doesn't even touch my temporary storage drive when inputting the exact same data. So I expected it to be the same for this version.

How to reproduce it (as minimally and precisely as possible):
A single node, running on a basic VM receiving about 20k tx's.

Anything else do we need to know:
This VM has a C: drive of 126GB which doesn't get used apart from the normal storage things you'd expect. However the (already half filled) D: drive "Temporary Storage" only contains a total of 7GB, which causes Tendermint to crash after it fills up all the way. As soon as I close the Tendermint application, the D: drive instantly clears up.

[melekes]

How big are transactions?

[melekes]

20k tx's given 1byte tx = 20000 * 3600 * 24 = 1 728 000 000 ~ 1,6GB of raw tx data per day
20k tx's given 1KB tx = 20000 * 3600 * 24 = 1 728 000 000 ~ 1647GB of raw tx data per day

[melekes]

Could you provide a script?

[bijlar]

I'm not entirely sure how to measure that, but if I just paste an input in http://bytesizematters.com/ it comes to about 335-350 bytes give or take. Every input is roughly the same in terms of size, so that shouldn't fluctuate too much.

What surprised me is that the previous version I was using (0.10.3) didn't seem to use the temporary storage whatsoever. I already tried pushing 50k of the same transactions before in a short amount of time and it worked just fine. But with the new versions that suddenly is becoming a big issue.

Also I'm using a custom application to simulate user input, so I'm not sure how to provide a script in this case. I can just give you an example of input though. Because of my use case, I encrypt the data before it gets to the ABCI, so it looks like this:

{
  "MessageType": "Encrypted",
  "SessionHash": "UkYlb5Dfnc2Lg0g67mhYz2yFgzonLJ0rMUQIoI9mnuk=",
  "Input": "coDkuLYueCoWYs554PH1B0pAsoF7TJID7HVVKvmoQFFaLfi+nU8pFzqdio7nXTX8TK1lEKABrwnammDLXbzU2Y91aJ8OoSSMPW6QdsUaORtNN/i4q4RmcM/2GBxML1fv3Iv6KXYzePZLOQ54dNoDHdx/qRqPEGunI5L7dMewIV5V6/iBWMDPMF3dQznJOTc5uKJusq/Kd8uPs59OUlLukrlW9bssRhhAcVUWA192yLM="
}

[melekes]

Can you print the disk usage stats for ~/.tendermint/data right before the crash and after? The analog on Linux:

du -h .tendermint
 12K    .tendermint/data/trusthistory.db
 64K    .tendermint/data/state.db
240K    .tendermint/data/blockstore.db
452K    .tendermint/data/cs.wal
4.0K    .tendermint/data/mempool.wal
 20K    .tendermint/data/tx_index.db
792K    .tendermint/data
808K    .tendermint

https://serverfault.com/questions/298876/is-there-a-windows-equivalent-to-the-unix-du-command

[bijlar]

This is leading up to the crash, it's very hard to get it right before the crash, but this is pretty close.

 0.56 MB   .tendermint/data/state.db
 5.09 MB   .tendermint/data/blockstore.db
 7.36 MB   .tendermint/data/cs.wal
 6.48 MB   .tendermint/data/mempool.wal
 2.45 MB   .tendermint/data/tx_index.db
21.95 MB   .tendermint/data
21.96 MB   .tendermint

And right after it crashed.

 0.70 MB   .tendermint/data/state.db
 3.97 MB   .tendermint/data/blockstore.db
 8.59 MB   .tendermint/data/cs.wal
 7.48 MB   .tendermint/data/mempool.wal
 5.24 MB   .tendermint/data/tx_index.db
25.96 MB   .tendermint/data
26.00 MB   .tendermint

As you can see I also included what the drives looked like before and after. I´m pretty sure that´s why it´s crashing, because it´s running out of temporary storage space. However I don´t understand why that is suddenly an issue. As I said before, in the previous version I was using (0.10.3) the temporary storage wasn't even being used at all. Also as a side note, if I stop my data insertion before it crashes and then restart Tendermint, it also cleans out the temporary storage drive.

[melekes]

Thank you!

Now we know that at least data inside .tendermint folder is not growing rapidly. Can it be /tmp analog for Windows?

restart Tendermint, it also cleans out the temporary storage drive.

looks very much like so.

[melekes]

Can you do the same thing (I mean du -h) for the whole D: drive?

[bijlar]

That's the odd thing, when I look at the D: drive, the only file there is one called DATALOSS_WARNING_README.txt which just contains some generic Microsoft warning about how the data stored on this drive is subject to loss. Even though that file is only 650 bytes and the drive is clearly a lot more full than that, I can't find any other files or folders, no matter what tool I use to look at the drive.

[ebuchman]

@melekes do you think this is fixed by the recent memleak fixes? Weird that they didn't notice anything in 0.10.3 - I don't think the memleak stuff was introduced after that ?

Anyways, this is hopefully fixed on develop. @bijlar can you try it out and let us know? Thanks!

[bijlar]

@ebuchman I tried to checkout and install the develop version (0.16.0-dev), but when I start the ABCI and run tendermint node it results in an error in the ABCI:

E[02-20|09:26:05.990] Connection error
module=abci-server error="Error reading message: Varint overflow"

And a corresponding error in the Tendermint node:

E[02-20|09:26:05.994] Stopping abci.socketClient for error: EOF
module=abci-client connection=query

After this both applications still seem to be doing something, but when I try to reach the blockchain through port 46657 like usual, it doesn't work. Checking my ports also reveals this port is not in use by any application.
Maybe I made some errors while installing the develop version, but I can't seem to get it to work as it stands.

[melekes]

@bijlar thank you for trying. It looks very much like compatibility issue. What ABCI are you using? Tendermint 0.16-dev might not work with older versions (latest: https://github.com/tendermint/abci/tree/develop)

[ebuchman]

^^ You'll need to use the latest abci version to be compatible, sorry for not being clear

[bijlar]

@melekes @ebuchman Sorry for the delay, I've been trying to get the ABCI to work but I either keep running into the error I provided above or a ton of different other errors probably related to some library that has probably not been updated properly. I'll try to do a wipe of my existing libraries at some point and update them all to the latest version to see if that fixes it. To be clear; do I still need to use any versions from the develop branch or is it alright to use the master branch now that version 0.16 has been released?

[ebuchman]

You should be fine to use master across the board for tendermint repos. v0.16 should compile completely with master (ie. you should be able to just go get it :) )

[odeke-em]

How's it going here @bijlar?

[bijlar]

@odeke-em I haven't had time to test it further I'm afraid. When I do I'll make sure to report back on my findings.

[isaack0329]

@melekes ,@ebuchman I also experienced the same problem as bijlar. I unzipped the latest version at (latest: https://github.com/tendermint/abci/tree/develop) <<.
How do you install the latest version of abcdi?

[isaack0329]

@ebuchman help me please...
<< Ethermint >>>
ethermint --datadir ~/.ethermint --rpc --rpcaddr=0.0.0.0 --ws --wsaddr=0.0.0.0 --rpcapi eth,net,web3,personal,admin
INFO [03-15|18:00:10] Starting peer-to-peer node instance=ethermint/v1.6.7-stable/linux-amd64/go1.8.3
INFO [03-15|18:00:10] Allocated cache and file handles database=/root/.ethermint/ethermint/chaindata cache=128 handles=1024
INFO [03-15|18:00:10] Initialised chain configuration config="{ChainID: 15 Homestead: 0 DAO: DAOSupport: false EIP150: EIP155: 0 EIP158: 0 Metropolis: Engine: unknown}"
WARN [03-15|18:00:10] Ethash used in fake mode
INFO [03-15|18:00:10] Initialising Ethereum protocol versions="[63 62]" network=1
INFO [03-15|18:00:10] Loaded most recent local header number=0 hash=2b72f9…d468ac td=64
INFO [03-15|18:00:10] Loaded most recent local full block number=0 hash=2b72f9…d468ac td=64
INFO [03-15|18:00:10] Loaded most recent local fast block number=0 hash=2b72f9…d468ac td=64
INFO [03-15|18:00:10] Starting P2P networking
INFO [03-15|18:00:10] RLPx listener up self="enode://1c55fa69494b0fe1a2356195da71f925ff3d8ae1d94327ab72a8d32fc28de0031811fa1ae3d28f0a44e02fca616e17a35fc2aaf1fc231abdba98c91adb4127d6@[::]:30303?discport=0"
INFO [03-15|18:00:10] Waiting for tendermint endpoint to start err="Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused"
INFO [03-15|18:00:10] HTTP endpoint opened: http://0.0.0.0:8545
INFO [03-15|18:00:10] IPC endpoint opened: /root/.ethermint/geth.ipc
INFO [03-15|18:00:10] WebSocket endpoint opened: ws://0.0.0.0:8546
INFO [03-15|18:00:12] Starting ABCIServer module=abci-server impl=ABCIServer
INFO [03-15|18:00:12] Waiting for new connection... module=abci-server
INFO [03-15|18:00:13] Waiting for tendermint endpoint to start err="Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused"
INFO [03-15|18:00:13] Accepted a new connection module=abci-server
INFO [03-15|18:00:13] Waiting for new connection... module=abci-server
INFO [03-15|18:00:13] Accepted a new connection module=abci-server
INFO [03-15|18:00:13] Waiting for new connection... module=abci-server
INFO [03-15|18:00:13] Accepted a new connection module=abci-server
INFO [03-15|18:00:13] Waiting for new connection... module=abci-server

ERROR[03-15|18:00:13] Connection error module=abci-server error="Error reading message: Varint overflow"

INFO [03-15|18:00:16] Waiting for tendermint endpoint to start err="Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused"
INFO [03-15|18:00:19] Waiting for tendermint endpoint to start err="Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused"
INFO [03-15|18:00:22] Waiting for tendermint endpoint to start err="Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused"
^Ccaptured interrupt, exiting...
INFO [03-15|18:00:24] Stopping ABCIServer module=abci-server impl=ABCIServer

---

tendermint --home ~/.ethermint/tendermint node
I[03-15|09:00:13.963] Starting multiAppConn module=proxy impl=multiAppConn
I[03-15|09:00:13.963] Starting socketClient module=abci-client connection=query impl=socketClient
I[03-15|09:00:13.963] Starting socketClient module=abci-client connection=mempool impl=socketClient
I[03-15|09:00:13.963] Starting socketClient module=abci-client connection=consensus impl=socketClient
E[03-15|09:00:13.964] Stopping abci.socketClient for error: EOF module=abci-client connection=query
I[03-15|09:00:13.964] Stopping socketClient module=abci-client connection=query impl=socketClient
E[03-15|09:00:24.539] Stopping abci.socketClient for error: EOF module=abci-client connection=consensus
I[03-15|09:00:24.539] Stopping socketClient module=abci-client connection=consensus impl=socketClient
E[03-15|09:00:24.539] Stopping abci.socketClient for error: EOF module=abci-client connection=mempool
I[03-15|09:00:24.539] Stopping socketClient module=abci-client connection=mempool impl=socketClient

[isaack0329]

Currently my abci is 0.10.0 and I installed go get github.com/tendermint/abci. @ebuchman @odeke-em @bijlar

[ebuchman]

@isaack0329 Ethermint only works with Tendermint version 0.14.0. It has not been updated for anything higher than that. Please make sure you are using an older version of Tendermint (v0.14.0). This is noted in the Ethermint readme. Apologies for inconvenience

[brandonin]

I'm receiving these same issues when using following the instructions on http://cosmos-sdk.readthedocs.io/en/stable/basecoin-tool.html.

[ebuchman]

I believe this issue is fixed since a few versions ago. Please re-open or open a new one if it persists.

I'm receiving these same issues when using following the instructions on http://cosmos-sdk.readthedocs.io/en/stable/basecoin-tool.html.

Those docs (and probably that basecoin you're using) are out of date. See the new docs in progress at https://cosmos-sdk.readthedocs.io/en/develop/