Bigchaindb: Application of abnormal exit when testing concurrency

[xuhuigithub]

I built Bigchaindb cluster on four hosts of the system Centos7, which has four tendermint validitor, bigchaindb's version is bigchaindb 2.0.0a3, and tendermint's version is 0.19.2-64408a40. When I test the performance of database writes, bigchaindb takes the following error when it is subjected to concurrent (50 threads), and then tendermint drops out.

bigchaindb ERROR

[2018-05-10 06:03:04] [INFO] (bigchaindb.backend.mongodb.connection) Closing initial connection to MongoDB (webapi     - pid: 130)
[2018-05-10 06:03:04] [INFO] (bigchaindb.backend.mongodb.connection) Closing initial connection to MongoDB (webapi     - pid: 53)
ERROR     Server Error: Unexpected EOF while reading bytes
ERROR     Server Error: Error parsing message
ERROR     Server Error: Unexpected EOF while reading bytes
[2018-05-10 06:06:49] [ERROR] (abci.app)  Server Error: Unexpected EOF while reading bytes (MainProcess - pid: 1)
ERROR     Server Error: Error parsing message
[2018-05-10 06:06:49] [ERROR] (abci.app)  Server Error: Error parsing message (MainProcess - pid: 1)
/usr/local/lib/python3.6/site-packages/abci/encoding.py:62: RuntimeWarning: Unexpected end-group tag: Not all data was converted
  m.ParseFromString(slice)
ERROR     Server Error: getattr(): attribute name must be string
ERROR     Server Error: getattr(): attribute name must be string
ERROR     Server Error: Error parsing message
[2018-05-10 06:06:49] [ERROR] (abci.app)  Server Error: getattr(): attribute name must be string (MainProcess - pid: 1)
ERROR     Server Error: Error parsing message
ERROR     Server Error: Error parsing message
[2018-05-10 06:06:49] [ERROR] (abci.app)  Server Error: Error parsing message (MainProcess - pid: 1)
ERROR     Server Error: Error parsing message
ERROR     Server Error: Error parsing message
[2018-05-10 06:06:49] [ERROR] (abci.app)  Server Error: Error parsing message (MainProcess - pid: 1)
ERROR     Server Error: Error parsing message
ERROR     Server Error: Error parsing message
ERROR     Server Error: Error parsing message

Bigchaindb Config

{
    "server": {
        "bind": "0.0.0.0:9984",
        "loglevel": "info",
        "workers": 144
    },
    "wsserver": {
        "scheme": "ws",
        "host": "0.0.0.0",
        "port": 9985,
        "advertised_scheme": "ws",
        "advertised_host": "0.0.0.0",
        "advertised_port": 9985
    },
    "database": {
        "backend": "localmongodb",
        "connection_timeout": 5000,
        "max_tries": 3,
        "ssl": false,
        "ca_cert": null,
        "certfile": null,
        "keyfile": null,
        "keyfile_passphrase": null,
        "crlfile": null,
        "host": "mongodb",
        "port": 27017,
        "name": "bigchain",
        "replicaset": "bigchain-rs",
        "login": null,
        "password": null
    },
    "keypair": {
        "public": null,
        "private": null
    },
    "keyring": [],
    "backlog_reassign_delay": 120,
    "log": {
        "file": "/root/bigchaindb.log",
        "error_file": "/root/bigchaindb-errors.log",
        "level_console": "info",
        "level_logfile": "info",
        "datefmt_console": "%Y-%m-%d %H:%M:%S",
        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",
        "fmt_console": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "fmt_logfile": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "granular_levels": {},
        "port": 9020
    },
    "CONFIGURED": true
}

tendermint config

proxy_app = "tcp://127.0.0.1:46658"
moniker = "bdb-node2"
fast_sync = true
db_backend = "leveldb"
db_path = "data"
log_level = "main:info,state:info,*:error"
genesis_file = "config/genesis.json"
priv_validator_file = "config/priv_validator.json"
node_key_file = "config/node_key.json"
abci = "socket"
prof_laddr = ""
filter_peers = false
[rpc]
laddr = "tcp://0.0.0.0:46657"
grpc_laddr = ""
unsafe = false
[p2p]
laddr = "tcp://0.0.0.0:46656"
seeds = ""
persistent_peers = ""
addr_book_file = "config/addrbook.json"
addr_book_strict = true
flush_throttle_timeout = 100
max_num_peers = 50
max_packet_msg_payload_size = 1024
send_rate = 512000
recv_rate = 512000
pex = true
seed_mode = false
auth_enc = true
private_peer_ids = ""
[mempool]
recheck = true
recheck_empty = true
broadcast = true
wal_dir = "data/mempool.wal"
[consensus]
wal_file = "data/cs.wal/wal"
timeout_propose = 3000
timeout_propose_delta = 500
timeout_prevote = 1000
timeout_prevote_delta = 500
timeout_precommit = 1000
timeout_precommit_delta = 500
timeout_commit = 1000
skip_timeout_commit = false
max_block_size_txs = 10000
max_block_size_bytes = 1
create_empty_blocks = true
create_empty_blocks_interval = 0
peer_gossip_sleep_duration = 100
peer_query_maj23_sleep_duration = 2000
[tx_index]
indexer = "kv"
index_tags = ""
index_all_tags = false

Is there anything wrong with these configuration files? Please look for it. Thank you!

[kansi]

@xuhuigithub If possible, could you please share the test code? Or any hints on what kind of transactions you were trying to POST?

[xuhuigithub]

@kansi This is my code.

from bigchaindb_driver import BigchainDB
from bigchaindb_driver.crypto import generate_keypair
from datetime import datetime
import threading
# bdb_root_url = 'http://10.12.6.6:9984'

bdb = BigchainDB('http://192.168.61.201:9984')

class ThreadClass(threading.Thread):
    def run(self):

        bicycle = {
            'data': {'pripid': '88273712778381','reg_no':'88273712778381','origin_entname':'xxxxxxxxxxx'},
        }
        utctp = datetime.now().utcnow().timestamp()
        metadata = {date":utctp}

        for i in range(100):
            now1 = datetime.now().timestamp()
            alice, bob = generate_keypair(), generate_keypair()
            prepared_creation_tx = bdb.transactions.prepare(
                operation='CREATE',
                signers=alice.public_key,
                asset=bicycle,
                metadata=metadata,
            )

            fulfilled_creation_tx = bdb.transactions.fulfill(prepared_creation_tx, private_keys=alice.private_key)

            sent_creation_tx = bdb.transactions.send(fulfilled_creation_tx)

            now2 = datetime.now().timestamp()
            print("%s inster bigChainDB time: %s" % (self.getName(), now2-now1))

now1 = datetime.now().timestamp()
for i in range(100):
    t = ThreadClass()
    t.start()
now2 = datetime.now().timestamp()

[kansi]

@xuhuigithub We have added a fix (#2281) for the same in the master branch. So, you can fetch the latest master and run the test.

[ttmc]

PR #2281 is now merged but we'd like to do some stress tests on a live network before we close this issue.

[xuhuigithub]

Thank you @ttmc @kansi .

[ttmc]

We published the results of our stress tests / performance tests about a week ago. See:

• BEP-23, the technical report
• the blog post