Tendermint: TMSP flush causes socket to close

[uhrobots]

I've been implementing TMSP in c++ and after I send the initial commit response, nothing happens. I then try to send a flush response, I get the core application to process the previous commit but it then closes the existing socket connection...
Is this intended behavior? If so, it seems to be rather inefficient

[srmo]

@AaronShea did you try to run the node with debug level to get an idea what's going on? Maybe you misaligned the varint or sending the wrong return code?
log_level = "debug" in config.toml

[srmo]

PS: remember that you need to listen for several connection attemps. The first connection will be closed anyway after the initial handshake and per protocol definition, there are supposed to be 2 connections anyway on the same socket (anyone care to clarify? I'm unable to find the docs about this).

[srmo]

_cough_ Sorry for this spam, found it:

Tendermint Core creates two TMSP connections to the application; one for the validation of transactions when broadcasting in the mempool, and another for the consensus engine to run block proposals.
see http://tendermint.com/blog/tendermint-socket-protocol/

[ebuchman]

how are you testing? using tmsp-cli? can we see some log output?

[uhrobots]

I've figured it out, it was not accepting multiple connections. Thanks guys!
It should be noted in a formal spec do somewhere though, getting started with TMSP has been a bit rough.

[ebuchman]

thanks Aaron. Would you mind making a short list of other hangups or things that stumped you? It's hard for us to know having been steeped in the stuff for months now. Would defntly like to make this as easy as possible for all!

[mdyring]

@AaronShea: you can find an example of the "counter" app implemented in C++ here: https://github.com/mdyring/cpp-tmsp

[uhrobots]

@ebuchman I'm unsure of the order of responses to send back.
For example, what do I do in this situation?

I send an HTTP RPC request to broadcast a tx. My application sees the checktx message, and I respond OK, and flush. The core application seems to no do anything after that. Is there a step missing here?

General pseudo code would be really helpful in handling the different requests coming into the proxy application.

Correction, I seem to get "[<core_types.TMResult Value> <error Value>]" in the console after I process the request. Strange.

[uhrobots]

@mdyring Your example application seems to be using the older protobuf spec used by Tendermint, I'm using the latest version.

[ebuchman]

[<core_types.TMResult Value> <error Value>] is just go printing the types. so you are sending a flush back from the app? I don't think you should be sending flush back from the app unless you received one from the core. The checktx returns successfully, but the tx never gets passed to appendtx? does the core stop making blocks?

[uhrobots]

In my code I listen for the protobuf messages coming from the core application, after responding to the first commit message the consensus tick starts up. I send an http RPC call with valid bytes and I can see my code hit the CheckTx message handler.

auto transactionBytes = msg->check_tx().tx();
// Do some validation here
res.mutable_check_tx()->set_code(types::CodeType::OK);
this->sendResponse(&res, dest); // dest being the destination socket to send the message to

After this, I get the log of the go types in the terminal, then nothing else happens. The core logs that it responded to the HTTP call, and consensus ticks along. I never see AppendTx happen at all. So I assume that it's waiting for another message?

[mdyring]

@AaronShea are you seeing two connections being established from tendermint core -> your app?
You will see CheckTx on one of them and AppendTx (and possibly also CheckTx I believe) on the second one..

[uhrobots]

Yes, I have two sockets open, and they both are checked for data

[mdyring]

@AaronShea Check to see if you are setting the type of your response: response.set_type(types::CheckTx);

[uhrobots]

set_type does not exist in the newest version of the protobuf definition.
Edit: see https://github.com/tendermint/tmsp/blob/master/types/types.proto#L9

[ebuchman]

Hm. We did update the protobuf to use oneof instead of packing all the possible fields in the Request/Response type, so set_type is no longer there. As for the connections, there are two separate ones: one passes _only_ CheckTx, the other passes _everything but_ CheckTx (ie. blocks!).

@AaronShea are you getting the correct result from the RPC call? can you paste some relevant tendermint logs?

[uhrobots]

@ebuchman I send a request: http://localhost:46657/broadcast_tx_sync?tx=%22417c42245a%22

The response is:

{"jsonrpc":"2.0","id":"","result":[96,{"code":0,"data":"417C42245A","log":"Transaction approved!"}],"error":""}

And the full log is here:

http://p.defau.lt/?XdBfsqmfqNfQEuQ_AzxgBw

I apologize for the character encoding being messed up, we're testing on Windows at the moment.

[ebuchman]

Thanks. This is actually an issue! You have fast-sync enabled, and the node never switches on the consensus reactor. Are you using latest master? How many validators?

NOTE[07-28|12:02:01] ConsensusReactor                         module=consensus fastSync=true

[uhrobots]

I am on latest master yes, it's actually currently a tendermint.exe single node + the application while I'm trying to implement the protocol correctly.
Running 2+ of the applications is something that will come at a slight difficulty, as it's a large application.
However, I did just find issue #197 where you stated that one node may require turning off fast-sync.
I've done that now, however I can still not get AppendTx to fire.

Sent request *types.Request check_tx:<tx:"A|B$Z" >
Sent request *types.Request flush:<>
Received response responseType *types.Response check_tx:<data:"A|B$Z" log:"Transaction approved!" >
Received response responseType *types.Response flush:<>
INFO[07-28|12:48:43] HTTPRestRPC                              module=rpcserver method=/broadcast_tx_sync args="[<[]uint8 Value>]" returns="[<core_types.TMResult Value> <error Value>]"

I uncommented some debugging logs in TMSP so I could see the responses and requests going in and out. (Something I think should really be put back in!) And I can see the CheckTx and Flush request going in and my responses going back. I can see the consensus engine starting up http://p.defau.lt/?wXTvLdt4q6BcGMBVKhu7Ig

[ebuchman]

Ah. It doesn't look like you're making any blocks. Does the public key in the priv_validator.json match the one in the genesis.json in your $TMROOT (defaults to ~/.tendermint) ?

[srmo]

@AaronShea fastSync is only relevant once you add another node N2 and use your N1 as seed. Experiments show, that N2 should use fastSync = false at the moment

Edit: your single node setup should not be affected at all

[srmo]

Another note, if, for some reason, you already added another validator to the genesis file, remember that block-building will only work if > 2/3 of validators are online. So for 2 Validators, both have to be running.

[mdyring]

FYI https://github.com/mdyring/cpp-tmsp is updated now with latest protobuf changes.

[ebuchman]

I wrote a brief developers guide: https://github.com/tendermint/tendermint/wiki/Application-Developers

Hopefully it is helpful. I think I will enable options for better debugging in the tmsp client as well. Perhaps it might also be nice to have some tool which takes a file containing a list of transactions and runs some tests against your app as if it were a consensus engine

[uhrobots]

Awesome stuff, thanks!
I got the whole thing to kick off and start to make blocks. but now I'm in an interesting situation.
At the startup of the core, it sends a commit message. That's all fine, then I see it sends a flush.

I never get the flush message. In fact, I never actually see it come across the network. Here's a packet capture:

After that, it just sits there doing nothing waiting for the flush response. My application never actually got it. It's odd.

[ebuchman]

Oh man. Weird. Is there any way I can try and replicate? is your server/app on github? or at least can I see more logs?

[uhrobots]

@ebuchman never mind that, I didn't understand the steaming protocol and how two messages are sent directly attached to each other (in this case the commit and flush). Everything seems to be working perfectly. Thanks for working though this with me!

[ebuchman]

Amazing! Thanks Aaron. Will close the issue now