Steem: Subscription API

[jnordberg]

Hi, I'm trying to use the subscription api over the websocket rpc.

set_block_applied_callback works and provides a stream of

{ previous: '00c4973217aed660f336c465337e3718c64e0f36',
      timestamp: '2017-06-16T23:14:45',
      witness: 'krnel',
      transaction_merkle_root: '5eefe40a36b2ba64ef28c3b05b36e9c53dfbd898',
      extensions: [],
      witness_signature: '205546f70accf4460402edd6e29c2a358dd48c015bb894ea4ae985ff8c574593e958af49a304a23105b2d2417de9107dc2cc92e805ed622aafa8de752468d60d6f' }

But the other two calls set_subscribe_callback and set_pending_transaction_callback are accepted but no events are sent. Is this a bug or does the node have to be configured somehow before events are sent?

[theoreticalbts]

The set_subscribe_callback and set_pending_transaction_callback API calls are leftovers from the BitShares UI infrastructure. They don't actually do anything in Steem. We'll get rid of them in the next release.

[jnordberg]

Okay, thanks for the clarification.

Any plans to add more subscription calls specific to Steem? It would be great to be able to subscribe to events for a specific user. And a block callback that sends the entire block would be useful as well (right now many applications I've seen just spam get_dynamic_global_properties to see if a new block has arrived)

[theoreticalbts]

The dev team isn't going to be adding subscription semantics to steemd, and existing subscription calls are deprecated. The main reason is that load balancing is an important part of making steemit.com function, and load balancing is much easier to implement if you don't have long-lived or stateful connections.

Another technological issue with subscriptions in general is, when the blockchain switches forks, some notifications that have already been sent may have been on the minority forks. So any notification protocol must include a way for the sender to tell clients that they need to revert subscription data they've already received, and clients would have to properly respond to that condition.

[jnordberg]

You're not planning on deprecating web sockets altogether? As a third party developer the web socket rpc is the killer feature of steem.

Another technological issue with subscriptions in general is, when the blockchain switches forks, some notifications that have already been sent may have been on the minority forks.

Doesn't that hold true regardless if the client is pulling the data or having it pushed via subscription? For example if I wanted to implement a steem stats dashboard (something like realtimebitcoin.info, shameless plug 😄) and I'm polling get_dynamic_global_properties to get the current block height and requesting blocks as I see the head block# increment. Would I have to worry that the data I've gotten might be incorrect, and how would I know?

[sneak]

The state only changes every 3 seconds; polling even at 1r/s is not much of an issue. I can't say I agree with you about websockets being a killer feature; it's quite a pain to run in production. We're moving toward an all-json-rpc-over-normal-http (v1/v2).

There is a vague idea of adding some sort of new-block-notification system to steemd, so it can send a message to a queue or external program or somesuch that the head block has changed and that there is new state. That would pretty effectively resolve this use case, I think?

Alternately, just have a sidecar daemon that polls it a few times a second and fires off a message to your queue or fanout pubsub system letting downstream things know that there is a new block to ingest.

We'll be releasing all of our tooling around this. For now, there's steemit/sbds (on GitHub) that may scratch some of that itch, though we intend to improve the whole stack to reduce latency.

[jnordberg]

Just to clarify why I think web sockets are a killer feature:

• They are exempt from the same origin policy which makes the API easily accessible even from a "web native" app hosted on e.g. IPFS.
• Low latency, even if the lb uses http2 it still needs to negotiate a new connection for each request.
• Low overhead, the HTTP headers are often larger than the payload.
• Lets you have both push and pull APIs.
• Lets clients keep track of their connection state.
• Interoperability with WebRTC.

In my experience load balancing web sockets are pretty manageable even with a round-robin setup if the timeouts are low and the server properly implements the ping/pong frames.

Requiring 3rd party developers to run their own full node is a huge barrier for adoption.

[marvin-we]

The cancel_all_subscriptions method has been removed too - If a callback has been registered using the set_block_applied_callback method, there is now no chance to cancel this, right? Are there any plans to bring this feature back or is it still possible and I missed something?