Steem: gRPC or Swagger API Provider plugin for generated api clients

[faddat]

Basically, I'm writing to ask if it's possible.

If it is possible, and anytone will give me pseudocode and/or advice that can get me started, then I'm happy to do the work of implementing this. We as a community / economic entity need this badly. Both gRPC and swagger are designed to take the time burden off of development teams and to make well-documented interfaces something that's both standard and easy.

Off the top of my head, 100% of devs I know working on steem projects think the API is a weak point, and have complained to me about it at length, just as I have complained to them about it as well. I'm very happy to help change that, provided that anyone can:

1) Let me know if I am on the right track
2) Let me know if this can be implemented through a plugin
3) Tell me about the dev env where steem what tools and processes are in use, so that I might mirror them on my own system and enjoy the benefits of sharing the procedural idioms.

To clarify: I don't need pseudocode for the grpc / swagger part, I've checked out their docs and despite my relative lack of experience in c++, I'm pretty sure that I can handle this. I'd just need some pointers on how to efficiently interface with steemd via a plugin, and if I'm to undertake this then I definitely want to be able to configure my dev setup the same way as core team members so that we have shared context.

Another point that is probably worthy of mention here is that gRPC uses protocol buffers, meaning that we could reasonably expect to enjoy a speed-boost in communication between the blockchain and its clients by implementing gRPC.

[abitmore]

I have little to add to this discussion, it seems https://github.com/steemit/steem/issues/347 is related?

[faddat]

It might as well be #347 , sorry about that, I think I'd forgotten it existed. I'll close that, as I feel this one better describes the situation.

[theoreticalbts]

I'm not familiar with gRPC or Swagger.

There are two different things you might want to do:

• (a) If you think the layout of methods is poorly designed, illogical, or don't expose needed functionality, and you want to add new methods and make them available over the existing HTTP and Websocket RPC's, you can already do so in a way that's well-supported and used by many of our own plugins.
• (b) If you want to make the existing methods available over an alternate wire protocol, that's more tricky.

For the rest of this comment, I'll be talking about option (b).

The simplest way to do this would be a proxy which translates requests and responses -- a script in Go or Python or something that talks gRPC or Swagger to downstream clients, and Websocket to upstream steemd. It lives out-of-process and would require no patches to steemd unless bugs are found -- as far as we're concerned, you'd just be another client.

If you want to get fancy, you can add hooks / configuration making it simpler to add caching to the proxy, or use a single proxy as the frontend to more than one backend node (the exact policies for caching and multi-backend would be highly specific to a particular deployment of a particular application, and it sounds like you want more of a general-purpose library, so the best you can do is add support for the most common use cases).

The more complicated way is to implement functions natively in C++. The proper place to put these modifications is in application which is responsible for managing the plugins and the overall networking configuration. That being said, plugins are arbitrary C++ code which can hook into the node startup, and it would probably be technically possible to develop this as a plugin -- which might also simplify keeping your work coordinated with core development by providing a better-defined boundary between the two projects.

[theoreticalbts]

If you do this in C++, you'd have to worry a lot about multitasking. Except for the p2p network code, steemd is mostly single-threaded. It uses a multitasking system from @bytemaster's FC library. Presumably the new RPC would want to use a third-party library for opening sockets and reading bytes on the wire, and that library would have its own opinions on how it multitasks.

I don't know if FC's merely a wrapper around Boost, or if there's more going on -- I do know that there's e.g. some fairly complex and low-level scheduling logic here for example. The moral of the story is that:

• The networking I/O of your code would need to be non-blocking in a way that allows the core code to run
• Any API calls dispatched from your code need to be scheduled to run in the main thread

Basically you'd probably want to run your networking and (de)serialization code in a separate thread, then schedule API calls in the FC main thread. @bytemaster would probably be a better person to ask about further details.

[theoreticalbts]

So a quick look at the docs for gRPC tells me more about the core problem. I think the central requirement is this:

• You want to automatically generate bindings for our API in the client language.

I'm working on a Python API framework called steemwatch which is available on my personal Github, which allows a client to write code like this:

node = ApiNode(io_loop=io_loop, websocket_url="ws://127.0.0.1:8790")
node.start()
await node.wait_for_connection()
db_api = node.get_api("database_api")
block = await db_api.get_block(1234)

Doing this in a statically typed language would require generating a class for db_api which contains method definitions for all methods in database_api, which is the problem that gRPC seems designed to solve. The on-the-wire format (Protocol Buffers) doesn't really affect the client's dev experience, the main thing that affects their experience is being able to write code with the above.

In steemwatch I avoid this problem because Python is dynamic enough that with judicious overriding of __getattr__() and __call__() we can simply create an object which allows any method name to be called with any number/type of arguments and translate it into an on-the-wire call.

I should really document steemwatch and release it properly.

[faddat]

Yes, you got it: I want any user of any programming language to be able to easily add STEEM blockchain support to their applications. I think that's the single most important thing we could do for the financial health of the community as a whole.

I'm a little confused: steemwatch would work for golang? Or generate code for clients in any language?

[theoreticalbts]

No, steemwatch is Python only. It uses Python specific features to let you call any method without a code generator.

[faddat]

Dang, I thought that might be the case.

Well, I'll keep looking for where/how to bring in gRPC. The whole push on my part is really about:

• Increasing the overall size of the app ecosystem that uses STEEM
• Reducing duplicated effort
• Making STEEM the #1 place to try out innovative uses of blockchain tech generally speaking

[sneak]

gRPC is not entirely dissimilar from the current websocket protocol, except that gRPC uses HTTP/2 framing instead. As for Swagger, it's a description language for APIs, and isn't really applicable to the current websocket implementation AFAICT. (integrating the existing gRPC code in the current app is likely infeasible, for reasons described above.)

I think the immediate way forward in the next few months for the steem RPC is going to be HTTP/1, not HTTP/2 or websockets, but that's still up in the air/being discussed and isn't final. HTTP is, of course, widely supported by client libraries. :)

Until that's settled (3-4 weeks away, at a minimum), there won't be any significant changes on the RPC front. The websocket API serves the purpose for now.

[testzcrypto]

Steem Swagger API available here: https://esteemapp.github.io/steemapi/

GitHub repo: https://github.com/eSteemApp/steemapi

[faddat]

@testzcrypto,

I'd really like to know how you built the swagger API. If you've any time, please drop a line at [email protected]

[feruzm]

Hey @faddat,

as you can see from source code swagger.json file has some methods listed so you can just clone and add new/missing api there. It is using steemjs and you can also make so it will be none dependent to any server but works locally, in that case you will have to wrap Steem API/functions to Rest formatted functions so swagger can make use of it. Hope it helps...