Plumber: Stream large data

[dipterix]

Thanks for designing such a wonderful package. I plan to use it in my project. However, there are some bottlenecks.

I need to send 400MB data (generated by R) from data server to logic server via intranet within 5 seconds. However, serializing data to JSON is too slow.

Is there any native support to send binary content type (application/octet-stream)? Right now I'm planning to fast serialize the data into binary files and change the response header to receive the file and unserialize.

Best.

[dipterix]

To be clear. I want to use plumber as "data servers" to collect and clean data. This process is usually fast but requires large RAM and secured connections. I also have many logic servers running with algorithms (slow, blocking) and one shiny server to monitor these logic servers.

                             +------ (module 1)--+
(shiny server, non-blocking)-+------ (module 2)--+-(data server / database, large dataset)
                             |------ (module 3)--|
                             | ...               |
                             +------ (module n)--+

The fastest serialization method I found via the internet is fst. However, that package must work with a file system instead of connections (I guess it has its own way to deal with multi-thread writing).

It'll be quite strong if plumber can support binary streaming and serialization internally via memory.

[schloerke]

There are lots of things going on in this issue. Please let me know if I've missed anything.

Best,
Barret

---

1. Allow for Content type: application/octet-stream. Please see the example in section 4.2.1. Bypassing Serialization. Link: https://www.rplumber.io/docs/rendering-and-output.html#serializers

2. The end goal is for you to execute plumber as the data server. Given your diagram, I'm guessing you'd like for this server to handle multiple, large, blocking processes at the same.

3. Unfortunately, this is not natively possible. https://www.rplumber.io/docs/runtime.html#performance-request-processing talks about plumber being single threaded and how to address some speed concerns.

   • If you can allow for delayed processing of the data, but favor handling requests immediately, look into the later package. I could imagine you receiving a request, setting up serialization / forwarding in a later::later command, and immediately returning a "received receipt" to the caller. When the R process is free, the data would be processed in a background thread. This would allow for receiving many calls at once and hopefully the single threaded processor catches up on the later queue eventually.

4. Speed in intranet. If you're already going to have to write the data to disk for serialization AND you have access to the same file system, there is nothing wrong with forwarding only the file path of where you saved it. This would reduce the information sent to the logic server to essentially nothing (compared to 400mb). The logic server could then perform the same multithreaded read that plumber was going to have to do.

[shrektan]

@dipterix I believe you are only using plumber to provide an API to another R user (maybe yourself), right? You may use base::serialize() to do that. I've tested an R data.table with size over 700MB and it costs only 5 seconds on my computer. Moreover, it's flawless (it means you can use base::unserialize() to restore the R object as if it's read from an RDS file)!

You may find more info on my just-add PR #345 . Hope it solves your problem.

[dipterix]

@shrektan You are right. The whole problem goes to fast serialization and the reason why I choose R as server language is because the client language is also R so I can use non-standard serialization method to speed up this process.

As for the non-blocking part, it's more a load balancing issue that can be resolved by nginx. I can run multiple instances and pool them via nginx.

One problem could be caching, or state. Let's say app1 has already processed and cached data1 in file system or in memory while app2 hasn't. Now if client asks for data1, how can I redirect this request to app1 without inquiry into my file system?

A potential solution to this problem is to have a router or another instance (app0) running on different process to keep track of all the app status. Once app1 finished caching, it could tell app0 to update its status. Clients always talk to app0 first. Next time when some client asks for data1, app0 will forward this task to either app1 or app2 (is this possible for plumber?). This also requires app0 to know the status of app1. If app1 is running (blocked), we might want to use app2.

It'll be great if plumber has any classes implementing app0. It's easier to implement, because it only needs to run locally, and it only needs two public methods: forward(instance_id) (to forward connection to other instances) and update_status(instance_id, status) (update instance status). And developers can decide which instances to forward to depending on the status.

[dipterix]

To correct, app0 is not running locally. It's running, listening to a public port.

[shrektan]

@dipterix Sorry, I'm confused. Since you previously mentioned the fst package for serialization, I thought your main problem is that the serialization to JSON is slow.

So does using base::serialize() fit your case or not?

Are you asking a second question here, which is to implement a caching data server? Your example states the data is about 400 MB, if the clients number is small and the data is usually this size, I think caching is not a necessary - because it only takes a few seconds to serialize the data and compared to the network speed, it's really small... So caching may only complicate things.

However, if such complicate caching is really required, it may mean that you need a SFTP server or something rather than plumber. Implementing such in plumber sounds complicate and unstable to me, since most of the time you are only hosting static files while plumber is meant to be hosting dynamic functions...

[shrektan]

One idea comes to my mind is using plumber to generate the data to a database and the API returns the database's ip, port, and table names. The client get the data by fetching data directly from the database. The database can handle multiple connections and large volume of data transferring by design. It may fit your needs.

[dipterix]

You are right I've already solved the first question. The problem is my data is generated dynamically with 100-500 chunks, each chunk is 400MB~800MB. I guess I might need a database :/

[dipterix]

ah, didn't see your last comment. You are right. Thanks for the help 👍