Service-fabric: Each instance of stateful service (.net core) on Linux cluster takes at least 250MB of temp storage.

[lucasmaj]

This is huge limitation on Linux (Ubuntu) SF cluster. Windows SF does not have that issue.

Just trying to follow great recommendations from https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-concepts-scalability

Please help!

[lucasmaj]

Looks like *.stlog in /mnt/sfroot/.../work files are each 268MB. Can the size by configured to be smaller like 2MB? What are the implications?

[tyms]

Thanks for raising this issue!

Currently, this per-replica file (the "staging log") is not configurable in size, though we can make it so in a future version. Just for additional context, on Linux, the one-per-replica 256MB staging logs replace the one-per-machine shared log that exists in Windows clusters, which is by default 8GB.

There would be two main implications of reducing the size of the staging log too much:

1. Increased chance of encountering write throttling. If the incoming write rate is bursty, or the write size is large compared to the size of the staging log, it may approach a log-full state more quickly. If this is the case, write throttling may kick in to avoid the log becoming full. This will result in backpressure on the service (via failed reliable collections write operations).
2. Potentially increased disk usage in the replica's dedicated log. Writes to the staging log are kept low-latency by issuing writes to disk immediately (regardless of how much of the written blocks contains actual data vs "wasted" space), however writes to the dedicated log are delayed to allow them to be efficiently packed and waste as little disk as possible. Whenever the shared log becomes close to full, it accelerates the writes to the dedicated log so that it can free up space, which means the packing won't be as efficient.

Whether you would encounter each of these cases would depend on the particulars of your service: its write rate, write size, etc. You would need to do testing with accurate scale and load in order to know how small you can make the staging logs without causing a problem.

To help us prioritize, could you please give some additional info about your scenario? It would be helpful to know what density you are trying to achieve (replicas per machine), your expected write rate, and how much disk space you have available per machine. It would also be useful to know if you would need to be able to resize the staging logs in an existing cluster, or if it would be sufficient to be able to configure it for new staging logs.

-Tyler

[lucasmaj]

@tyms Thank you so much for responding!

I think my scenario would fall into the category mentioned in the linked article. I wanted to have a stateful service per customer/shopping cart/IoT device. Thousands of these services per D2v2, as many as possible to maximize the utilization of each node. The nature of the state would be transient, extremely low persisted amounts but possibly bursty in volume and in mutations at times. Really just need a reliable local queue for each service so that external queues (ServiceBus) won't be necessary. Maybe 2 or 3 replicas. I was _not_ going to create a database out of reliable services.

I have a suspicion that 256MB per replica was picked to favor large but few reliable services, while my scenario is of many-but-tiny nature which is currently better served by SF on Windows approach.

From a SF consumer perspective, I don't think I should need to worry about different approaches for replica implementations on Windows vs Linux. I would argue to keep parity and offer same choices (shared log, dedicated static log but configurable size) on Windows and Linux. Or choices can't be offered here due to OS limitations? I also should be able to have metrics/counters etc to judge (during evaluation and later as load could change) if another approach would better suit my load, or to realize some bottleneck has been reached.

Right now I wish the Linux implementation had same behavioral characteristics as Windows implementation (or at least not such glaring differences). Performance would be the differentiator we could be picking upon.

Maybe that's too much to ask but thanks anyway.

-Lucas

[juho-hanhimaki]

This issue also raises my interest because we're considering using Linux clusters in future. In our IoT scenario we have a few reliable services that are instantiated per customer to isolate each customer. The services can potentially be small with only one partition or very large with hundreds of partitions.

It's hard to predict the density per node, but if one assumes that there will be many customers but the actual load from the customers will be light (small IoT deployments) the disk space could quite easily become the limiting factor for density per node.

[tyms]

Thanks for the additional info on your scenarios, it's useful! These are prime SF/RC scenarios, but unfortunately are likely better served by Windows clusters at the moment.

I agree wholeheartedly that the Linux vs. Windows behavioral differences are a burden, and that this disk usage is a potential sticking point for trying to achieve high density with stateful services on Linux. Behavioral and performance parity continue to be important goals for us, however we made some tradeoffs in order to enable Linux support (much) sooner. For context, on Windows, the shared log is implemented as a Windows kernel driver, making it one of the least portable components in Service Fabric, and requiring considerable work to replace in-kind. We are currently considering several different options for unifying our Linux and Windows approaches in this area.

Thanks again (to both of you) - feedback like this is really useful, as it better helps us direct our development resources to address customer need. I can't promise any specific solutions or timeframes, but this is an area that we are actively looking at improving.

-Tyler

[erikadoyle]

Per the above merged PR, the docs have been updated to include this information. #sign-off