Service-fabric: Allow singleton stateless services to have multiple instances per node bypassing placement constraint

[guibirow]

I've posted this as an Idea in UserVoice, but I think this is a bug, so I will post it here as well.
https://feedback.azure.com/forums/293901-service-fabric/suggestions/19664074-allow-singleton-services-to-have-multiple-instance

Currently, service fabric does not allow you to have multiple instances of the same service in the same node to avoid data loss. But for stateless services it is not the case, as we do not have data to loose neither replication, the orchestrator should know that if my stateless service has more instances than the node count, it should be smart enough to split these instances around the nodes without warning about it.

Unhealthy event: SourceId='System.PLB', Property='ServiceReplicaUnplacedHealth_Secondary_ba654c6a-3e83-4560-9ea1-9fadbe7b115d', HealthState='Warning', ConsiderWarningAsError=false. The Load Balancer was unable to find a placement for one or more of the Service's Replicas: Secondary replica could not be placed due to the following constraints and properties: TargetReplicaSetSize: 9 Placement Constraint: N/A Parent Service: N/A Constraint Elimination Sequence: Existing Secondary Replicas eliminated 3 possible node(s) for placement -- 0/3 node(s) remain.

[alltej]

I am getting this same issue. I added a stateful actor. It runs fine locally but when deploy to a remote environment it get's the exception. The SF app is fine. It's only the SF actor service is getting the error.

[MrMint]

@alltej while you are getting the same error, this issue is specifically talking about stateless services and the inability to scale the instance count past the node count due to the placement constraint.

Not having this feels very odd for "worker" stateless services that you do not need to communicate with. We are working around this by partitioning our stateless service to get the instance count up, but it is very clear that this workaround is a hack and generally feels bad.

Is there a better workaround?

[guibirow]

I didn't find a better approach, the other option would be disable the placement constraints ReplicaExclusionStatic and ReplicaExclusionDynamic on your cluster, but it would be worse as it will affect the other services.

The SF could handle that scenario better or have the option to ignore server constraints per service.
The partition is the better approach by now.

[alphagridio]

Any updates on this issue ? I'm facing the same issue now. I have 5 nodes cluster , running 5 stateless services. I want to run each service a 10 instances but it doesn't allow me and throws the same exception. This is putting back our migration from cloud services to SF. I have deployed 4 core machine nodes, they have lot of resources unused and i want use them by running more service instances than my node. Any update would really be highly appreciated.

[nosu]

@techiearun We faced this issue as well, and it seems there is no updates for now. We are working around this by partitioning the stateless service as @MrMint mentioned, and it has worked as expected without any errors.

[masnider]

Have you folks tried just creating more services? New-ServiceFabricService a bunch of times?

We can consider this enhancement, but it's unlikely to happen as it breaks a whole bunch of assumed invariants in SF's placement and constraint logic about _never_ allowing two instances from the same partition on the same node.

[guibirow]

Creating more services would be more cumbersome than partitioning the service, that is the current way we workaround this issue.

If you think that will be too difficult to bypass this constraint for stateless service when the replica count is greater than node count, another option would be allow us to ignore some constraints on application manifest or service creation, this way we would have more control which constraints we can apply or ignore for a specific service.

[MrMint]

Our goal was to use services to process messages off of service bus. We would have liked to be able to control the amount of concurrency/throughput by dynamically tweaking the instance count for the service. This would keep the code for each service simple, and leverage service fabric to meet our concurrency/throughput needs. We ended up ditching partitioning the stateless service as the inherit limitations meant we couldn't control the count of active worker services simply/granular enough. Instead we introduced the concurrency complexity into the service itself, which is not ideal but works for now.

@masnider while we certainly could provision new services dynamically, this would require more effort as we would have to manage creating/deleting instances dynamically, rather than just setting an instance count. While this is not unreasonable, it seems odd that stateless services have these limitations and require us to orchestrate it.

[nareshkhatri81]

I am also facing the same issue. could someone help me with workaround ?

[ghost]

Multiple instances of a single partition cannot be placed on a node. This is a hard constraint. If multiple instances of the same service are desired, the current approach is to have multiple partitions for the service.

[nareshkhatri81]

I have stateless service where we would like to place multiple instance of same service on same node. We have 2 VMs and we have configure service fabric as 3 nodes. 2 nodes on 1VM and 1 node on second VM.

I am seeing 2 instance are getting deployed 1 in 1VM and 2nd in 2VM. I wanted to deploy 10 instance on those 2 VMs.

[ghost]

I'm assuming that your service has a single partition. If you want 10 instances per VM, then you should partition the service to have 10 partitions.

[nareshkhatri81]

Thanks for info.yes we have single partition stateless service. Without petitioning, is it possible to run multiple instance of same service and service fabric runtime distribute load randomly or to service which has less load. We want one or two service which are heavily used to have more instances so they can handle more requests.

[ghost]

Without partitioning it is not possible to run multiple instances of the same service on a single node.

I would also like to understand why do you want to run multiple instances on a single node? If the service is properly written, two service instance cannot give you a better resource utilization and throughput than a single instance. And if the concern is fault tolerance, then placing multiple instances on a single node doesn't help with that, as if the node fails then all the instances need to be failed over.

[MrMint]

@Mansoor-Mohsin My previous comment outlines why we would have liked multiple instances on the same node. It would have been the least complex way to dynamically scale throughput for our worker services (services processing messages off of service bus).

[nareshkhatri81]

There is limitation on number of concurrent request one instance of micro service(webapi) cam handle. After that request starts to go in waiting state. We want to avoid that by having multiple instance of stateless in same instance and SF randomly do load balancing between these instances.

[ghost]

We've noted down the feature request. We'll consider it for the upcoming releases.

[nareshkhatri81]

is there any update when this feature will be implemented ? i still don't any milestone yet attached to it. its part of backlog still.

[masnider]

No timeline at this time. We would have to create a new type of service that wasn't bound to our normal service placement rules, or introduce a way to mark such a service with that particular placement rule. It's still under discussion, but there's no timeframe to share about how or when this would show up.

[johnib]

Quoting @Mansoor-Mohsin:

Without partitioning it is not possible to run multiple instances of the same service on a single node.

I would also like to understand why do you want to run multiple instances on a single node? If the service is properly written, two service instance cannot give you a better resource utilization and throughput than a single instance. And if the concern is fault tolerance, then placing multiple instances on a single node doesn't help with that, as if the node fails then all the instances need to be failed over.

Here's a valid scenario, which I've hoped that SF would solve - disappointed to find it's not going to:

1. I have an Azure EventHub with 100 partitions.

2. The amount of partitions is constant, and all of them have more or less the same node - most of the time.

3. The "Partition Manager" that we use have its overhead, and its management cannot handle and cover well all corner cases - causing, on rare (yet critic) cases, partition ownership being handed from one instance to another for some period (5-10 minutes).

4. My thought was to run 100 instances of my service on the SF cluster - each instance would have a single EventHub partition for all of its life, and partition ownership never changes.

5. On cases when the load on the EventHub (in whole) is low - my cluster would be running 10-20 VMs - having 5-10 service instances running on each (respectively).

6. On cases when the load on the EventHub (in whole) is high - it would be running on 100 VMs, each running a single service instance.

7. On cases when the load on the EventHub is not spread evenly, and the load on one partition (partition-x) is higher than the others' - the specific instance (instance-x) would consume a lot more CPU than the other instances, and SF would know (or I would tell it) to put instance-x in a dedicated VM to allow it the maximum resource utilization possible.

[ghost]

If my understanding is correct, you are have 100 stateless instances running on the SF cluster. When the load is low, the 100 instances run on 10-20 VMs, and when the load is high these 100 instances run on 100 VMs. If the load is high for some of the instances, they run on dedicated VMs; and the lightly loaded instances are packed on the remaining VMs.

How are you modeling the 100 stateless instances that run on the SF cluster? If you model the stateless instances as 100 SF partitions with each partition containing a single instances, then this should be able to handle all the above requirements.

To summarize, there are two models:

1. 1 service, 1 partition containing 100 instances
2. 1 service, 100 partitions with each partition containg a single instance

With option 2, you should be able to scale anywhere from 1 to 100 VMs.

[sandyplace]

We are using kafka for our service partitioning, we essentially are using SF as a way of deploying and managing our stateless worker service, and would like SF to manage those instances. These services are fairly lightweight and would like to better utilize our computing resources by running multiples of them on each machine in order to increase throughput. Now we will have to figure out our own Task management at the service layer to scale in this scenario.

[masnider]

Ghost is @mansoormohsin, had a bit of an account kerfuffle.

[guibirow]

@ghost, @masnider: looks like there is a misunderstanding here with the issue I have opened a while ago.
When I first raised the issue, it was clear that we could reach a similar result with partitioning, I didn't open it to find a way to do it, but to make it better, because the partitioning is a hack way of doing it and has it's limitations.

if you guys need a example to understand the problem, I will try to show the limitation I had when I created the issue:

I had to design a stateless service listening for messages in a queue, I would pool that queue read one message and process it, once completed, I would fetch another one and keep the cycle. Looks very simple and straight forward. Once our system start receiving more load, the number of messages in the queue increases and will stay on hold for longer time, so a queue monitor notice the message increase and scale up the number of instances, without interrupting the current running services. This would be a very happy path.
The problem start when the number instances reach the number of nodes, if we didn't have this limit the number of instances could keep growing until we exhaust our servers.

Then, we have to find other approaches:

• Partitioning: with a dynamic scale, I would have to map the partitions to a very large number and keep them running, wasting server resources when load is low, otherwise I would have to remove the services to repartition them and start again every time the load increase\decrease, so it would first increase the interruption and also the complexity to repartition the services.
• Each node handle multiple messages in parallel: This would be nice, but also increase the complexity, as I would have to control the number of services running, the number of tasks each one is processing and keep them in sync to balance the load evenly.
• Creating multiple Named Services from same service type: Would also increase the maintenance and complexity, in scenarios where we have a 5 node, and need 100 instances running, we would need to initiate 20 named service with 5 instances each, so is not manageable.

We could find many approaches, but it is just a work around the main problem, that should be handled with a simple scale up & down of a service without a single line of code being changed.

I don't know the internals of the Placement Constraints, but I would see this feature like a flag we set in the service(named instance) to ignore the cluster hard constraints on deployment or have it as optional constraints where we could define overrides.

[usamaazhar]

@guibirow very well explained and clears alots of misconceptions, currently Iam also designing an SF app similar to your problem domain and looks like I would also have to do a similar workaround, I have a table storage with bulk data and want the services (100 of them) to process this data and write the result to a queue.

[OlegKarasik]

Hi @guibirow, @usamaazhar

Each node handle multiple messages in parallel: This would be nice, but also increase the complexity, as I would have to control the number of services running, the number of tasks each one is processing and keep them in sync to balance the load evenly.

Can you guys elaborate a bit more about this point and provide a bit more details about what kind of processing your service does with the items from queue?

The problem with multiple service instance on the same node and pure CPU workloads is that eventually you would create so many instances on each node ( > than available logical processors on node) than each new instance would slow down the processing because there would be too many threads that require pure CPU power.

[MrMint]

@OlegKarasik In our case the stateless service processing messages is I/O bound due to calling out to other services/databases.

No matter the workload (CPU, I/O or otherwise), capacity planning and appropriately scaling for that capacity still needs to happen. The issue today is we are unable to utilize the orchestration mechanics of service fabric to scale up, and instead have to hand roll our own solutions.

[guibirow]

The problem with multiple service instance on the same node and pure CPU workloads is that eventually you would create so many instances on each node ( > than available logical processors on node) than each new instance would slow down the processing because there would be too many threads that require pure CPU power.

This is the issue with handling multiple items in parallel per service instance, you would have to be aware about the load on each instance so that you don't process too much items and the other services too little, you would also have to worry about Memory, CPU, IO. This makes the service more complex, and they should only worry about processing a message.
One of the many examples I can give is: 10 services, each service with capacity for processing 100 messages in batches, if you have 1000 messages to process to load will be even, but when the number of messages get's below 1000, for example 400, 4 services would get a batch of 100 and the rest will wait for messages. There are many ways to handle this, but wouldn't be simpler if my service should not worry about the number of messages?

This is where service fabric orchestration comes in, the auto-scale, a new feature to scale the service based on metrics has been added to SF since I opened this issue, if my services are consuming too much memory, another instance would be initiated, but we still have this limitation, currently is not possible to let our services scale more than the node count. Now with the auto-scale of partitions, made the process a bit easier, but I still hope to see a clean solution where I don't have to worry about partitions as well.

At the time, my solution had many queue processing logic, from Heavy CPU to IO, but the issue is not about the load type, is the flexibility.

[OlegKarasik]

@guibirow, @MrMint

As far as I understand from the ideology stand point here are a few concepts used by Service Fabric:
_(All of the bellow are based on my personal understanding of the Service Fabric documentation)_

1. Each service instance can utilize all resources.
   _I think so because in the documentation for Service Fabric there is not a word about Cluster Manager changing instance count of the service. The only exception is when the stateless service defines the InstanceCount to -1 and we have new nodes added._
2. Reporting metrics allows Cluster Manager to effectively organize service instances across the nodes to effectively utilize available resources.
   _If we assume that 1 is true then this behavior makes sense._
3. In case when you need to have a more sophisticated scaling mechanism there are huge amount of API that allows you to dynamically change instance count, service object count and even application object count etc.
   _The idea here is that each of your services isn't a simple general purpose endpoint. You should define your workloads and manage each workload separately - i.e. if your services can process CPU high, Memory high and IO requests then you can create three service object each with the InstanceCount=-1. The CPU and Memory based workloads probably would consume all available resource and creating more than one instance per node won't help here. Regarding the IO bound part then if the IO queue is overloaded then we can simply spawn one more service object to double the IO processing capacity._

I understand that allowing multiple instance per-node looks like the easiest solution but at the same time it isn't flexible too.

Getting back to your example:

• How many additional service instance should be created by Cluster Manager?
• When additional instance should be removed? What is the maximum amount of instances per node?
• Imagine that in you example you have 4 service instance per node then Node1 would process all of 400 requests and the rest of the cluster will do nothing. What would be the solutions?

I am not trying to say - you did everything wrong and now need to re-implement (no one can implement the application that would run everywhere). I am just saying that to utilize Service Fabric effectively you need to think in terms of Service Fabric and for some complex scenarios it definitely would require additional development and probably some refactoring of the application.

[guibirow]

I think you didn't get the point of this Issue:

Each service instance can utilize all resources.

They can, does not mean that they will.

Reporting metrics allows Cluster Manager to effectively organize service instances across the nodes to effectively utilize available resources.

This is why I should not worry about the resource consumption, and focus on the business logic instead, my service will do one thing and SF does the rest to balance the load.

In case when you need to have a more sophisticated scaling mechanism there are huge amount of API that allows you to dynamically change instance count, service object count and even application object count etc.

The main goal of this request was to make it simple, not convoluted with advanced requirements for simple things.

[OlegKarasik]

@guibirow

Then how do you propose to solve the issues I mentioned above?

Getting back to your example:

• How many additional service instance should be created by Cluster Manager?
• When additional instance should be removed? What is the maximum amount of instances per node?
• Imagine that in you example you have 4 service instance per node then Node1 would process all of 400 requests and the rest of the cluster will do nothing. What would be the solutions?

If we assume that

Each service instance can utilize all resources.

Then all these questions are already answered but if we don't then it becomes a very complicated problem.

The main goal of this request was to make it simple, not convoluted with advanced requirements for simple things.

It is great to make thing simple but in this case allowing having multiple instances per node would make them very complex.

[guibirow]

@OlegKarasik

There is no such rule that says a service should consume all resources in a node, if you design your application with this in mind, your application scalability & resilience will be impacted by the node limits or will throw money away with unused resources.

Most of SF talks suggest that you focus on high density nodes, that is totally different, in this case, many services consumes all the resources in a node, this way is correct to say "all services hosted in a node can\should utilize all resources available" to avoid waste of resources.

Abstracting the computation from the nodes capacity will make your services scale up to the cluster limits, this is the approach used by hadoop and now the same is being done with SF Mesh, where the nodes are taken away and you focus only on computing.

With the current placement limitation, you can't do that, because you have to design your services with these limits in mind.

Regarding the issues you've posted, they are Auto-Scaling issues, not placement issues(like this request), In any case, the answer for these questions does not matter, because they are stateless services, any service you remove from the pool does not have anything to loose. For stateful services, they are already handled by the SF Auto Scale logic, and if I have an application or script that does that for me, service fabric should not impede me of doing it.

[OlegKarasik]

Hi @guibirow

Most of SF talks suggest that you focus on high density nodes, that is totally different, in this case, many services consumes all the resources in a node, this way is correct to say "all services hosted in a node can\should utilize all resources available" to avoid waste of resources.

This is the exact case I was talking about. To make high density nodes Service Fabric should be able to reorganize service instances according the load metrics they provide.

When I was saying Service should consume all resources in a node this doesn't mean that if service need only one thread to do it's job efficiently then it should create N+1 thread just to consume resources.

The main idea is that if the service requires N+1 threads, 4+ GB or RAM or any other resources it should take these resources instead of expecting that there somewhere should be other instance who would share its work.

This allows you to perform better planning.

Imaging if one service instance can process 400 requests / s by using the whole CPU resource and you expect ~600 requests / s - then you need only two instances with the load balancing configured they both would consume around 50% of CPU on each node.

Taking this information (from metrics) into account Service Fabric can reorganize cluster in a way to compact these high CPU services with low CPU services to create high density nodes.

In case if load will grew up only a few instances of high CPU service will be created and along with that cluster reorganization will again essentially compact high CPU with low CPU services.

The thing here is that you always can solve the problem by using less service instances if service can efficiently utilize node resource. In this case new service instance would be created only when there is not enough processing power and if there is no available nodes then the only way to scale is to add more VM's or get better VM's.

[lovelangy]

We have the same problem, and we solved it by the 3rd solution that @guibirow mentioned.(by creating multiple Named Services from same service type).

by running New-ServiceFabricService multiple times with different ServiceName, you can create the same service type service with different name.

New-ServiceFabricService -ServiceTypeName "myseviceType" -Stateless -ApplicationName "fabric:/myappName" -ServiceName "fabric:/name1 -PartitionSchemeSingleton -InstanceCount 3 -PlacementConstraint "NodeType == xxxx"

it would be greate if SF team can enhance it.

[OlegKarasik]

Hello @lovelangy

Service Fabric design allows you to have very fine grained control over service scaleability by allowing you to partition service, create multiple services of the same type or even creating multiple applications of the same type.

But usage of these scaling capabilities requires a proper application design.

Image an example when you processing messages from multiple topics. You can have one service with singleton partitioning that will process all these messages or:

1. You can have multiple partition, one for each topic.

2. You can have a separate service for each topic.

   _This would even allow you to have different instance count depending on the processing power required for each topic._

In cases when data cannot be so easily distinguished you can always use Uniform partitioning and simply specify required instance count by setting partitions count + instance count i.e. if you need to have 3 instance per node then you can set partition count to 3 and instance count to -1.

[moldyseaswimmer]

If you included a UI feature in SFX to do something like below that would be suitable but allow us to specify the number of times New-ServiceFabricService is called

Connect-ServiceFabricCluster localhost:19000
New-ServiceFabricService -ApplicationName 'fabric:/rnd2' -ServiceName 'fabric:/rnd2/Stateless3' -ServiceTypeName 'Stateless2Type' -Stateless -PartitionSchemeSingleton -InstanceCount 1

Remove-ServiceFabricService -ServiceName fabric:/rnd2/Stateless3