Azure-docs: How to trigger deletion of Unhealthy Instance of Azure VMSS

[Deepudev13]

Current Scenario

I have setup an Azure VMSS behind Azure Standard LB.
There HTTP Probe is configured to check Instance health which is working great.
As soon as LB gets signal of unhealthy VM it stops the further incoming traffic to be routing to unhealthy VM.
Auto-scale configuration is configure as

• Scale out when CPU utilization is greater than 65 %
• Scale in when CPU utilization is less than 40 %

Lets assume Current State of Azure VM is

• Instance 1 --> 60 % CPU Utilization
• Instance 2 --> 60 % CPU Utilization
• Instance 3 --> 60 % CPU Utilization

Average is 60 % CPU Utilization

Problem

Let say at some point of time there are 3 Instances are running of VMSS.
Instance 1 got unhealthy and LB is now aware about it. So the incoming request would stop to coming to Instance 1. As expected, Instance 2 and Instance 3 keeps on serving web services to public request. But the issue is Azure LB doesn't stop/remove/delete/restart the unhealthy VM, i.e., Instance 1.
Now the new load of the virtual machine would be

• Instance 1 --> 0 % CPU Utilization
• Instance 2 --> 90 % CPU Utilization
• Instance 3 --> 90 % CPU Utilization

Average is 60 % CPU Utilization

This can be very critical condition as in such a case Application pool might be crashed of the healthy VM and can lead to Unhealthy state.

As Auto-scaling configured to scale out vm only when CPU percentage above 65 %. Hence the whole setup would be in the same state and would be Worse than the worst.

Prospect Solution (But it does not exist as of now, I think)

Unhealthy Instance must be deleted. So the case would be

• Instance 2 --> 90 % CPU Utilization
• Instance 3 --> 90 % CPU Utilization

Average is 90 % CPU Utilization

As CPU utilization is greater than the 65 %. Hence Auto-scale of Azure VMSS would trigger a new provisioning of new Instance Instance 4

Can you please suggest me solution as even after using vmss we have to keep eye on Azure VMs which is not pure automation.

[himanshusinha-msft]

@Deepudev13 : Thanks for reaching out to us , but at this time it does not look like you are referring to a document but to a specific scenario . If that's the case we request you to please post your question on MSDN and other forums , we have our engineers and others who are monitoring them and will be happy to help .

If you have a document which you are following and if you think it needs to an update , please let us know about that here .

[jakaruna-MSFT]

I will check with the internal teams and let you know.

[jakaruna-MSFT]

@Deepudev13 Looks like we dont have an automated way to remove unhealthy VM's.
I also suspect that Auto scaling uses all the machines CPU usage irrespective or healthy or unhealthy.

[jakaruna-MSFT]

@Deepudev13 Please submit a feedback here. Also post the feedback url here so that it will help the community to upvote. Product teams picks the items from the feedback's provided in the above link.
I will also take up this issue and report it to the product team.

I will close this issue for now. If you need additional help, Please mention me in the comment.

[Deepudev13]

Hi @jakaruna-MSFT
I have posted my idea
https://feedback.azure.com/forums/217313-networking/suggestions/37311568-azure-loadbalancer-must-delete-unhealthy-vm-of-azu
Please vote it if you think so...