Virtual-environments: Ubuntu1804 image using VMSS takes up to an hour to provision new DevOps agents

Created on 15 Jul 2020 · 8Comments · Source: actions/virtual-environments

Describe the bug
When using a Virtual Machine Scale Set in UKSouth with the ubuntu1804 image from this repository new instances either hang in Creating(Running) or take over an hour to provision.
When not utilising the Azure DevOps extension the instances are able to spin up within a minute or two

Area for Triage:

Question, Bug, or Feature?:
Bug

Virtual environments affected

[ ] macOS 10.15
[ ] Ubuntu 16.04 LTS
[ x] Ubuntu 18.04 LTS
[ ] Ubuntu 20.04 LTS
[ ] Windows Server 2016 R2
[ ] Windows Server 2019

Expected behavior
A DevOps VMSS Instance agent should be created in a reasonable timeframe

Actual behavior
Create a Virtual Machine Scale Set in UKSouth
Set to 0 instances
Enable System Managed Identity
Set to use a Managed Image based on ubuntu1804 from this repository

Add a new VMSS Agent Pool - 1 Standby instance, 5 Maximum
Create a new pipeline using the agent pool

Observe thhe VMSS instances hanging and multiple health events being raised

external

Source

mihsoft

Most helpful comment

I pushed out a fix today. It should be everywhere within a week.

WillLennon on 15 Jul 2020

👍2

All 8 comments

Hi @mihsoft !
Could you, please, confirm that the same issue cannot be reproduced with any other Ubuntu image from Azure marketplace?

vmapetr on 15 Jul 2020

Hi Mihsoft,
Can you please file a ticket in the Azure DevOps developer community and include specific details like your account name and agent pool name. Thanks.
https://developercommunity.visualstudio.com/content/problem/post.html?space=21

WillLennon on 15 Jul 2020

@vmapetr deploying a new 1804-LTS VMSS using the following Terraform source configuration:

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "18.04-LTS"
    version   = "latest"
  }

Will report back as soon as I've got the pool up and running in Azure DevOps.

mihsoft on 15 Jul 2020

@WillLennon thanks - Looks like it has already been raised by another user - https://developercommunity.visualstudio.com/content/problem/1099127/scale-set-vms-taking-long-time-to-create-due-to-lo.html

Appears to be an issue with the specific Azure DevOps provisioning script attempting to chmod entire directories.

mihsoft on 15 Jul 2020

@WillLennon @vmapetr to confirm I have confirmed the issue does not affect the standard 1804-LTS image, mainly because there isn't any software pre-installed so the DevOps agents installation scripts run very quickly indeed. (<1 min)
Using Standard_DS2_v2 instances with Premium SSD storage

mihsoft on 15 Jul 2020