Azure-pipelines-agent: Azure virtual machine scale set agents with containers

[achamayou]

I am trying out the preview of Azure virtual machine scale set agents (https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/vmss?view=azure-devops), on https://dev.azure.com/MSRC-CCF/

I'm able to get a dynamic pool of agents running Standard_DC SKUs, that's fantastic! I want to run jobs on them that make use of docker though (ie. "container:" in my jobs), and that's not installed out of the box on the image I use (Ubuntu 18.04 LTS Gen2). So I've added a cloud-init script, which does seem to work as expected.

After I hand over the scale set to Pipelines for management though, it looks like the cloud-init has disappeared, and VMs spawned after that do not have docker on them. I'm guessing Pipelines is perhaps overriding the cloud-init settings rather than appending to them, and that's fair, the documentation even points out this will probably happen:

Azure Pipelines updates the configuration of the scale set. Any manual changes you make to the scale set may interfere with the operation of Azure Pipelines.

I don't particularly want a generic opportunity to customise VMs in the set or the set itself, all I need is docker, and I suspect that will be true for many other users. I'm wondering if I haven't missed something obvious, but looking back carefully, I can't find what it is.

Apart from this, the VM Scale sets agent look extremely promising!

[petenorth]

Is this still the observed behaviour or is it possible to use cloud-init scripts with vmss agents?

[craigforr]

@achamayou: Did you ever get this working? We are attempting to do the same thing: Run Azure Pipelines container jobs on top of a VM scale set agent.

[achamayou]

@craigforr unfortunately no, I've stayed with statically-managed pools. I did briefly contemplate adding an extra job to all my pipelines, on which all others would depend, and which would apt install docker before container jobs kicked off, but decided against it.

[xenalite]

Are you able to stick Docker on the VM any other way?

• Custom VM image with Docker pre-installed.
• Custom script VM Extension?

[achamayou]

1. I guess that's possible, but I'd have to make a custom image, host it somewhere the VMSS setup can pick it up. It's a lot of hassle to enable what I imagine must one of the most widely used Azure Pipelines features (on Linux anyway).
2. I hadn't heard of those (I guess you mean https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux), it looks like it could work (assuming I can tweak the ARM template that's used by the VMSS, but that sounds quite likely). I'll give that a try, thank you for the suggestion @xenalite!

[qmfrederik]

Hmm, in my scenario, it seems like the cloud-init script _does_ run, although it seems to race with the Azure Pipelines Extension:

• Sometimes, Docker is installed when the first build kicks off, and all is fine
• Sometimes, Docker is not installed when the first build kicks off, but installation will complete a bit later, and restarting the job works
• Sometimes, Docker seems to never get installed, and the VM needs to be completely reimagined or has to be deleted

@xenalite I'm wondering how a VM extension would work. Is there a way to force the Azure extension (which installs the agent and registers the agent with VSTS) to run after the custom VM extension?

Other than that, I agree with @achamayou that maintaining a custom VM image for Docker only seems like a lot of overhead, for what feels like a common scenario...

[qmfrederik]

Based on https://github.com/Azure/WALinuxAgent/issues/1938#issuecomment-657293920, it looks like this cloud-init configuration should work. It will delay the start of the Linux Agent until after the cloud-init script has completed installing Docker:

#cloud-config

bootcmd:
  - mkdir -p /etc/systemd/system/walinuxagent.service.d
  - echo "[Unit]\nAfter=cloud-final.service" > /etc/systemd/system/walinuxagent.service.d/override.conf
  - sed "s/After=multi-user.target//g" /lib/systemd/system/cloud-final.service > /etc/systemd/system/cloud-final.service
  - systemctl daemon-reload

apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

packages:
  - docker-ce
  - docker-ce-cli

groups:
  - docker