Aks-engine: Use pools with different availabilityProfiles (AvailabilitySet and VirtualMachineScaleSets)

[palmerabollo]

Is this a request for help?:

Yes.

---

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

---

What version of aks-engine?:

v0.30.1

---

Kubernetes version:

1.10

What happened:

It is not possible to use node pools with "availabilityProfile": "AvailabilitySet" and "availabilityProfile": "VirtualMachineScaleSets" at the same time.

mixed mode availability profiles are not allowed. Please set either VirtualMachineScaleSets or AvailabilitySet in availabilityProfile for all agent pools (see code)

I would like to know if it is impossible due to a technical restriction or if it is something that could eventually be implemented.

What you expected to happen:

aks-engine should support deployments with mixed types of node pools.

How to reproduce it (as minimally and precisely as possible):

Deploy a cluster with the following agentPoolProfiles:

      {
        "name": "pool1",
        "count": 1,
        "vmSize": "Standard_DS1_v2",
        "availabilityProfile": "AvailabilitySet"
      },
      {
        "name": "pool2",
        "count": 1,
        "vmSize": "Standard_DS1_v2",
        "availabilityProfile": "VirtualMachineScaleSets"
      },

[serbrech]

Hi, thanks for reporting this.
By curiosity, could you give us more details on why this use-case matters to you?

[palmerabollo]

@serbrech the use case is being able to add a new node pool that supports low-priority instances to an existing cluster that already uses AvailabilitySet-based pools.

We have an existing K8S cluster that is not using vmss. Now, we want to add a new node pool to run batch jobs on the same cluster, but aks-engine doesn't allow it. We need to use low-priority instances (only compatible with vmss) in this new pool because otherwise our solution would be very expensive.

I see different solutions:

1. Allow mixed mode availability profiles. I've forked aks-engine to remove that restriction and I've deployed a cluster with both types of profiles and everything seems to be working fine. This is why I'd like to know if there is some kind of technical restriction I'm not aware of.
2. Allow rolling upgrades to be able to change the type of a node pool in an existing cluster without service disruption. We have other clusters in AWS with kops and it is able to deal with these kind of upgrades quite well.
3. Allow low-priority instances in AvailabilitySet-based pools, which I think is not possible with Azure yet. If I'm not wrong, low-pri instances must be associated with a scaleset.

[palmerabollo]

There might be an issue with the cluster-autoscaler addon, that works different for vmms and for non-vmss pools. If that's the case, I'd add a check to apply the restriction (disable mixed mode availability profiles) only when the addon is enabled or, even better, support for two autoscalers (one for the vmss and another one for the non-vmss pools). Are you open to contributions in this regard?

[jackfrancis]

Let's spike on simply removing the validation error and evaluate what doesn't work.

Cluster-autoscaler addon is currently very limited (one nodepool, VMSS only), so shouldn't be a blocker. It's possible to completely customize a cluster-autoscaler implementation that conforms to a particular cluster configuration. In other words, we won't block just because our current, limited cluster-autoscaler addon implementation doesn't work.

[palmerabollo]

We managed to get mixed profiles working with our own aks-engine fork (disabling the check and the autoscaler addon).

We are also able to deploy our own autoscalers (one for vmss and one for standard agent pools, one of them with leader-elect disabled because the resource name they use to acquire a lock is hardoded in the azure version of the cluster-autoscaler).

It is not possible to update an existing cluster (see #528) but it works in new Kubernetes clusters for us. It would be great to officially support it in Azure/aks-engine.

[yevgenytr]

@palmerabollo How did you forked the code?
You commented out the check part and compiled it yourself?

EDIT: I cloned the code, commented out the line and built it (build instructions can be found here).
Now I'm waiting for the official fix.

[palmerabollo]

@yevegnytr yes, that's exactly what I did, using the following Dockerfile to build it:

FROM golang:1.11.5-alpine3.9 as build

ENV AKS_VERSION v0.32.3

RUN apk --no-cache add git make bash
RUN git clone https://github.com/Azure/aks-engine.git \
    -b ${AKS_VERSION} --single-branch \
    /go/src/github.com/Azure/aks-engine

WORKDIR /go/src/github.com/Azure/aks-engine
RUN CGO_ENABLED=0 make LDFLAGS='-extldflags "-static"'

The binary is left at /go/src/github.com/Azure/aks-engine/bin/aks-engine. You can use a multistage Dockerfile to get rid of everything else, if needed:

FROM alpine:3.9

RUN apk --no-cache add ca-certificates
COPY --from=build /go/src/github.com/Azure/aks-engine/bin/aks-engine /bin/aks-engine
ENTRYPOINT ["/bin/aks-engine"]

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.