Aks-engine: aad-pod-identity addon scheduling mic pod to Windows pool

[jackfrancis]

Events:
  Type     Reason            Age                   From                      Message
  ----     ------            ----                  ----                      -------
  Warning  FailedScheduling  23m (x2 over 23m)     default-scheduler         0/4 nodes are available: 2 node(s) had taint {node-role.kubernetes.io/master: true}, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/aad-pod-identity-not-ready: true}, that the pod didn't tolerate.
  Warning  FailedScheduling  23m (x4 over 23m)     default-scheduler         0/5 nodes are available: 2 node(s) had taint {node.kubernetes.io/aad-pod-identity-not-ready: true}, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: true}, that the pod didn't tolerate.
  Warning  FailedScheduling  23m (x2 over 23m)     default-scheduler         0/6 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/aad-pod-identity-not-ready: true}, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: true}, that the pod didn't tolerate.
  Normal   Scheduled         22m                   default-scheduler         Successfully assigned kube-system/mic-669596b74f-brczm to 2941k8s06000000
  Normal   Pulling           22m (x3 over 22m)     kubelet, 2941k8s06000000  Pulling image "mcr.microsoft.com/k8s/aad-pod-identity/mic:1.2"
  Warning  Failed            22m (x3 over 22m)     kubelet, 2941k8s06000000  Failed to pull image "mcr.microsoft.com/k8s/aad-pod-identity/mic:1.2": rpc error: code = Unknown desc = image operating system "linux" cannot be used on this platform
  Normal   SandboxChanged    22m (x7 over 22m)     kubelet, 2941k8s06000000  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed            2m35s (x92 over 22m)  kubelet, 2941k8s06000000  Error: ImagePullBackOff

We should add a Linux nodeSelector to the mic deployment so that it's never scheduled onto a Windows node.

[jackfrancis]

@sozercan @kkmsft can you quickly confirm that we should enforce Linux-only scheduling for the mic component?

[sozercan]

@chewong @aramase

[aramase]

@jackfrancis yeah, we can enforce linux only scheduling. We already do that in the aad-pod-identity repo - https://github.com/Azure/aad-pod-identity/blob/master/deploy/infra/deployment-rbac.yaml#L159-L160. The aks-engine aad-pod-identity is using an old version of image and template. I'll open a PR to update it.

[jackfrancis]

Thanks! I've created #3420 to fix the mic nodeSelector oversight. Is there anything more that should go in there?

[aramase]

That PR should be sufficient to ensure it doesn't land on windows nodes. I'll open a separate PR to bump the manifests and image version to latest release.

[jackfrancis]

@aramase thanks!

[aramase]

Opened PR to update all the manifests and version - https://github.com/Azure/aks-engine/pull/3423