Azure-docs: Cannot deploy application

[dr0pdb]

Hi,

I am following the guide on how to create Windows Server Containers on Azure Kubernetes Service. Link

I pulled the sample image demonstrated in the guide, tagged it and uploaded it to Azure container registry.
Upon deploying the application to Kubernetes, the pods are unable to pull the image from the container registry.

The output of the describe command for one of the pods is shown below

Name:               hello-world-56c76d8549-7248k
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aksnpwin000000/10.240.0.35
Start Time:         Sat, 01 Jun 2019 19:33:21 +0530
Labels:             app=hello-world
                    pod-template-hash=56c76d8549
Annotations:        <none>
Status:             Pending
IP:                 10.240.0.47
Controlled By:      ReplicaSet/hello-world-56c76d8549
Containers:
  hello-world:
    Container ID:
    Image:          pocaspnetcoreweb.azurecr.io/pocaspnetcoreweb:v1
    Image ID:
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  800m
    Requests:
      cpu:        100m
      memory:     300m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-m647n (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-m647n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-m647n
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason          Age                   From                     Message
  ----     ------          ----                  ----                     -------
  Normal   Scheduled       18m                   default-scheduler        Successfully assigned default/hello-world-56c76d8549-7248k to aksnpwin000000
  Normal   SandboxChanged  17m                   kubelet, aksnpwin000000  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          17m (x3 over 17m)     kubelet, aksnpwin000000  Failed to pull image "pocaspnetcoreweb.azurecr.io/pocaspnetcoreweb:v1": [rpc error: code = Unknown desc = a Windows version 10.0.18362-based image is incompatible with a 10.0.17763 host, rpc error: code = Unknown desc = Error response from daemon: Get https://pocaspnetcoreweb.azurecr.io/v2/pocaspnetcoreweb/manifests/v1: unauthorized: authentication required]
  Normal   Pulling         16m (x4 over 17m)     kubelet, aksnpwin000000  Pulling image "pocaspnetcoreweb.azurecr.io/pocaspnetcoreweb:v1"
  Warning  Failed          2m55s (x65 over 17m)  kubelet, aksnpwin000000  Error: ImagePullBackOff

The Events section has an event with the message

Warning  Failed          17m (x3 over 17m)     kubelet, aksnpwin000000  Failed to pull image "pocaspnetcoreweb.azurecr.io/pocaspnetcoreweb:v1": [rpc error: code = Unknown desc = a Windows version 10.0.18362-based image is incompatible with a 10.0.17763 host, rpc error: code = Unknown desc = Error response from daemon: Get https://pocaspnetcoreweb.azurecr.io/v2/pocaspnetcoreweb/manifests/v1: unauthorized: authentication required]

The message seems clear enough to me since it complaints about the windows version mismatch between the container host and the node. Am I right about it? If yes, then how do I make sure that the version match.

My kubernetes deployment file(has 1-2 minor changes from the one given in the sample).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  template:
    metadata:
      name: hello-world
      labels:
        app: hello-world
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": windows
      containers:
      - name: hello-world
        image: pocaspnetcoreweb.azurecr.io/pocaspnetcoreweb:v1
        resources:
          limits:
            cpu: 1
            memory: 800m
          requests:
            cpu: .1
            memory: 300m
        ports:
          - containerPort: 80
      imagePullSecrets:
      - name: acr-auth
  selector:
    matchLabels:
      app: hello-world
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
  selector:
    app: sample

Any help would be appreciated. Thank you!

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 6de70aec-9ebe-5655-6657-5c171de374b1
• Version Independent ID: c9eb35c8-8703-511b-987b-e2a441be2e47
• Content: Preview - Create a Windows Server container on an Azure Kubernetes Service (AKS) cluster
• Content Source: articles/aks/windows-container-cli.md
• Service: container-service
• GitHub Login: @TylerMSFT
• Microsoft Alias: twhitney

[mimckitt]

Thanks for the feedback! We are currently investigating and will update you shortly.

[jakaruna-MSFT]

@srv-twry Can you give me the base image for pocaspnetcoreweb.azurecr.io/pocaspnetcoreweb:v1

[dr0pdb]

@jakaruna-MSFT
Hi, As I mentioned already, I just pulled the mcr.microsoft.com/dotnet/framework/samples:aspnetapp image from docker hub as mentioned in the guide, tagged it with the required name and pushed it to ACR.

[eriawan]

@srv-twry

The error is quite obvious: The kubelet has reported that it doesn't support image based on 10.0.18362 and this is equal to Windows Server 1903. Unfortunately, AKS is using 10.0.17763, this is equal to Windows Server 2019, This means AKS supports Windows Server 2019 and the OS it runs is also using Windows Server 2019.

According to MS' Docker Hub of .NET Framework Samples the tag of mcr.microsoft.com/dotnet/framework/samples:aspnetapp is using Windows Server Core 1903 by default:

Therefore, I suggest that you should base your image on Windows Server 2019 based image:
mcr.microsoft.com/dotnet/framework/samples:aspnetapp-windowsservercore-ltsc2019

To @jakaruna-MSFT
The documentation should clarify clearly that Azure Kubernetes Service doesn't support Windows Server 1903 based image. Please update the doc page. User should be aware and informed about this kind of Docker's Windows based image default supported OS and compared with AKS latest supported OS for Windows based images.

[TylerMSFT]

@srv-twry - sorry for the frustration. I have just been assigned this content and will update which image to use in the topic.

[dr0pdb]

@eriawan Thank you!
@TylerMSFT No worries! I was able to successfully deploy the application using mcr.microsoft.com/dotnet/framework/samples:aspnetapp-windowsservercore-ltsc2019 image.

[jakaruna-MSFT]

@TylerMSFT I have assigned this issue to you.

[eriawan]

@TylerMSFT

Could we have update in this AKS doc page on which OS is supported and also clarification that 1903 (10.0.18362) based image is not supported?

Update
Aha, looks like it's been working on by @TylerMSFT
Thanks @jakaruna-MSFT 👍

[TylerMSFT]

Hi,
I don't think it's true that 1903 (10.0.18362) is not supported. I think something else went wrong in @srv-twry's journey. Following the doc works fine for me.

please-close

[dr0pdb]

@TylerMSFT
Hi, what went wrong is that the nodes created in the windows node pool were running the 10.0.17763 version. Hence I should have used the sample for that version and not the 1903 version. This I was already able to figure out from the logs. My question asked in the issue description is still unanswered though, so I'll ask it again.

When a user is creating a windows node pool to be added to an existing AKS cluster, Is it possible that he/she can also specify that this node pool should contain nodes running a specific windows version?

For eg. Can I say that "hey in this node pool, create 2 nodes both of which should be running the 1903 version"?

[TylerMSFT]

@srv-twry

When a user is creating a windows node pool to be added to an existing AKS cluster, Is it possible that he/she can also specify that this node pool should contain nodes running a specific windows version?

No. You can't define the version of Windows server when creating a node, just as you can't define the version of Ubuntu on a Linux node.

[feitzi]

Any updates?
We have the problem that we can't build our images on base image **-windowsservercore-ltsc2019 because it has not up to date root certs.

[MikkelHegn]

Tracking here: https://github.com/Azure/AKS/issues/1454