Aks-engine: AKS-Engine v0.51.0 - Azure Stack Hub - Kubernetes 1.17.5 Cannot create persistent volume claims

[dbrennand]

Describe the bug
We are unable to create persistent volume claims using the provided StorageClasses:

NAME                PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
default (default)   kubernetes.io/azure-disk   Delete          Immediate           false                  3d3h
managed-premium     kubernetes.io/azure-disk   Delete          Immediate           false                  3d3h
managed-standard    kubernetes.io/azure-disk   Delete          Immediate           false                  3d3h

We attempted to deploy Grafana stable helm chart and the pod fails to bind the volume to the container with the following error:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  15s   default-scheduler  error while running "VolumeBinding" filter plugin for pod "grafana-5468d8d98c-zk94n": pod has unbound immediate PersistentVolumeClaims

Running kubectl get pvc shows pending pvc:

NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
grafana   Pending                                      default        15s

Upon further inspection of the pvc using kubectl describe pvc grafana:

Name:          grafana
Namespace:     monitoring
StorageClass:  default
Status:        Pending
Volume:
Labels:        app.kubernetes.io/instance=grafana
               app.kubernetes.io/managed-by=Helm
               app.kubernetes.io/name=grafana
               app.kubernetes.io/version=7.0.3
               helm.sh/chart=grafana-5.3.0
Annotations:   meta.helm.sh/release-name: grafana
               meta.helm.sh/release-namespace: monitoring
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    grafana-5654c579df-n26cs
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  7s (x3 over 36s)  persistentvolume-controller  Failed to provision volume with StorageClass "default": compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="NoRegisteredProviderFound" Message="No registered resource provider found for location 'azurestacklocation' and API version '2017-12-01' for type 'disks'. The supported api-versions are '2017-03-30'. The supported locations are 'azurestacklocation."

I have tried altering the StorageClass to managed-standard as well. No such luck. The issue looks like it's to do with the API version being used is not compatible with the version used by ARM on Azure Stack Hub.

Steps To Reproduce

API Model (kubernetes-azurestack.json):

{
    "apiVersion": "vlabs",
    "location": "",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorRelease": "1.17",
            "orchestratorVersion": "1.17.5",
            "kubernetesConfig": {
                "cloudProviderBackoff": true,
                "cloudProviderBackoffRetries": 1,
                "cloudProviderBackoffDuration": 30,
                "cloudProviderRateLimit": true,
                "cloudProviderRateLimitQPS": 3,
                "cloudProviderRateLimitBucket": 10,
                "cloudProviderRateLimitQPSWrite": 3,
                "cloudProviderRateLimitBucketWrite": 10,
                "kubernetesImageBase": "mcr.microsoft.com/k8s/azurestack/core/",
                "useInstanceMetadata": false,
                "networkPlugin": "kubenet",
                "kubeletConfig": {
                    "--node-status-update-frequency": "1m"
                },
                "controllerManagerConfig": {
                    "--node-monitor-grace-period": "5m",
                    "--pod-eviction-timeout": "5m",
                    "--route-reconciliation-period": "1m"
                },
                "addons": [
                    {
                        "config": {
                            "workspaceGuid": "redacted",
                            "workspaceKey": "redacted"
                        },
                        "name": "container-monitoring",
                        "enabled": true
                    }
                ]
            }
        },
        "customCloudProfile": {
            "portalURL": "https://portal.example.com",
            "identitySystem": ""
        },
        "featureFlags": {
            "enableTelemetry": true
        },
        "masterProfile": {
            "dnsPrefix": "kubemaster2115647254",
            "distro": "aks-ubuntu-16.04",
            "count": 3,
            "vmSize": "Standard_D3_v2"
        },
        "agentPoolProfiles": [
            {
                "name": "linuxpool",
                "distro": "aks-ubuntu-16.04",
                "availabilityProfile": "AvailabilitySet",
                "AcceleratedNetworkingEnabled": false,
                "vmSize": "Standard_D3_v2",
                "count": 3
            }
        ],
        "linuxProfile": {
            "adminUsername": "azureuser",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "redacted"
                    }
                ]
            }
        },
        "servicePrincipalProfile": {
            "clientId": "",
            "secret": ""
        }
    }
}

Grafana Helm chart values.yaml relevant to this issue:

# Tried this
persistence:
  type: pvc
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 10Gi
# Also tried this
persistence:
  enabled: true

Expected behavior
PersistentVolumeClaims are provisioned correctly without any errors.

AKS Engine version
v0.51.0

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"bb95a3814e91321a3821d669afc3707a8f35015f", GitTreeState:"clean", BuildDate:"2020-04-28T22:50:11Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Azure Stack Hub is running version: 1.2002.28.93

Just trying to create a PVC using the following yaml does not work either:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azsh-pvc-test
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-standard
  resources:
    requests:
      storage: 5Gi

[jadarsie]

hello @dbrennand, azure disk regressed on v1.17, could you please move to v1.16?
I will push a PR to remove v1.17 support.

[dbrennand]

Thanks for the response @jadarsie on this issue.

We can of course re-deploy our Kubernetes cluster with 1.16? But I don't understand why 1.17 has this issue? Is k8s v.1.17 ever going to have support for Azure Stack Hub?

Many thanks.

[jadarsie]

I do not think v1.17 support on Stack will make it on the next release (v0.53) as that's supposed to be out fairly soon. We are targeting v0.54 or v0.55.

[dbrennand]

Cool thanks @jadarsie 😄