Aks-engine: Mounting volume to pods fails randomly.

[jkniahnicki]

Is this a request for help?:

Yes.

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

What version of aks-engine?:
aks-engine version
Version: v0.29.1
GitCommit: b35549b

GitTreeState: clean

Kubernetes version:
1.13

What happened:
When I wanted to mount one volume to couple of pods, error occured at one of installed pods. After that this volume couldn't be mounted to any other pod.

What you expected to happen:
Volume can be mounted many times and used by many different pods.

How to reproduce it (as minimally and precisely as possible):

1. Create volume: az storage share create --account-name $STORAGE_ACCOUNT_NAME --account-key $STORAGEKEY --name "logs" --quota 1
2. Install pods which use "logs" volume. I.e.: helm install empty-pod --name empty-pod-1 (see pod.txt for details of pod)
3. After installing some pods, next pod hangs in state 'ContainerCreating' and 'kubectl describe pod' shows error: MountVolume.SetUp failed for volume "logs" (see emptyPodLogs_aks.txt for more details)
4. Every other pod which uses that volume hangs in this state after installation.

Anything else we need to know:
By using acs-engine mount was failing on second installed pod.

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[fberlin]

Same issue here..

[mboersma]

For reference, this error appears to be coming from this Kubernetes code: https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount_windows.go#L100

New-SmbGlobalMapping : Generic failure \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...

[andyzhangx]

is it on windows? could you provide the details info, e.g. k8s version, windows version, I am responsible for making azure file working on k8s.

[jkniahnicki]

@andyzhangx Kubernetes version is v1.13. Those are windows containers. In attached example 'empty-pod' mcr.microsoft.com/powershell:nanoserver was used. Other pods were using 1809 as base image.

[fberlin]

@andyzhangx
Kubernetes: 1.13.1
Nodes: 2 x Windows 1809
The docker image in my pods are based on mcr.microsoft.com/dotnet/framework/aspnet:4.7.2-windowsservercore-ltsc2019

[fberlin]

@andyzhangx
My steps to reproduce:

1. Create a Storage account in Azure with a file share.
2. Generate secret in Kubernetes to connect to the file share.
3. Create PV + PVC in Kubernetes using secret and azurefile.
4. Spin up a deployment with one pod using PVC above for volume mount.
5. Everything works. I can use the volume as intended.
6. Manually delete the pod and watch Kubernetes fail to spin up a new.

[andyzhangx]

@fberlin I have identified that this is an upstream bug on Windows, it only affects azure file mount on Windows. If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since New-SmbGlobalMapping command would fail if file share is already mounted on the node, details could be found:
https://github.com/kubernetes/kubernetes/issues/73087

[andyzhangx]

I will fix this issue later.

[pszlendak]

@andyzhangx What is the ETA for the fix? This problem is blocking the development of our backoffice on Azure ...

[andyzhangx]

@pszlendak it would be fixed in v1.14, and then back port this fix to prior version. The complete fix could be long time, while I could provide a quick fix for you if it's really blocking your development issue.

[pszlendak]

@andyzhangx I'd really appreciate a quick fix or a workaround to unblock development.

[fberlin]

@andyzhangx Me and my team is standing still due to this bug so we also would like a quick fix to get going again.

[andyzhangx]

@fberlin I pushed a quick fix commit(https://github.com/andyzhangx/kubernetes/commit/11a6d5a640ecf0f9d46a1c38fb13e9e6ed201edc) for this issue, the branch azurefile-windows-quickfix is based on k8s v1.13.1, you could use this patch on any branch, and build you own kubelet. To verify that, only replace the c:\k\kubelet.exe on all windows agent nodes by:

stop-service kubeproxy
stop-service kubelet
#copy kubelet.exe to c:\k
start-service kubeproxy
start-service kubelet

To completely fix this issue, I need to do a little refactor on this function which would require more time. I will do that after back from my vacation.

[CJJ69]

Hi @andyzhangx
We are also hitting this problem
Kubernetes: 1.13.2
Nodes: Windows 1809 - windowsservercore-ltsc2019

Cluster created with latest aks-engine

We need a shared data folder accessible to multiple pods.

We tried PersistentVolume with StorageClass azurefile; provisioner: kubernetes.io/azure-file

But the pods error when creating:

MountVolume.SetUp failed for volume "pvc-9698aafc-1f2d-11e9-8fc3-000d3ab8ccf8" : azureMount: SmbGlobalMapping failed: exit status 1, only SMB mount is supported now, output: "New-SmbGlobalMapping : Generic failure At line:1 char:190 .. CategoryInfo : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMapping], CimException FullyQualifiedErrorId : HRESULT 0x80041001,New-SmbGlobalMapping"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fileshare-pvc
  labels:
    type: azurefile
spec:
  storageClassName: azurefile
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi

---  
apiVersion: apps/v1
kind: Deployment
...
        volumeMounts:
          - mountPath: C:/Repository
            name: shared-data
            subPath: Repository
          - mountPath: C:/TaskLogs
            name: shared-data
            subPath: TaskLogs

    volumes:
      - name: shared-data
        persistentVolumeClaim:
          claimName: fileshare-pvc

[andyzhangx]

Thanks for all reporting, looks like azure file mount on windows is very popular and I need to re-prioritize this work item. cc @PatrickLang

[andyzhangx]

By following command to build a kubelet.exe binary:

~/go/src/k8s.io/kubernetes# KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet

You may need to download following v1.13.1-1int.zip and replace with new kubelet.exe and then rezip

# azuredeploy.parameters.json
    "kubeBinariesSASURL": {
      "value": "https://acs-mirror.azureedge.net/wink8s/v1.13.1-1int.zip"
    },

[CJJ69]

@andyzhangx thanks for the suggested quick fix.
I tried it and we got a bit further...
Now if the pod restarts we get a different exception:

MountVolume.SetUp failed for volume "pvc-6fd5955f-2002-11e9-86e6-000d3ab4e027" : azureMount: SmbGlobalMapping failed: exit status 1, only SMB mount is supported now, output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again

[andyzhangx]

@CJJ69 I had a discussion with windows team, they confirmed this is a bug on windows level, which means re-map the same remote path should always work. So there is nothing to do on k8s level for this issue since newer windows version will fix this bug.
On the other hand, windows team provide an alternative to fix this issue, I will try to provide a temp workaround fix.

[fberlin]

@andyzhangx We have applied your fix with great success. Thanks!

[andyzhangx]

After a few back and forth discussion with windows team, I decided to fix it on k8s level, here is the PR:
https://github.com/kubernetes/kubernetes/pull/73661
You could use this PR to do the workaround now, I will cherry pick the fix to prior version when this PR is merged.

[andyzhangx]

kubernetes/kubernetes#73661 has been merged, and I am cherrying pick this fix to ks8 v1.10 ~ v1.13
You could find most updated details about the cherry pick version here:
https://github.com/andyzhangx/demo/blob/master/issues/azurefile-issues.md#11-azure-file-remount-on-windows-in-same-node-would-fail

Let me know if you have any question, thanks.
/close

[acs-bot]

@andyzhangx: Closing this issue.

In response to this:

kubernetes/kubernetes#73661 has been merged, and I am cherrying pick this fix to ks8 v1.10 ~ v1.13
You could find most updated details about the cherry pick version here:
https://github.com/andyzhangx/demo/blob/master/issues/azurefile-issues.md#11-azure-file-remount-on-windows-in-same-node-would-fail

Let me know if you have any question, thanks.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.