Kops: Unable to use xfs with StorageClass

[vvbogdanov87]

1. What kops version are you running?
   Version 1.8.1 (git-94ef202)

2. What Kubernetes version are you running?
   Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
   Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?
   AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ci-slaves
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  zone: us-west-2c
  fsType: xfs

```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ci-merge-build
labels:
app: jenkins
name: ci-merge-build
spec:
accessModes:
- ReadWriteOnce
storageClassName: ci-slaves
resources:
requests:
storage: 40Gi

I use kubernetes plugin for jenkins. ci-merge-build pvc is used as workspace persistent volume in pod template.
```console
kubectl describe pod merge-build-dpmhc

1. What happened after the commands executed?

Warning  FailedMount            1m (x9 over 3m)  kubelet, ip-172-20-97-92.us-west-2.compute.internal  MountVolume.MountDevice failed for volume "pvc-02506491-1b11-11e8-b9f1-0254cc476042" : executable file not found in $PATH

1. What did you expect to happen?
   kubernetes creates jenkins slave pods

Looks like xfs format tool(mkfs.xfs?) is not included in https://kubeupv2.s3.amazonaws.com/kops/1.8.1/images/protokube.tar.gz

[gambol99]

Hi @vvbogdanov87, i don't think this is a kops issue really ... If you want mkfs.xfs on the host operating system I'd suggest using a hook

[vvbogdanov87]

As I understand from makefile, protokube is built using kops/images/protokube/Dockerfile. In the Dockerfile e2fsprogs(ext2-4 utils) is explicitly installed. Can xfs be added to this image as well?

[gambol99]

So protokube is a sidekick service which is responsible for managing the etcd cluster running on the master nodes. It wouldn't be involved in making filesystem's on attached volumes; this would done by the compute kubelets, with the kubelet using local binaries

@chrislovecnm @KashifSaadat

[vvbogdanov87]

In "kops update" output I found that protokube is included in userdata of all IGs(not only in master IG). That's why I thought it can be used to deliver tools to node instances.
So xfs should be included in node AMI?
https://github.com/kubernetes/kube-deploy/blob/d90cb4e0c37a27afe4cb940e93d00d6e9aca0964/imagebuilder/templates/1.8-jessie.yml#L93

[chrislovecnm]

@vvbogdanov87 protokube is a helper container, used for etcd bootstrapping and a few other things. You need to get xfs installed on the ami, using a hook can do that for you.

[vvbogdanov87]

Thanks! Found related issue kubernetes/kube-deploy#287

[simplypost]

@vvbogdanov87 : Have you resolved the issue ?

[vvbogdanov87]

@simplypost

apiVersion: kops/v1alpha2
kind: InstanceGroup
...
spec:
  hooks:
  - execContainer:
      command:
      - sh
      - -c
      - chroot /rootfs apt-get update && chroot /rootfs apt-get install -y xfsprogs
      image: busybox
...