External-storage: Build v2.3.0 of efs-provisioner fails to start

[markAcomm]

We were just doing a rolling update on our cluster and this pulled a new image of quay.io/external_storage/efs-provisioner:latest (only 7 hours old). This new version failed to start up on our cluster with the following output:

/efs-provisioner flag redefined: log_dir
panic: /efs-provisioner flag redefined: log_dir

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc000094120, 0x159ed00, 0x214fe70, 0x13da152, 0x7, 0x1405662, 0x2f)
    /usr/local/go/src/flag/flag.go:805 +0x529
flag.(*FlagSet).StringVar(0xc000094120, 0x214fe70, 0x13da152, 0x7, 0x0, 0x0, 0x1405662, 0x2f)
    /usr/local/go/src/flag/flag.go:708 +0x8a
github.com/kubernetes-incubator/external-storage/vendor/k8s.io/klog.InitFlags(0xc000094120)
    /go/src/github.com/kubernetes-incubator/external-storage/vendor/k8s.io/klog/klog.go:411 +0x7b
main.main()
    /go/src/github.com/kubernetes-incubator/external-storage/aws/efs/cmd/efs-provisioner/efs-provisioner.go:275 +0x3c

When we reverted back to the prior version: v2.2.0-k8s1.12 the efs-provisioner was able to run successfully.

Was there potentially something wrong with the build? Or is it possible we have something configured incorrectly?

[martymh]

Likewise for us.

The latest image is failing to start and is in a crash loop backoff: v2.3.0, sha256:15ddb18ffb9598841c4987f3ba0ef89dc175b137d2ee529bb647e1ae5e8939cd

The previous image appears to be working fine: sha256:c12dae08925371318ae8a6de2e21a27c581f1d8bbd59876e8ae4292d36e0a300

But there does not appear to be a tag for this anymore.

[mozimos]

I also was able to start pod using v2.2.0-k8s1.12.

[xwang2713]

Got the same with quay.io/external_storage/efs-provisioner:latest
A month ago the latest version worked. Guess it was updated recently

[wongma7]

Yes, please revert. I'm going to rebuild and repush the previous version to the latest tag.

[richstokes]

Broken here also

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"37d169313237cb4ceb2cc4bef300f2ae3053c1a2", GitTreeState:"clean", BuildDate:"2019-08-19T10:44:49Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

/efs-provisioner flag redefined: log_dir
panic: /efs-provisioner flag redefined: log_dir

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc000094120, 0x159ed00, 0x214fe70, 0x13da152, 0x7, 0x1405662, 0x2f)
    /usr/local/go/src/flag/flag.go:805 +0x529
flag.(*FlagSet).StringVar(0xc000094120, 0x214fe70, 0x13da152, 0x7, 0x0, 0x0, 0x1405662, 0x2f)
    /usr/local/go/src/flag/flag.go:708 +0x8a
github.com/kubernetes-incubator/external-storage/vendor/k8s.io/klog.InitFlags(0xc000094120)
    /go/src/github.com/kubernetes-incubator/external-storage/vendor/k8s.io/klog/klog.go:411 +0x7b
main.main()
    /go/src/github.com/kubernetes-incubator/external-storage/aws/efs/cmd/efs-provisioner/efs-provisioner.go:275 +0x3c

[richstokes]

@wongma7 Did the correct build get pushed to latest please? So we know if/when we should be good to try again?

[wongma7]

@richstokes I just moved the tag so that latest should point to the previous version. Sorry for the delay.

[richstokes]

Thanks! It's starting up OK now 👍

[szmytson]

Would it be possible to bump the version of alpine to 3.10 as well? (See https://github.com/kubernetes-incubator/external-storage/pull/1219)

[wongma7]

v2.4.0 is out with the crash fix AND the alpine bump. It works on my end, but would appreciate if any brave person would like to give it a try before I close this issue. https://quay.io/repository/external_storage/efs-provisioner?tab=tags

[whereisaaron]

v2.4.0 is working for me so far @wongma7 👍
No crash and dynamic provisioning, mounting, deletion, all good with EKS 1.14.

The IAM Service Account credentials appear to be working too, at least I don't see a session error or check error. (It would be nice if the test has a success log message @wongma7 🙏).

I tested with eksctl create iamserviceaccount --config-file=cluster-config.yaml

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: efs-provisioner
        namespace: efs-provisioner
        labels:
          aws-usage: cluster-ops
      attachPolicyARNs:
      - 'arn:aws:iam::aws:policy/AmazonElasticFileSystemReadOnlyAccess'

and

helm upgrade --install efs-provisioner \
  --namespace=efs-provisioner \
  --values efs-provisioner.yaml \
  --version=0.8.0 \
  stable/efs-provisioner