fluxctl snap cannot exec auth helpers (ex: aws-iam-authenticator)

[stealthybox]

Describe the bug
It's not possible to auth with KUBECONFIG's that specify an exec for auth tokens.
This is true in the case of EKS, GKE, and likely many other IaaS/KaaS providers.

To Reproduce

1. configure your KUBECONFIG to point to an EKS cluster /w fluxd running
2. install the fluxctl snap
3. attempt to run fluxctl sync

Expected behavior
It's unreasonable to expect that the snap could have every binary needed to perform exec's for auth tokens for every single cluster provider.
The fluxctl snap should be able to exec other binaries on the system.
It likely needs to respect the user's PATH as well.
This mirror's kubectl's needs.

I'm not sure if there are more precise ways to accomplish this level of access.
kubectl uses classic confinement.
I believe we also need to have fluxctl be a classic snap for similar reasons.

Logs

# `eksctl utils write-kubeconfig` produced this kubeconfig:
tail -n9 ~/.kube/config
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - stealthybox-appmesh
      command: aws-iam-authenticator
      env: null

fluxctl sync --k8s-fwd-ns flux
Error: Could not create a dialer: Could not get pod name: Listing pods in kubernetes: Get https://0EE43986120C1158DFB3B283D67144C6.sk1.eu-west-2.eks.amazonaws.com/api/v1/namespaces/flux/pods?fieldSelector=status.phase%3DRunning&labelSelector=app%3Dflux: getting credentials: exec: exec: "aws-iam-authenticator": executable file not found in $PATH
Run 'fluxctl sync --help' for usage.

Versions
fluxd: docker.io/fluxcd/flux:1.15.0
fluxctl: v1.15.1 (https://snapcraft.io/fluxctl)

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-07T14:30:40Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

/cc @dholbach

[stealthybox]

Related:
https://github.com/fluxcd/flux/pull/2427

^ we use personal-files for access to very specific $HOME subdirs, but kubernetes client auth could be told to access files from many places.

If classic confinement is not possible, perhaps we could provide a best-effort list of up-to-date auth token binaries.

We'll need to add personal-files access to directories like ~/.aws, ~/.config/gcloud, and many others if we go that route.

[JVMartin]

I am also having this problem with DigitalOcean Kubernetes. Kube config excerpt:

users:
- name: <redacted>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kubernetes
      - cluster
      - kubeconfig
      - exec-credential
      - --version=v1beta1
      - --context=default
      - <redacted>
      command: doctl
      env: null

Error message:

» fluxctl identity --k8s-fwd-ns=flux
Error: Could not create a dialer: Could not get pod name: Listing pods in kubernetes: Get https://<redacted>.k8s.ondigitalocean.com/api/v1/namespaces/flux/pods?fieldSelector=status.phase%3DRunning&labelSelector=app%3Dflux: getting credentials: exec: exec: "doctl": executable file not found in $PATH
Run 'fluxctl identity --help' for usage.

Does anyone have a workaround for now?

[JVMartin]

Found a workaround. I installed fluxctl v1.14.2 and all is well:

» fluxctl version
1.14.2

» fluxctl identity --k8s-fwd-ns=flux
ssh-rsa ...

[dholbach]

Mh. Is your 1.14.2 fluxctl installed from the snap?

In other news, I pinged the Snap folks regarding classic confinement: https://forum.snapcraft.io/t/fluxctl-personal-files-was-fluxctl-snap-wants-to-be-classic/11073/27

[dholbach]

To everyone who's affected by this bug, please test the snap I just uploaded to the edge channel of fluxctl - you will need to use --classic. Please report back on #2529.