Helmfile: KUBECONFIG environment variable does not work with relative paths

[dgarbus]

Problem

While working with a new EKS cluster with a separate kubeconfig file, I noticed that helmfile does not seem to find/use the KUBECONFIG file when a relative path is specified. Strangely, the same KUBECONFIG passed into helm itself works just fine.

$ KUBECONFIG=kubeconfig ./helmfile_darwin_amd64 charts
< redacted >
Error: could not get Kubernetes config for context "aws": context "aws" does not exist
Error: could not get Kubernetes config for context "aws": context "aws" does not exist
err: exit status 1
err: exit status 1

$ KUBECONFIG=kubeconfig helm --kube-context aws --debug list
[debug] Created tunnel using local port: '63788'

[debug] SERVER: "127.0.0.1:63788"

$ echo $?
0

These do not work

KUBECONFIG=kubeconfig helmfile charts
KUBECONFIG=./kubeconfig helmfile charts

This works

KUBECONFIG=$PWD/kubeconfig helmfile charts

Environment

Helmfile version: v0.19.0 (helmfile_darwin_amd64)
Platform: OS X
Helm version: 2.9.1 (installed via Homebrew)
Kubernetes client version: 1.10.4 (installed via Homebrew)

To Reproduce

1. Create a kubeconfig file in your current working directory with a context that does not exist in your default kubeconfig file (~/.kube/config).
2. Update your helmfile to reference the context name you set in step 1: context: aws
3. Run helmfile with the relative path to your kubeconfig: KUBECONFIG=kubeconfig helmfile charts
4. Confirm that you get an error indicating that the context was not found:

Error: could not get Kubernetes config for context "aws": context "aws" does not exist

1. Run helmfile again without a relative path set for your KUBECONFIG and confirm that things work as expected: KUBECONFIG=$PWD/kubeconfig helmfile charts

[mumoshu]

@dgarbus Hey! Does it work if you upgrade your helm binary to v2.10?

I can't simply reproduce it with helm v2.10. Does it happen only with kubeconfig for an EKS cluster, by any chance? My thought is that it may have something to do with the exec authentication provider introduced since k8s v1.10.

[dgarbus]

Hi @mumoshu,

Unfortunately, it still happens with helm 2.10. In trying to see if this was something with my local workstation, I got the idea to try reproducing the issue using the official helmfile docker container. Here's what I did:

1. Created a test directory on my workstation that contained the kubeconfig file for my local minikube cluster (this should also rule out it being an EKS-related issue) and a test helmfile.
2. Ran the container image with the test directory mounted: docker run --privileged -v /path/to/my/test/dir:/test -it quay.io/roboll/helmfile:v0.22.0 bash
3. Installed strace (apk add strace) and monitored the syscalls:

cd /test

# Helmfile with a relative kubeconfig path
bash-4.4# export KUBECONFIG=minikube.kubeconfig && strace -q -y -f helmfile status 2>&1 | grep $KUBECONFIG
[pid    20] openat(AT_FDCWD, "minikube.kubeconfig", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

# Helm with the same kubeconfig path
bash-4.4# export KUBECONFIG=minikube.kubeconfig && strace -q -y -f helm status 2>&1 | grep $KUBECONFIG
[pid    98] openat(AT_FDCWD, "minikube.kubeconfig", O_RDONLY|O_CLOEXEC) = 3</test/minikube.kubeconfig>
[pid    98] epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 3</test/minikube.kubeconfig>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2642788096, u64=140194670296832}}) = 0
[pid    98] fcntl(3</test/minikube.kubeconfig>, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
[pid    98] fcntl(3</test/minikube.kubeconfig>, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE <unfinished ...>
[pid    98] fstat(3</test/minikube.kubeconfig>,  <unfinished ...>
[pid    98] read(3</test/minikube.kubeconfig>, "apiVersion: v1\nclusters:\n- clust"..., 867) = 355
[pid    98] read(3</test/minikube.kubeconfig>, "", 512) = 0
[pid    98] epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_DEL, 3</test/minikube.kubeconfig>, 0xc420647494) = 0
[pid    98] close(3</test/minikube.kubeconfig>) = 0

# Helmfile with an absolute path
bash-4.4# export KUBECONFIG=/test/minikube.kubeconfig && strace -q -y -f helmfile status 2>&1 | grep $KUBECONFIG
[pid   118] openat(AT_FDCWD, "/test/minikube.kubeconfig", O_RDONLY|O_CLOEXEC) = 3</test/minikube.kubeconfig>
[pid   118] epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 3</test/minikube.kubeconfig>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4059107072, u64=140578338705152}}) = 0
[pid   118] fcntl(3</test/minikube.kubeconfig>, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
[pid   118] fcntl(3</test/minikube.kubeconfig>, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
[pid   118] fstat(3</test/minikube.kubeconfig>, {st_mode=S_IFREG|0600, st_size=355, ...}) = 0
[pid   118] read(3</test/minikube.kubeconfig>,  <unfinished ...>
[pid   118] read(3</test/minikube.kubeconfig>,  <unfinished ...>
[pid   118] epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_DEL, 3</test/minikube.kubeconfig>, 0xc42095d494) = 0
[pid   118] close(3</test/minikube.kubeconfig>) = 0

I also noticed a chdir() syscall for helmfile that is not present with helm which I think may be responsible. Notice that it occurs within the same process and before the open call for $KUBECONFIG:

bash-4.4# export KUBECONFIG=minikube.kubeconfig && strace -q -y -f helmfile status 2>&1 | egrep "chdir|minikube"
[pid   224] chdir("/tmp/helmfile-helm-exec734186500") = 0
[pid   224] openat(AT_FDCWD, "minikube.kubeconfig", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Hope this helps!

[mumoshu]

@dgarbus Ahh. Good catch!

237 should fix the issue. Thanks again for the detailed report ☺️

[cablespaghetti]

Hi. Sorry to drag up an old bug but I think this might be back again. I upgraded from 0.40.1 to 0.45.1 and now I need an absolute path to my kubeconfig file in the KUBECONFIG environment variable or I have this problem.

Workaround for anyone coming across this was to change export KUBECONFIG=./kubeconfig to export KUBECONFIG=$(readlink -f ./kubeconfig)

[mumoshu]

@cablespaghetti Hey! Thanks for reporting.

This sounds like it's a regression/ introduced by me via #448, and you work-around sounds great 👍

In the mean time, please keep using your work-around. I'd consider how I can fix this. Since #448 helmfile actually change the cwd to that of the targeted (sub-)helmfile, helmfile should likely be improved to internally use the absolute path to KUBECONFIG.