eksctl does not report error if it can not write kubeconfig file

[vkuusk]

What happened?

I launched "eksctl utils write-kubeconfig ...." in a subprocess of an automation script while that subprocess did not have permissions to write to filesystem.

Instead of reporting an error, the output of the command was:

[ℹ] using region us-west-2
[✔] saved kubeconfig as ""

Note: it just showed the filename as blank. so my first thought was that I did not pass "--kubeconfig" parameter correctly.

What you expected to happen?
I would expect that eksctl report an error instead of ( [✔] saved kubeconfig as "" )
Something like ( [✖] Permission denied. Could not write to file "provided_path_to_kubeconfig")

How to reproduce it?
It can be reproduces if one tries to write kubeconfig in a directory where user does not have write permissions. Actual problem was in Linux, but it's the same in Mac OS X
--- the following works fine:

$ eksctl utils write-kubeconfig -n eks-testing-1 --kubeconfig /tmp/kubeconfig
[ℹ]  using region us-west-2
[✔]  saved kubeconfig as "/tmp/kubeconfig"

--- the following does not work:

$ eksctl utils write-kubeconfig -n eks-testing-1 --kubeconfig /usr/bin/kubeconfig
[ℹ]  using region us-west-2
[✔]  saved kubeconfig as ""

Anything else we need to know?
I'm using eksctl inside the AWS Lambda and I'm using downloaded binary.
Lambda is assuming an appropriate IAM role which has "eks:*" in it's policy,
So "eksctl cluster create ...." works fine.

Versions
Please paste in the output of these commands:

$ eksctl version
[ℹ]  version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.5.3"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Logs
Include the output of the command line when running eksctl. If possible, eksctl should be run with debug logs.

$ eksctl utils write-kubeconfig -n eks-testing-1 --kubeconfig /usr/bin/kubeconfig -v 4
2019-10-04T09:29:13-07:00 [ℹ]  using region us-west-2
2019-10-04T09:29:14-07:00 [▶]  role ARN for the current session is "arn:aws:sts::XXXXXXXXXXXX:assumed-role/Admins/XXXXXXXXXXXXX"
2019-10-04T09:29:14-07:00 [▶]  cluster = {
  Arn: "arn:aws:eks:us-west-2:XXXXXXXXXXXX:cluster/eks-testing-1",
  CertificateAuthority: {
    Data: "XXXXXXXXXXXXX"
  },
  CreatedAt: 2019-10-03 16:38:54 +0000 UTC,
  Endpoint: "https://XXXXXXXXXXXXX.yl4.us-west-2.eks.amazonaws.com",
  Identity: {
    Oidc: {
      Issuer: "https://oidc.eks.us-west-2.amazonaws.com/id/XXXXXXXXXXXXXXX"
    }
  },
  Logging: {
    ClusterLogging: [{
        Enabled: false,
        Types: [
          "api",
          "audit",
          "authenticator",
          "controllerManager",
          "scheduler"
        ]
      }]
  },
  Name: "eks-testing-1",
  PlatformVersion: "eks.4",
  ResourcesVpcConfig: {
    EndpointPrivateAccess: false,
    EndpointPublicAccess: true,
    SecurityGroupIds: ["sg-XXXXXXXXXXXXXXX"],
    SubnetIds: [
      "subnet-XXXXXXXXXXXXXXX",
      "subnet-XXXXXXXXXXXXXXX",
      "subnet-XXXXXXXXXXXXXXX",
      "subnet-XXXXXXXXXXXXXXX",
      "subnet-XXXXXXXXXXXXXXX"
    ],
    VpcId: "vpc-XXXXXXXXXXXXXXX"
  },
  RoleArn: "arn:aws:iam::XXXXXXXXXXXXXXX:role/eksctl-eks-testing-1-cluster-ServiceRole-XXXXXXXXXXXXXXX",
  Status: "ACTIVE",
  Version: "1.13"
}
2019-10-04T09:29:14-07:00 [▶]  merging kubeconfig files
2019-10-04T09:29:14-07:00 [▶]  setting current-context to [email protected]
2019-10-04T09:29:14-07:00 [✔]  saved kubeconfig as ""

[vkuusk]

And just to add one more detail:

When running as subprocess inside AWS Lambda, other binaries can write to /tmp, and only "eksctl" can not.

In python I'm executing different commands as:

subprocess.run(get_kubeconfig_cmd, shell=True, stdout=subprocess.PIPE)

kubeconfig_file_name = '/tmp/kubeconfig'

The following works (both "echo" and simple python script) :

get_kubeconfig_cmd = ['/bin/echo testing > {0}'.format(kubeconfig_file_name)]
get_kubeconfig_cmd = ['/var/task/test-write.py -f {0}'.format(kubeconfig_file_name)]

But the eksctl command (binary compiled from Go) can not write to file:

get_kubeconfig_cmd = ['/var/task/eksctl.{0} utils write-kubeconfig -n {1} --kubeconfig {2}'.format(os_type, cluster_name, kubeconfig_file_name)]

[sayboras]

The issue with file permission is due to the below line, in which the error is not bubble up to caller
https://github.com/weaveworks/eksctl/blob/079c5b71ee90b9fcfa6d4c10f07e81a2c8fc0ac0/pkg/utils/kubeconfig/kubeconfig.go#L152

Please confirm again you can run above python snippet with /tmp/kubeconfig again, I tried but it is working as expected for me.

>>> subprocess.run(get_kubeconfig_cmd, shell=True, stdout=subprocess.PIPE)

CompletedProcess(args=['./eksctl utils write-kubeconfig -n sandbox --kubeconfig /tmp/testing'], returncode=0, stdout=b'[\xe2\x84\xb9]  using region ap-southeast-2\n[\xe2\x9c\x94]  saved kubeconfig as "/tmp/testing"\n')

[vkuusk]

@sayboras

I can write to /tmp when executing locally on my machine
AND
I can not write to the /tmp when executing inside AWS Lambda.

As far as I understand there are 2 issues here:

1. There is no error message when one tries to create kubeconfig into directory where one does not have permissions to write.
   To reproduce: run the following command as a regular user on MacOS or on Linux:

eksctl utils write-kubeconfig -n eks-testing-1 --kubeconfig /usr/bin/kubeconfig

Or into any other directory where you do not have permission to write.
This is related to the line in Go code which you showed.

1. Why eksctl can not write into /tmp inside AWS lambda, while other processes called the same way as subprocess can?
   Inside the AWS Lambda function I can create a child process with "subprocess" and child process can write to /tmp. But if I call eksctl inside "subprocess" it can not write.

--- So if the #1 is resolved, then at least we can catch this error inside Lambda.
--- And #2 needs more research/clarification why it's happening. ( It is reproducible even running AWS Lambda locally on your machine using SAM ( https://aws.amazon.com/serverless/sam/ )

sam local invoke --event ../test/eksctlmgr-test-event.json eksctlmgr

where eksctlmgr is the lambda I mentioned above.

[sayboras]

Point 2 seems to be related to AWS/SAM local setup or configuration. I don't think there is anything we can do from eksctl side.

Might need inputs from others as well. @martina-if @cPu1

[vkuusk]

About Point 2: Yes. it's most likely question to AWS lambda setup.
I'm working with AWS support to identify the reason why eksctl's behavior might be different inside the lambda function.

[vkuusk]

@sayboras One more update:

--- I added infinite loop inside Lambda handler function and executed locally using sam:

$ sam local invoke --event ../test/eksctlmgr-test-event.json eksctlmgr

Then logged into container

$ docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS               NAMES
19d8bb323414        lambci/lambda:python3.7   "/var/rapid/init --b…"   10 seconds ago      Up 10 seconds                           adoring_hofstadter

$ docker exec -ti 19d8 /bin/bash

and did the following:

bash-4.2$ ./eksctl.linux_amd64 get cluster -n eks-testing-1
NAME        VERSION STATUS  CREATED         VPC         SUBNETS                 SECURITYGROUPS
eks-testing-1   1.13    ACTIVE  2019-10-09T21:41:06Z    vpc-0ed513683df7f8d15   subnet-02eab9e9b5fe86222,subnet-04679f7ef97b79d03,subnet-050d9ddd59939d923,subnet-0cd8e12f7a6daa309,subnet-0d850cf8b7010c98a    sg-03b91cf7cc136aedc
bash-4.2$ whoami
sbx_user1051
bash-4.2$ ls -la /tmp
total 12
drwx------ 1 sbx_user1051  495 4096 Oct 10 17:53 .
drwxr-xr-x 1 root         root 4096 Oct 10 17:53 ..
-rw-r--r-- 1 sbx_user1051  495 2266 Oct 10 17:53 eksctl-config.yml
bash-4.2$ echo "Write something" > /tmp/testingfile.txt
bash-4.2$ cat /tmp/testingfile.txt 
Write something
bash-4.2$ ./eksctl.linux_amd64 utils write-kubeconfig -n eks-testing-1 --kubeconfig /tmp/kubeconfig
[ℹ]  using region us-west-2
[✔]  saved kubeconfig as ""
bash-4.2$ ls -la /tmp
total 16
drwx------ 1 sbx_user1051  495 4096 Oct 10 17:55 .
drwxr-xr-x 1 root         root 4096 Oct 10 17:53 ..
-rw-r--r-- 1 sbx_user1051  495 2266 Oct 10 17:53 eksctl-config.yml
-rw-r--r-- 1 sbx_user1051  495   16 Oct 10 17:55 testingfile.txt
bash-4.2$

As you can see:
a) user can write to /tmp
b) eksctl works and talks to EKS
c) "eksctl utils" CAN NOT write to /tmp

:-) Now the question is "What is so special about system calls which eksctl uses to write to a file?
And how can I debug this?"
Any suggestions would be appreciated.

And: I'm continuing to work with AWS support.

[vkuusk]

One more fact, which might help to identify potential issue in eksctl:

--- Inside Lambda using default file name for kubeconfig works !!! if set $HOME to /tmp:

bash-4.2$ export HOME=/tmp
bash-4.2$ ./eksctl.linux_amd64 utils write-kubeconfig -n eks-testing-1
[ℹ]  using region us-west-2
[✔]  saved kubeconfig as "/tmp/.kube/config"
bash-4.2$ cat /tmp/.kube/config 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <deleted>
    server: https://<deleted>.gr7.us-west-2.eks.amazonaws.com
  name: eks-testing-1.us-west-2.eksctl.io
contexts:
- context:
    cluster: eks-testing-1.us-west-2.eksctl.io
    user: botocore-session-<deleted>@eks-testing-1.us-west-2.eksctl.io
  name: botocore-session-<deleted>@eks-testing-1.us-west-2.eksctl.io
current-context: botocore-session-<deleted>@eks-testing-1.us-west-2.eksctl.io
kind: Config
preferences: {}
users:
- name: botocore-session-<deleted>@eks-testing-1.us-west-2.eksctl.io
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - eks-testing-1
      command: aws-iam-authenticator
      env: null

[sayboras]

the PR https://github.com/weaveworks/eksctl/pull/1406 has been merged, it will not fix issue with /tmp permission. However, can you help to take latest eksctl from master and try to run again ? Just want to see the actually error message.