Python: The kubeconfig loader should run refresh command to update token when it is expired

[nhumrich]

I am using google container engine, and trying to use this to access the k8s api. Trying to follow the example on the readme

from kubernetes import client, config

config.load_kube_config()
api = client.CoreV1Api()
pods = api.list_pod_for_all_namespaces(watch=False)


for p in pods.items:
    print(p.metadata.name, p.status.phase)

which gives me the following error:

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 6, in <module>
    config.load_kube_config()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 311, in load_kube_config
    client_configuration=client_configuration).load_and_set()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 216, in load_and_set
    self._load_authentication()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 155, in _load_authentication
    if self._load_gcp_token():
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 171, in _load_gcp_token
    self.token = "Bearer %s" % self._get_google_credentials()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 124, in <lambda>
    GoogleCredentials.get_application_default()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1271, in get_application_default
    return GoogleCredentials._get_implicit_credentials()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials
    raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

If I add the GOOGLE_APPLICATION_CREDENTIALS env-var and download a google json credential file, I then get a generic 401.

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module>
    pods = api.list_pod_for_all_namespaces(watch=False)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:25:24 GMT', 'Content-Length': '13'})
HTTP response body: Unauthorized

If I try to add an api key (client.configuration.api_key['authorization'] = 'AbX.....SYh' I get another error.

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module>
    pods = api.list_pod_for_all_namespaces(watch=False)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:40:29 GMT', 'Content-Length': '119'})
HTTP response body: User "system:anonymous" cannot list pods at the cluster scope.: "No policy matched.\nUnknown user \"system:anonymous\""

Are there any examples of how I authenticate with kubernetes/google container engine so that I can get this working?

Note: one possible solution is to run gcloud auth application-default login but that isn't automated and only works locally.

[jonathan-kosgei]

Hi Nick, seems you're trying to use gcloud credentials to auth to the Kubernetes cluster. The two are very different, instead of using a gce serviceaccount as the credential source in load_config you need to use your kube config file. If you can run kubectl commands locally then you have one under ~/.kube/config

---- On Tue, 23 May 2017 00:43:06 +0300 [email protected] wrote ----

I am using google container engine, and trying to use this to access the k8s api. Trying to follow the example on the readme

from kubernetes import client, config

config.load_kube_config()
api = client.CoreV1Api()
pods = api.list_pod_for_all_namespaces(watch=False)

for p in pods.items:
print(p.metadata.name, p.status.phase)
which gives me the following error:

Traceback (most recent call last):
File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 6, in
config.load_kube_config()
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 311, in load_kube_config
client_configuration=client_configuration).load_and_set()
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 216, in load_and_set
self._load_authentication()
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 155, in _load_authentication
if self._load_gcp_token():
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 171, in _load_gcp_token
self.token = "Bearer %s" % self._get_google_credentials()
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 124, in
GoogleCredentials.get_application_default()
File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1271, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials
raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
If I add the GOOGLE_APPLICATION_CREDENTIALS env-var and download a google json credential file, I then get a generic 401.

Traceback (most recent call last):
File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in
pods = api.list_pod_for_all_namespaces(watch=False)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
(data) = self.list_pod_for_all_namespaces_with_http_info(kwargs)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
_request_timeout=_request_timeout)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
headers=headers)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
query_params=query_params)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:25:24 GMT', 'Content-Length': '13'})
HTTP response body: Unauthorized
If I try to add an api key (client.configuration.api_key['authorization'] = 'AIz.....SYU' I get another error. ```
Traceback (most recent call last):
File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in
pods = api.list_pod_for_all_namespaces(watch=False)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
(data) = self.list_pod_for_all_namespaces_with_http_info(kwargs)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
_request_timeout=_request_timeout)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
headers=headers)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
query_params=query_params)
File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:40:29 GMT', 'Content-Length': '119'})
HTTP response body: User "system:anonymous" cannot list pods at the cluster scope.: "No policy matched.\nUnknown user "system:anonymous""

Are there any examples of how I authenticate with kubernetes/google container engine so that I can get this working?

Note: one possible solution is to run gcloud auth application-default login but that isn't automated and only works locally.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[nhumrich]

I tried doing that which is the first option. I also tried passing in the config file location, but that doesnt change anything, as that is the default anyways.

[mbohlool]

Does kubectl works? if yes, can you provide content of ~/.kube/config file (remove any key/token/etc. from the file, only "current context" should be enough. e.g. if your current-context is "x" provide user "x" and cluster "x".

[nhumrich]

Here is my kube config file

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...Something goes here...
    server: https://some.url
  name: gke_infastructure-111111_us-central1-a_my-cluster
contexts:
- context:
    cluster: gke_infastructure-111111_us-central1-a_my-cluster
    user: gke_infastructure-111111_us-central1-a_my-cluster
  name: gke_infastructure-111111_us-central1-a_my-cluster
current-context: gke_infastructure-111111_us-central1-a_my-cluster
kind: Config
preferences: {}
users:
- name: gke_infastructure-111111_us-central1-a_my-cluster
  user:
    auth-provider:
      config:
        access-token: zzzzzz.some-token-here.zzzzz
        cmd-args: config config-helper --format=json
        cmd-path: /opt/google-cloud-sdk/bin/gcloud
        expiry: 2017-05-24T19:57:52Z
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

[rafitadiaz]

+1 On this one. We are having the same problem. kubectl is working fine, but we get the 401 when using the library. And our config file look like this one above.

Anyone can send an example of how to do it properly or is it impossible make it works with GKE clusters?

[mbohlool]

The kubeconfig loader should run refresh command to update token when it is expired. To solve this, we can find the corresponding code in go client (which kubectl uses) and port it to python. Added it to milestone to make sure it will be looked at for the next release.

[zweizeichen]

Just ran into this issue myself. You can use this dirty hack to get auth on GKE in the meantime. Please note that the token retrieved will expire within a rather short timeframe. This should not be used in production.

In kube_config.py add these imports:

import subprocess
import json

modify KubeConfigLoader's __init__ method like this:

if get_google_credentials:
        self._get_google_credentials = get_google_credentials
else:
        self._get_google_credentials = lambda: (
            self._get_gcp_cmd_credentials()
        )

and add the following method to the class:

def _get_gcp_cmd_credentials(self):
    cmd_path = self._user.value['auth-provider']['config']['cmd-path']
    cmd_args = self._user.value['auth-provider']['config']['cmd-args'].split(' ')
    output = subprocess.run([cmd_path]+cmd_args, stdout=subprocess.PIPE, check=False).stdout.decode('utf-8')

    return json.loads(output)['credential']['access_token']

Done!

[nhumrich]

Update:
I was able to work around this issue by creating a serviceaccount in kubernetes.

I then ran kubectl describe serviceaccount myserviceaccount and that will give you a secret name, then use that secret name to run: kubectl describe secrets [secret-name] and then copy the token field. One you have the token field, all you need to do is set the api token in the client:

config.load_kube_config()
client.configuration.api_key['authorization'] = 'your token goes here'
client.configuration.api_key_prefix['authorization'] = 'Bearer'

This worked great for me. If you dont want to use the kube config file at all, you can also set the host and cert yourself:

client.configuration.api_key['authorization'] = 'your token goes here'
client.configuration.api_key_prefix['authorization'] = 'Bearer'
client.configuration.host = 'https://some.domain-or-ip.example'
client.configuration.ssl_ca_cert = 'cert/location.crt'

[zweizeichen]

Hah I'm using the API to set up the service accounts - so it is a kind of chicken and egg problem. Of course using a service account is the preferred and more durable solution.

[nhumrich]

@zweizeichen you could always use kubectl to create a single serviceaccount, then go from there.

[jeremad]

I am still experiencing that issue (version 5.0.0 on mac)

[OraserZhang]

I am still experiencing that issue (version 5.0.0 on mac)

[richerlariviere]

I am still experiencing that issue (version 6.0.0 on mac and I also tried HEAD version) in Kubernetes 1.10.1

[efim-a-efim]

Same issue here - version 6.0.0 on Windows/WSL (Ubuntu). kubernetes 1.10.5

[tomplus]

The current implementation refreshes a token when configuration is loaded. It doesn't refresh it for long living applications. PTAL https://github.com/kubernetes-client/python-base/issues/59

[mooperd]

+1 I'm confused why the library isn't able to use the Kubeconfig properly.

[tellet]

I'm facing this issue with version 7.0.1 (on mac), kubernetes 1.10.11. Workarounds don't help.