Python: [8.0.0] connect_get_namespaced_pod_exec return 403 despite `kubectl exec` working in shell

[ericqt]

This is the traceback for calling connect_get_namespaced_pod_exec

Listing pods with their IPs:
10.70.1.112 staging-integrations    backend-worker-celery-685f8fddc9-47sfv
Traceback (most recent call last):
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 72, in __init__
    self.sock.connect(url, header=header)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/websocket/_core.py", line 223, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/websocket/_handshake.py", line 79, in handshake
    status, resp = _get_resp_headers(sock)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/websocket/_handshake.py", line 152, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message)
websocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kube_call.py", line 38, in <module>
    main()
  File "kube_call.py", line 31, in main
    stdout=True, tty=False
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/stream/stream.py", line 32, in stream
    return func(*args, **kwargs)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 835, in connect_get_namespaced_pod_exec
    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 935, in connect_get_namespaced_pod_exec_with_http_info
    collection_formats=collection_formats)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/stream/stream.py", line 27, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File "/Users/eric.tan/miniconda3/envs/pykube/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 255, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 403 Forbidden

This is my code:

from kubernetes import config
from kubernetes.client.apis import core_v1_api
from kubernetes.client import Configuration
from kubernetes.stream import stream


def main():
    # Configs can be set in Configuration class directly or using helper utility
    config.load_kube_config("/Users/default/.kube/test.yml")
    c = Configuration()
    c.assert_hostname = False
    Configuration.set_default(c)
    api = core_v1_api.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = api.list_namespaced_pod(
        namespace='staging-integrations',
        watch=False,
        label_selector='app=backend-worker-celery'
    )
    for i in ret.items:
        print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

    pod_name = ret.items[0].metadata.name
    exec_command = ['/bin/sh', 'echo hello world']
    resp = stream(
        api.connect_get_namespaced_pod_exec,
        name=pod_name,
        namespace='staging-integrations',
        command=exec_command,
        stderr=True, stdin=False,
        stdout=True, tty=False
    )

    print("Response: ", resp)


if __name__ == '__main__':
    main()

I have used kubectl exec backend-worker-celery-685f8fddc9-47sfv echo hello world and that works perfectly fine with the same kubeconfig.

I've tried using this code with an elevated permissioned kubeconfig and that works but I do not want to go that route. Is there some other api call that connect_get_namespaced_pod_exec is calling that requires the elevated permissions?

[delwaterman]

I am getting this same error

[joshua-xia]

I met the same error:

stdout=True, tty=False)

File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/stream/stream.py", line 32, in stream
return func(args, *kwargs)
File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 835, in connect_get_namespaced_pod_exec
(data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, *kwargs)
File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 935, in connect_get_namespaced_pod_exec_with_http_info
collection_formats=collection_formats)
File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/stream/stream.py", line 27, in _intercept_request_call
return ws_client.websocket_call(config, *args, *kwargs)
File "/usr/local/python/python-3.6.5/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 255, in websocket_call
raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 403 Forbidden

[delwaterman]

Ok I found a work around for this issue. When you create the role to grant this permission I had thought that all I needed was this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
rules:
  # ...
  - apiGroups:
      - ""
    resources:
      - pods/exec
    verbs:
      - create

but it turns out you need to add both create and get. This WILL work:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
rules:
  # ...
  - apiGroups:
      - ""
    resources:
      - pods/exec
    verbs:
      - create
      - get

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.