Python: Upgrade request required when running exec command with python-kubernetes client

[genadic]

I saw this image already appeared in https://github.com/kubernetes-incubator/client-python/issues/58 and was solved, but now it's broken again in 4.0

[dulek]

I was looking at that as well, so here's some additional info:

58 was fixed by #120, but @dims code is now gone due to commit #353.

[dims]

hmmmmmmm....

[dims]

@dulek do you have some time to re-patch please? i thought i had tests in there

[dulek]

@dism: I can try. Those tests were disabled by #259.

[dulek]

410 is the commit. I was however unable to verify it as connection is now failing with "Reason: Handshake status 404".

[mpwibm]

I can run from the command line:
kubectl exec secret-pod --namespace=production cat data/busybox/password
and
kubectl exec secret-pod --namespace=production cat data/busybox/username
and they are working successfully

But when I run the following code in python
I get
"Upgrade request required"
def _get_user_token(self):
user=''
passwd=''
name = 'secret-pod'
namespace = 'production'
exec_cmd = ['cat /data/busybox/username']

    try:
        user = self.core_api_instance.connect_get_namespaced_pod_exec(name="secret-pod",
                                                                      namespace="production",
                                                                      command=["cat /data/busybox/username"],
                                                                      stderr=True, stdin=False,
                                                                      stdout=True, tty=False)
        pprint(user)
    except ApiException as e:
        print("Exception when calling CoreV1Api->get_namespaced_pod_exec: %s\n" % e)

    exec_cmd = ['cat /data/busybox/password']
    try:
        passwd = self.core_api_instance.connect_get_namespaced_pod_exec(name="secret-pod",
                                                                        namespace='production',
                                                                        command=["cat /data/busybox/password"],
                                                                        stderr=True, stdin=False,
                                                                        stdout=True, tty=False)
        pprint(passwd)
    except ApiException as e:
        print("Exception when calling CoreV1Api->get_namespaced_pod_exec: %s\n" % e)

    return {'user': user, 'password': passwd}

Full error messages
send: 'GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fusername%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1\r\nHost: 192.168.42.218:8443\r\nAccept-Encoding: identity\r\nContent-Type: application/json\r\nauthorization: Bearer heoJMu0hqlrLWNJ2lV9MsaIWp1kPuKvwPhmhcmj9-Kg\r\nAccept: /\r\nUser-Agent: Swagger-Codegen/4.0.0/python\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Cache-Control: no-store
header: Content-Type: application/json
header: Date: Mon, 30 Apr 2018 19:50:05 GMT
header: Content-Length: 139
2018-04-30 15:50:05,528 DEBUG https://192.168.42.218:8443 "GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fusername%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1" 400 139
Exception when calling CoreV1Api->get_namespaced_pod_exec: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 30 Apr 2018 19:50:05 GMT', 'Content-Length': '139', 'Content-Type': 'application/json', 'Cache-Control': 'no-store'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

send: 'GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fpassword%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1\r\nHost: 192.168.42.218:8443\r\nAccept-Encoding: identity\r\nContent-Type: application/json\r\nauthorization: Bearer heoJMu0hqlrLWNJ2lV9MsaIWp1kPuKvwPhmhcmj9-Kg\r\nAccept: /\r\nUser-Agent: Swagger-Codegen/4.0.0/python\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Cache-Control: no-store
header: Content-Type: application/json
header: Date: Mon, 30 Apr 2018 19:50:05 GMT
header: Content-Length: 139
2018-04-30 15:50:05,536 DEBUG https://192.168.42.218:8443 "GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fpassword%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1" 400 139
Exception when calling CoreV1Api->get_namespaced_pod_exec: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 30 Apr 2018 19:50:05 GMT', 'Content-Length': '139', 'Content-Type': 'application/json', 'Cache-Control': 'no-store'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

[TopherGopher]

Is there any status update on this? Has anyone figured out a workaround?
I don't mind patching this, but I don't understand why that massive pull request wiped out this logic. Was it because it was bad logic? Or was it just a merge mistake?

[TopherGopher]

I found this issue mentioned (and apparently solved) over here: https://github.com/kubernetes-client/python/issues/526

resp = kubernetes.stream.stream(self.core_api_instance.connect_get_namespaced_pod_exec, name, namespace,command=exec_command1, stderr=True, stdin=False, stdout=True, tty=False)

You can then manipulate that pipe using the technique from the example code

My issue is that I'm still getting an error message when I run it inside a pod in the cluster -

>>> resp = k.stream.stream(k8s.connect_get_namespaced_pod_attach, 'datadog-agent-8pkwh', container="datadog-agent", namespace="default", stderr=True, stdin=False, stdout=True, tty=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/kubernetes/stream/stream.py", line 32, in stream
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 711, in connect_get_namespaced_pod_attach
    (data) = self.connect_get_namespaced_pod_attach_with_http_info(name, namespace, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 808, in connect_get_namespaced_pod_attach_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/stream/stream.py", line 27, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/stream/ws_client.py", line 255, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: hostname '10.100.0.1' doesn't match either of 'ip-10-0-113-207.us-west-2.compute.internal', 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '0e23207807f30610a20a7dc08f349bad.sk1.us-west-2.eks.amazonaws.com'

I'll let you know if I find the fix ;-)

[TopherGopher]

The fix was I upgraded from Python 2 to Python 3 and the hostname message went away, and I can confirm the Upgrade message is gone. If no one objects, could we close this issue?

[scher200]

for anyone ending up here Googling: connect_get_namespaced_pod_attach "message":"Upgrade request required"

    resp = stream( api_instance.connect_get_namespaced_pod_attach, NAME, NAMESPACE, stderr=stderr, stdin=stdin, stdout=stdout, tty=tty, _preload_content=False )

read more:
https://github.com/kubernetes-client/python/blob/master/examples/exec.py

[mpwibm]

@scher200 you also need to upgrade the ssl on python to make it work, definitely you need kubernetes stream but thats one aspect you also need to upgrade SSL and on my mac its a bit complicated not sure about windows/linux

[scher200]

here it worked out of the box with linux..

[mpwibm]

the older openssl versions still cause this issue using Streams

type "openssl version"
I believe you need a version 1.X and above
the 0.98 and earlier versions fail with the same error with streams

I have
OpenSSL 1.0.2o 27 Mar 2018 and it works with streams
the 0.98 and earlier versions fail with the same error with streams

you also may want to do
"which openssl"

and you can set
--openssldir=
to directory for new openssl install

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[kebab-mai-haddi]

Even after applying all the standard configurations and examples, applying the solutions of @scher200 , I am getting a bad request.
My code:

from __future__ import print_function
import time
import kubernetes.client
from kubernetes.client.rest import ApiException
from pprint import pprint
from kubernetes import client, config, stream
stream = stream.stream

# Configure API key authorization: BearerToken
configuration = kubernetes.client.Configuration()
configuration.host = "https://mine.eks.amazonaws.com"
configuration.verify_ssl = False
configuration.api_key['authorization'] = "my_token"
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
configuration.api_key_prefix['authorization'] = 'Bearer'
configuration.debug = True

api_instance = kubernetes.client.CoreV1Api(
    kubernetes.client.ApiClient(configuration))
name = 'jupyter-aviral'  # str | name of the PodExecOptions
namespace = 'jhub'  # str | object name and auth scope, such as for teams and projects
# str | Command is the remote command to execute. argv array. Not executed within a shell. (optional)
command = 'echo "hail aviral"'
# bool | Redirect the standard error stream of the pod for this call. Defaults to true. (optional)
stderr = True
# bool | Redirect the standard input stream of the pod for this call. Defaults to false. (optional)
stdin = True
# bool | Redirect the standard output stream of the pod for this call. Defaults to true. (optional)
stdout = True
# bool | TTY if true indicates that a tty will be allocated for the exec call. Defaults to false. (optional)
tty = True

try:
    api_response = stream(api_instance.connect_post_namespaced_pod_exec(
        name, namespace, command=command, stderr=stderr, stdin=stdin, stdout=stdout, tty=tty)
    )
    pprint(api_response)
except ApiException as e:
    print("Exception when calling CoreV1Api->connect_post_namespaced_pod_exec: %s\n" % e)

The traceback is:

/usr/local/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.3) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
2019-05-28 19:34:38,092 DEBUG Starting new HTTPS connection (1): 0d6fb20cb6b6acb7fbc1b2c264d5e9a1.yl4.ap-south-1.eks.amazonaws.com:443
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
send: b'POST /api/v1/namespaces/jhub/pods/jupyter-aviral/exec?command=echo+%22hail+aviral%22&stderr=True&stdin=True&stdout=True&tty=True HTTP/1.1\r\nHost: mine.eks.amazonaws.com\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAccept: */*\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/9.0.0/python\r\nauthorization: Bearer my_api_token\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Audit-Id: 88df8863-61b1-4fe7-9d39-d0e6059ea993
header: Content-Type: application/json
header: Date: Tue, 28 May 2019 14:04:38 GMT
header: Content-Length: 139
2019-05-28 19:34:38,420 DEBUG https://mine.eks.amazonaws.com:443 "POST /api/v1/namespaces/jhub/pods/jupyter-ankita/exec?command=echo+%22hail+aviral%22&stderr=True&stdin=True&stdout=True&tty=True HTTP/1.1" 400 139
Exception when calling CoreV1Api->connect_post_namespaced_pod_exec: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '88df8863-61b1-4fe7-9d39-d0e6059ea993', 'Content-Type': 'application/json', 'Date': 'Tue, 28 May 2019 14:04:38 GMT', 'Content-Length': '139'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Pega88]

I'm having the same issue with a way more simpler setup:
curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/namespaces/devops/pods/spinnaker-spinnaker-halyard-0/exec?command=ls I am getting the same response:
`{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "Upgrade request required",
"reason": "BadRequest",
"code": 400
}`

When I just ask for my pod details, it does work - so my Auth and Cert should be fine:
curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/namespaces/devops/pods/spinnaker-spinnaker-halyard-0/log --> logs printing correctly. Would like to /reopen

[vennilashanthi]

Hi all,
I am getting the same error which @Pega88 got. Is the issue resolved ? If so, could you please let me know how to get rid of this error ?