Python: SSL verification on python2.7 failing

[sebgoa]

I saw a commit flying by that said this was fixed but I get:

>>> import os
>>> config.load_kube_config(os.environ["HOME"] + '/.kube/config')
>>> v1=client.CoreV1Api()
>>> ret = v1.list_pod_for_all_namespaces(watch=False)
2016-11-23 10:26:29,195 ERROR Certificate did not match expected hostname: 192.168.99.100. Certificate: {'subjectAltName': (('DNS', 'kubernetes.default.svc.cluster.local'), ('DNS', 'kubernetes.default.svc'), ('DNS', 'kubernetes.default'), ('DNS', 'kubernetes'), ('IP Address', '192.168.99.100'), ('IP Address', '10.0.0.1')), 'notBefore': u'Nov 16 09:38:49 2016 GMT', 'serialNumber': u'02', 'notAfter': 'Nov 16 09:38:49 2017 GMT', 'version': 3L, 'subject': ((('commonName', u'minikube'),),), 'issuer': ((('commonName', u'minikubeCA'),),)}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "kubernetes/client/apis/core_v1_api.py", line 14377, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "kubernetes/client/apis/core_v1_api.py", line 14475, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "kubernetes/client/api_client.py", line 326, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "kubernetes/client/api_client.py", line 150, in __call_api
    _request_timeout=_request_timeout)
  File "kubernetes/client/api_client.py", line 349, in request
    headers=headers)
  File "kubernetes/client/rest.py", line 222, in GET
    query_params=query_params)
  File "kubernetes/client/rest.py", line 199, in request
    raise ApiException(status=0, reason=msg)
kubernetes.client.rest.ApiException: (0)
Reason: SSLError
hostname '192.168.99.100' doesn't match either of 'kubernetes.default.svc.cluster.local', 'kubernetes.default.svc', 'kubernetes.default', 'kubernetes', '192.168.99.100', '10.0.0.1'

[sebgoa]

it does work with python3.5

[mbohlool]

are you using latest urllib3? can you run setup.py install and try this again or try to manually update urllib3.

pip show urllib3
Name: urllib3
Version: 1.19.1
Summary: HTTP library with thread-safe connection pooling, file post, and more.

Also what is your python version?

$ python --version
Python 2.7.12

[mbohlool]

I figured this out. We had a patch that we removed because urllib3 is already doing the patch, however it only does that if package "ipaddress" is installed. I added that as a dependency so we should be good.

[mbohlool]

More concrete reason here: https://github.com/shazow/urllib3/blob/master/urllib3/packages/ssl_match_hostname/_implementation.py#L9

[sebgoa]

ok confirmed that it solved the issue.

[madhavmalhotra3089]

Still not working for python 2.x windows 10

[mbohlool]

https://github.com/kubernetes-incubator/client-python/blob/master/README.md#hostname-doesnt-match

[madhavmalhotra3089]

Hey

Thanks
On Sun, May 28, 2017 at 9:10 AM Mehdy Bohlool [e-mail]
wrote:

>

https://github.com/kubernetes-incubator/client-python/blob/master/README.md#hostname-doesnt-match

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/client-python/issues/36#issuecomment-304524086,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQel4SxEv6OCtfDYGMCYATi7E3UscAISks5r-ZxYgaJpZM4K6ZZl
.

>

Madhav Malhotra
Graduate Student
University of Victoria
Canada [phone]

[tahir24434]

Hi,
I followed the instructions described in this thread and I am still hitting the issue.

Exception when calling CoreV1Api->connect_post_namespaced_pod_exec: (0)
Reason: hostname '1.0.0.14' doesn't match either of 'minion-1-0-0-14', 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'

I have following versions of packages

python --version
Python 2.7.6

pip show ipaddress
Name: ipaddress
Version: 1.0.18
Summary: IPv4/IPv6 manipulation library
Home-page: https://github.com/phihag/ipaddress
Author: Philipp Hagemeister
Author-email: [email protected]
License: Python Software Foundation License
Location: /usr/local/lib/python2.7/dist-packages
Requires:

pip show urllib3
Name: urllib3
Version: 1.20
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.[email protected]
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires:

Any ideas please?
Regards

[mbohlool]

What is your kubernetes package version? what framework this is on? (os, anaconda?, etc.). Also please confirm that kubectl works.

[flaviamissi]

Same problem here:

$ python --version
Python 2.7.13

$ pip show ipaddress
Name: ipaddress
Version: 1.0.18
Summary: IPv4/IPv6 manipulation library
Home-page: https://github.com/phihag/ipaddress
Author: Philipp Hagemeister
Author-email: [email protected]
License: Python Software Foundation License
Location: $HOME/my-venv/lib/python2.7/site-packages
Requires:

$ pip show urllib3
Name: urllib3
Version: 1.22
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: [email protected]
License: MIT
Location: $HOME/my-venv/lib/python2.7/site-packages
Requires:

$ pip show kubernetes
Name: kubernetes
Version: 2.0.0
Summary: Kubernetes python client
Home-page: https://github.com/kubernetes-incubator/client-python
Author: Kubernetes
Author-email: UNKNOWN
License: Apache License Version 2.0
Location: /Users/flavia/go/src/mindoktor.io/md-venv/lib/python2.7/site-packages
Requires: websocket-client, ipaddress, setuptools, oauth2client, six, urllib3, pyyaml, python-dateutil, certifi

The actual error looks a little different, though the underlying problem seems exactly the same

c.connect_get_namespaced_pod_exec("podname", "namespace", command="/bin/bash -c 'ls'")
[...]
ApiException: (0)
Reason: hostname 'MY.IPADDR' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'

Additionally, kubectl works just fine.

[mbohlool]

You are calling exec. We use websocket library for exec. They introduced some bug and we limited the version in the dependency file (requirements.txt). If you search issues for websocket or just make sure you are using compatible websocket version, that should fix your problem.

[flaviamissi]

Thanks for the info! I have noticed that.
With websocket-client==0.44.0 I get

ApiException: (0)
Reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

The error I posted previously I got using websocket-client==0.40.0 (sorry I missed posting the info for it). Looking at some more examples I found that I needed to stop hostname assertion (doh):

from kubernetes.client import configuration
configuration.assert_hostname = True

After doing that I was able to move forward a bit, but not without another failure. Here's my full code:

from kubernetes.client import configuration
from kubernetes import config
from kubernetes.client.apis import core_v1_api

configuration.assert_hostname = False
config.load_kube_config()
c = core_v1_api.CoreV1Api()
c.connect_get_namespaced_pod_exec("mypod", "namespace", command="/bin/bash -c 'ls'")
# stacktrace...
# ApiException: (0)
# Reason: Handshake status 404

I feel like I'm missing something, appreciate the help!

[flaviamissi]

BTW I have looked at #144 and updated my snippet (as seen above), and I get the exact same issue there... I have tested on my local machine (OS X, OpenSSL 1.0.2l) and on a VM (Debian OpenSSL 1.0.1t) both share same requirements versions.

Have also done some debugging and the final url used in the websocket call is wss://my.master.ip/api/v1/namespaces/default/pods/mypod/exec. I've also made sure that mypod actually exists... although I know that 404 is a client error I'm clueless to why this is happening.

[mbohlool]

Interesting. Try kubectl (to call exec on your pod) with -v9 flag to see if it hits the same url. Also what is your cluster running on? GKE? On Prem? AWS? ...

[flaviamissi]

My cluster is on GKE, master version is 1.5.7, node version 1.4.7.

$ kubectl exec mypod ls -v 9
[...]
https://same.master.ip/api/v1/namespaces/default/pods/mypod/exec

(I've omited query strings on both urls)
So exactly same url, only different protocols.

[mbohlool]

We had a bug in gke config loader. Can you try latest beta release 3.0.0b1
to see if that fixes your problem?

On Jul 31, 2017 7:17 AM, "Flavia Missi" [e-mail] wrote:

My cluster is on GKE, master version is 1.5.7, node version 1.4.7.

$ kubectl exec mypod ls -v 9
[...]https://same.master.ip/api/v1/namespaces/default/pods/mypod/exec

(I've omited query strings on both urls)
So exactly same url, only different protocols.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/client-python/issues/36#issuecomment-319080361,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABic4HKzsku-uV9MFXEitOk6h2xPK2mAks5sTeIMgaJpZM4K6ZZl
.

[flaviamissi]

Tried updating it, didn't work...

But I found the issue somewhere else, the problem is on websocket_call:

query_params = [(key, value) for key, value in query_params if
                    key != 'command']

query_params value before the above line is [('command', 'ls')], after it, query_params is empty, and that's where my problem starts. If I don't explicitly give other arguments to connect_get_namespaced_pod_exec (like stdout or stderr), the url building will be inconsistent, my resulting url is /api/v1/namespaces/default/pods/mypod/exec&command=ls, note the lack of ? in the beginning of the query string. So giving connect_get_namespaced_pod_exec actually builds a correctly formed url, e.g: /api/v1/namespaces/default/pods/mypod/exec?stderr=True&command=ls. BAM, it works 🎆

I do believe this is a bug on your side though, since only name and namespace are the only required parameters, but using them alone will break. Shall I open an issue or will you?

[mbohlool]

Nice that you figured it out. Both/either issue and PR are welcome.

[prashantabkari]

Hi - I have hit the same issue with the following log
" raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.1.108', port=6443): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by SSLError(CertificateError("hostname '192.168.1.108' doesn't match either of 'master', 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local'",),))"

Following are my configurations
IPaddress
Metadata-Version: 2.0
Name: ipaddress
Version: 1.0.18

urllib3
Metadata-Version: 2.0
Name: urllib3
Version: 1.22

---

Metadata-Version: 1.1
Name: websocket-client
Version: 0.40.0

python version
Python 2.7.5

K8 cluster
kubectl cluster-info gives the correct info of master and kubedns running server

Is the above version of urllib3 supported?

UPDATE:
Following are the workaround suggested in https://github.com/kelproject/pykube/issues/29

1. CHange the kubeconfig file or
2. install python3.5

But looking for solutions with Python 2.7

[czz253861763]

from kubernetes.client import configuration
config.load_kube_config()
configuration.assert_hostname = False

this works fine for me

[nettyxiong]

from kubernetes import client
from kubernetes import config
from kubernetes.client.api import core_v1_api

config.load_kube_config('/root/.kube/config')
configuration = client.Configuration()
configuration.assert_hostname = False
configuration.verify_ssl = True
client.Configuration.set_default(configuration)

v1 = core_v1_api.CoreV1Api()
print("Listing pods with their IPs:")

ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

this works for me