Salt: SSLError Trying to use v3 API of Openstack Newton as provider.

[dxiri]

Description of Issue/Question

Trying to use salt-cloud with v3 API of Openstack Newton. I am unable to get any details from Openstack after correctly configuring the provider. I have made sure username and password are correct inside the file, but https handshake always fails and it seems that its impossible to get past that part.

Setup

Running salt-cloud, trying to query the VMs already in Openstack Newton. Provider config follows:
my-openstack-config:
identity_url: 'https://172.25.190.253:5000/v3/auth/tokens'
auth_version: 3
compute_name : nova
compute_region: RegionOne
service_type : compute
tenant: admin
user: admin
password: passwordgoeshere
driver: openstack

Steps to Reproduce Issue

1- configure an openstack provider using v3 of the API and use https url for the identity_url eg.
identity_url: 'https://172.25.190.253:5000/v3/auth/tokens'
2- Try any salt-cloud command relative to this provider, for example:
salt-cloud -l debug -Q
3- The above command will always return SSLError: ('[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)',) doesn't matter if you add "verify_ssl: False" or "insecure: true" to the provider config.

Versions Report

Salt Version:
Salt: 2016.3.4

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.12 (default, Jul 1 2016, 15:12:24)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-43-generic
system: Linux
version: Ubuntu 16.04 xenial

[dxiri]

I even tried Disabling SSL certificate validation straight from libcloud (which is what the openstack driver uses) and got the same result, here is what I did on a python shell:

import libcloud.security
import subprocess
libcloud.security.VERIFY_SSL_CERT = False
subprocess.call(['salt-cloud', '-l', 'debug', '-Q'])
[DEBUG ] Reading configuration from /etc/salt/cloud
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG ] Including configuration from '/etc/salt/cloud.providers.d/openstack.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.providers.d/openstack.conf
[DEBUG ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/openstack_profile.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/openstack_profile.conf
[DEBUG ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO ] salt-cloud starting
[WARNING ] /usr/lib/python2.7/dist-packages/salt/cloud/clouds/openstack.py:226: DeprecationWarning: This driver has been deprecated and will be removed in the Carbon release of Salt. Please use the nova driver instead.

[DEBUG ] Could not LazyLoad parallels.avail_sizes
[DEBUG ] LazyLoaded parallels.avail_locations
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] Could not LazyLoad saltify.destroy
[DEBUG ] Could not LazyLoad saltify.avail_sizes
[DEBUG ] Could not LazyLoad saltify.avail_images
[DEBUG ] Could not LazyLoad saltify.avail_locations
[DEBUG ] LazyLoaded rackspace.reboot
[DEBUG ] Could not LazyLoad openstack.list_locations
[DEBUG ] LazyLoaded rackspace.list_locations
[DEBUG ] Could not LazyLoad openstack.optimize_providers
[DEBUG ] The 'openstack' cloud driver is unable to be optimized.
[DEBUG ] Could not LazyLoad parallels.avail_sizes
[DEBUG ] LazyLoaded parallels.avail_locations
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] Could not LazyLoad saltify.destroy
[DEBUG ] Could not LazyLoad saltify.avail_sizes
[DEBUG ] Could not LazyLoad saltify.avail_images
[DEBUG ] Could not LazyLoad saltify.avail_locations
[DEBUG ] LazyLoaded rackspace.reboot
[DEBUG ] Could not LazyLoad openstack.list_locations
[DEBUG ] LazyLoaded rackspace.list_locations
[DEBUG ] OpenStack authenticating using password
[DEBUG ] Failed to execute 'openstack.list_nodes()' while querying for running nodes: ('[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)',)
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2369, in run_parallel_map_providers_query
cloud.cloudsdata['fun']
File "/usr/lib/python2.7/dist-packages/salt/cloud/libcloudfuncs.py", line 448, in list_nodes
nodes = conn.list_nodes()
File "/usr/lib/python2.7/dist-packages/libcloud/compute/drivers/openstack.py", line 176, in list_nodes
self.connection.request('/servers/detail', params=params).object)
File "/usr/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 202, in request
raw=raw)
File "/usr/lib/python2.7/dist-packages/libcloud/common/base.py", line 709, in request
action = self.morph_action_hook(action)
File "/usr/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 269, in morph_action_hook
self._populate_hosts_and_request_paths()
File "/usr/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 302, in _populate_hosts_and_request_paths
osa = osa.authenticate(**kwargs) # may throw InvalidCreds
File "/usr/lib/python2.7/dist-packages/libcloud/common/openstack_identity.py", line 1017, in authenticate
method='POST')
File "/usr/lib/python2.7/dist-packages/libcloud/common/base.py", line 786, in request
raise ssl.SSLError(str(e))
SSLError: ('[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)',)
[DEBUG ] LazyLoaded nested.output
0

>

[Ch3LL]

@dxiri unfortunately i'm very fuzzy in this area.

ping @gtmanfred any chance you can take a look at this. I'm nto aware of an openstack instance that I have access to to test this out.

[dxiri]

thank you! an additional note that may be helpful. I am using haproxy, and what I did for now to overcome this was to copy the haproxy cert on my remote machine over to the machine that I am running salt-cloud on, then I execute salt-cloud like this:

SSL_CERT_FILE=/root/haproxy.pem salt-cloud -l debug -Q

This works ok, so in the meantime I set an alias on my .bashrc:
alias salt-cloud='SSL_CERT_FILE=/root/haproxy.pem salt-cloud'

now I can run stuff like salt-cloud -Q and the variable will be set everytime.

Hope this helps someone as a workaround while this bug is fixed :)

[Enquier]

so @gtmanfred #36548 has a commit for fixing Nova to work with v3 keystone auth. After digging in the documentation of Keystone I found the place where nova client inserts a ssl verify into a session.

I have committed a version of the nova.py clients that allows a "verify" variable to your provider config file here:
https://github.com/Enquier/salt/tree/nova-ssl

Using @dxiri 's example:

my-openstack-config:
identity_url: 'https://172.25.190.253:5000/v3' #v3 api automatically adds auth/tokens so removed
auth_version: 3
compute_name : nova
compute_region: RegionOne
service_type : compute
tenant: admin
user: admin
password: passwordgoeshere
verify: '/root/haproxy.pem' #also accepts True/False default is False
insecure: false
use_keystoneauth: true
driver: nova

also note that if you aren't using the 'default' domain @gtmanfred added 'project_domain_name' and 'user_domain_name' to his fixes for nova.

EDIT
Updated with nova-ssl which is compatible with the develop branch and now has "True" as a default.

[gtmanfred]

you would also need to add use_keystoneauth

I am working on getting the code from there into salt this week, just need to get the documentation done...

[Enquier]

@gtmanfred Updated my comment

[dxiri]

thanks guys, but it seems the issue was a little misunderstood...although the fix mentioned here is in fact useful, the point of the bug is that even if you ignore the ssl (by setting "verify_ssl: False" or "insecure: true") this doesn't take effect so you are stuck with having to get the ssl working by virtue of adding the .pem file (which isn't ideal in certain circumstances). Setting verify_ssl: False should make salt not care if the certificate is valid or not.

[gtmanfred]

I am working on getting my openstack environment back online, I will take a look at this once I do.

[Enquier]

According to OpenStack documentation the variable I set within the api also
accepts true false statements so setting that ssl_verify_path to false
might work for your needs. It might be worth a try.

On Monday, December 12, 2016, dxiri [e-mail] wrote:

thanks guys, but it seems the issue was a little misunderstood...although
the fix mentioned here is in fact useful, the point of the bug is that even
if you ignore the ssl (by setting "verify_ssl: False" or "insecure: true")
this doesn't take effect so you are stuck with having to get the ssl
working by virtue of adding the .pem file (which isn't ideal in certain
circumstances). Setting verify_ssl: False should make salt not care if the
certificate is valid or not.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/37824#issuecomment-266590166,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFpLT4yT1bzIWizjP_A80ZNs6uB7fl6Hks5rHdx1gaJpZM4K4uj6
.

--
-Chas

[Enquier]

http://docs.openstack.org/developer/keystoneauth/api/keystoneauth1.html#keystoneauth1.session.Session

verify – The verification arguments to pass to requests. These are of the same form as requests expects, so True or False to verify (or not) against system certificates or a path to a bundle or CA certs to check against or None for requests to attempt to locate and use certificates. (optional, defaults to True)

[Enquier]

updated link that is develop compatible there was a small conflict between develop and the original solution for keystoneauth

https://github.com/Enquier/salt/tree/nova-ssl

[gtmanfred]

@Enquier https://github.com/saltstack/salt/pull/38647 is merged now if you want to open the PR for your changes

[Enquier]

Sorry for the delay. I have added a PR and provided a short note for the documentation

[gtmanfred]

Looks like this was merged!

Thanks @Enquier