salt-cloud scaleway driver API URLs (api_root setting) missing in the docs

[ek9]

Description of Issue/Question

salt-cloud scaleway driver has outdated API URLs which render it unusable. Most requests to API do not return any data and the control of instances seems impossible. It also does not give any error messages as the old API URLs use normal (200) status codes.

salt-cloud seems to use the old URL (https://api.cloud.online.net) when making API calls while scaleway API has been updated to use different URLs, which are based on the region where you server is hosted at. An excerpt from scaleway's documentation regarding API Endpoints:

Request and response

Our API works over https and is accessed from several endpoints:

cp-<region>.scaleway.com for Compute Power related actions, where <region> is par1 or ams1.

Source: https://developer.scaleway.com/#header-request-and-response

Setup

contents of /etc/salt/cloud:

cloud:
  providers:
    my-scaleway:
      access_key: REPLACE_WITH_SCALEWAY_ACCESS_KEY
      token: REPLACE_WITH_SCALEWAY_TOKEN
      driver: scaleway

  profiles:
    scaleway-archlinux:
      provider: my-scaleway
      image: Arch Linux

Steps to Reproduce Issue

1. I have an instance running on Scaleway and provider configured.
2. I run salt-cloud -l debug -p my-scaleway -F and get empty results:

[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Missing configuration file: /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: REDACTED
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy: 'saltify.destroy' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_sizes: 'saltify.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_images: 'saltify.avail_images' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_locations: 'saltify.avail_locations' is not available.
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Could not LazyLoad scaleway.optimize_providers: 'scaleway.optimize_providers' is not available.
[DEBUG   ] The 'scaleway' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy: 'saltify.destroy' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_sizes: 'saltify.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_images: 'saltify.avail_images' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_locations: 'saltify.avail_locations' is not available.
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] https://api.cloud.online.net/servers/
[DEBUG   ] LazyLoaded nested.output

1. As there is no output (and no error), I notice the printed old API URL, which I decide to change in salt/cloud/clouds/scaleway.py file, line 343 from:

default='https://api.cloud.online.net'

to:

default='https://cp-ams1.scaleway.com'

(this change is due to my server hosted in AMS1. Should it be in different location, I'd have to change it to something else).

1. After this change, I recompile the file and re-run salt-cloud -l debug -p my-scaleway -F again (it results the servers that are running correctly):

[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Missing configuration file: /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: REDACTED
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy: 'saltify.destroy' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_sizes: 'saltify.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_images: 'saltify.avail_images' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_locations: 'saltify.avail_locations' is not available.
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Could not LazyLoad scaleway.optimize_providers: 'scaleway.optimize_providers' is not available.
[DEBUG   ] The 'scaleway' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy: 'saltify.destroy' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_sizes: 'saltify.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_images: 'saltify.avail_images' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_locations: 'saltify.avail_locations' is not available.
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] https://cp-ams1.scaleway.com/servers/
[DEBUG   ] LazyLoaded nested.output
my-scaleway:
    ----------
    scaleway:
        ----------
        REDACTED:
            ----------
            arch:
                x86_64
            bootscript:
                ----------
                architecture:
                    x86_64

        REST OF OUTPUT IS REDACTED

Versions Report

Salt Version:
Salt: 2016.11.1

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.13 (default, Dec 21 2016, 07:16:46)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.2
ZMQ: 4.1.6

System Versions:
dist:
machine: x86_64
release: 4.8.15-2-ck-ivybridge
system: Linux
version: Not Installed

salt-cloud v0.8.11

[Ch3LL]

Looks like this was updated here: https://github.com/saltstack/salt/pull/31883

ping @paiou looks like you changed the api url to api.cloud.online.net

Since i have no knowledge in this area was there a particular reason you changed it to api.cloud.onlin.net from https://api.scaleway.com?

According to this you can indeed use this url so I am confused to which is the correct url?

[ek9]

@Ch3LL the #31883 might have been related to another update to Scaleway API. Scaleway has grown and expanded during the last year a lot and they sure had API changes too. I will wait for paiou to answer on this tho.

According to this you can indeed use this url so I am confused to which is the correct url?
https://www.scaleway.com/docs/interact-with-the-compute-api/

To answer your question regarding the resource you linked in: I was not aware of that, but I guess it might be a bit outdated. I have images running on scaleway, and I always get empty list of running images if I query the api.cloud.online.net.

Chaging it to api.scaleway.com won't work either, as you need to form query based on the region. Currently there are two possible regions (this might change in the future, so I believe the region must be settable via cloud providers configuration)

cp-<region>.scaleway.com for Compute Power related actions, where is par1 or ams1.

• Endpoint for par1 region (Servers in Paris): cp-par1.scaleway.com
• Endpoint for ams region (Server in Amsterdam): cp-ams1.scaleway.com

I believe you should only hardcode cp-*.scaleway.com in salt and allow specifying region in the cloud provider configuration file (/etc/salt/cloud.providers):

my-scaleway-provider:                                                                   
    access_key: ACCESS_KEY_HERE                              
    token: TOKEN_HERE                                
    driver: scaleway
    region: par1 # or ams1

My 0.02 and thanks for your reply!

[paiou]

At the time I did the update to handle C2 servers, api.scaleway.com was not working, so I changed to a working URL I found in a doc :-)
As you said @ek9, Scaleway has grown a lot and probably it had many impacts on the API. Launch of C2 servers was one of these big changes, opening of Amsterdam datacenter is another big one coming with breaking changes on the API.

I didn't created servers for long, so didn't notice Salt cloud was not working anymore, but for sure a fix is needed...

[Ch3LL]

Perfect thanks for your guys feedback on this! We will need to get this fixed this up then :)

[paiou]

I'm quite overloaded these days, but I'll try to have a look and submit a pull request if the fix is quick

[ek9]

I can try to dig into this, but I have very little experience with python so I am not sure how one would implement such parameter handlig (e.g. adding new region: parameter handling to /etc/salt/cloud.providers.

If you could give me any pointers (or at least submit initial changes to handle that), I could help you with the rest (doc changes etc.).

Feel free to ping me on the issue if I can be of any help.

[techhat]

@edouardb can you comment on this?

[paiou]

I looked into the code, and indeed the API endpoint is not hardcoded, it just has a default value.
So you can set api_root parameter to the proper url for Paris or Amsterdam in scaleway-config (where you have set your Scaleway token), and it should work just fine.

[ek9]

I have confirmed this to be working (setting api_root). Thank you and sorry for wasting your time.

I will leave this issue open with a suggestion to update documentation to even mention the api_root parameter, as it is nowhere mentioned in the documentation for users to find:

https://cse.google.com/cse?cx=004624818632696854117:yfmprrbw3pk&q=&q=%22api_root%22#gsc.tab=0&gsc.q=%22api_root%22&gsc.page=1

(that is custom search function at docs.saltstack.com website)

Please close it as appropriate, thanks.

[goten4]

Setting api_root to cp-REGION.scaleway.com didn't really solved the problem for me, because /images returns all versions of common images with pagination. For example, debian images are on the 2nd page, so it's not accepted by salt-cloud.

To have a better list of images, scaleway has just said to me to use instead https://api-marketplace.scaleway.com/images

The images are formatted in a different way (versions are nested in the top image object). There are a pagination too, but for the moment all images are on page 1.

To iterate on pages, you have to use the header Link :

$ curl -IsH 'X-Auth-Token: MYTOKEN' -H 'Content-Type: application/json' https://api-marketplace.scaleway.com/images
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 03 Feb 2017 16:14:27 GMT
Content-Type: application/json
Content-Length: 50486
Connection: keep-alive
X-Total-Count: 49
Link: </images?page=1&per_page=50&>; rel="last"
Strict-Transport-Security: max-age=15552000

The problem is that salt-cloud failed to decode the json response :

[DEBUG   ] https://api-marketplace.scaleway.com/images/
[ERROR   ] Failed to get the output of 'scaleway.avail_images()': 'ascii' codec can't encode character u'\u2019' in position 50: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 874, in image_list
    data[alias][driver] = self.clouds[fun]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/scaleway.py", line 104, in avail_images
    ret[image['id']][item] = str(image[item])
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 50: ordinal not in range(128)

[goten4]

You should use : github.com/scaleway/scaleway-cli/pkg/api (https://github.com/scaleway/scaleway-cli/blob/master/pkg/api/api.go)
The integration and maintenance would be easier ?

[ek9]

Setting api_root to cp-REGION.scaleway.com didn't really solved the problem for me, because /images returns all versions of common images with pagination. For example, debian images are on the 2nd page, so it's not accepted by salt-cloud.

The /images also pulls up a list of any custom images that can be created on your account from snapshots. Looking at scaleway control, they also to me seem like they are also based on location, Thus, you would most of the time want to list images on the location.

I have created the VMs on scaleway with SaltCloud successfully after I set api_root. I agree that image output should handle pagination, but that's completely different issue. If you pass a valid image ID, salt-cloud will work.

To have a better list of images, scaleway has just said to me to use instead https://api-marketplace.scaleway.com/images
You should use : github.com/scaleway/scaleway-cli/pkg/api (https://github.com/scaleway/scaleway-cli/blob/master/pkg/api/api.go)
The integration and maintenance would be easier ?

I believe this is quite separate issue / discussion. However, I doubt this link will list any images you have on your account (created from volumes / snapshots). This issue was originally started as I couldn't communicate with Scaleway. Setting api_root makes it possible. I've only left this issue open to mention this in the docs. I feel like this does discussion should probably find a place on a separate issue instead.

[Ch3LL]

Thanks for everyones work on this i will change this to a documentation issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[rjeschmi]

I've started working on cleaning up the scaleway module with some information from here. I'll PR the changes, but I need to test all the functionality.