Salt: Unable to create Digitalocean droplets using salt-cloud
Description of Issue
Digitalocean might have changed something on their side.
salt-cloud -p asx-salt-test asx-jj-test
[ERROR ] Error creating asx-jj-test on DIGITALOCEAN
The following exception was thrown when trying to run the initial deployment: An error occurred while querying DigitalOcean. HTTP Code: 400 Error: '{"id":"bad request","message":"found unpermitted parameters: ssh_interface"}'
Error: There was a profile error: Failed to deploy VM
Setup
Digitalocean provider configuration:
do:
driver: digital_ocean
personal_access_token: xxxxxxxxxxx
ssh_key_file: /etc/salt/pki/cloud/do.pem
ssh_key_names: asx-salt
script: bootstrap-salt
script_args: stable 2018.3.4
Digitalocean profile configuration:
asx-base:
provider: do
image: ubuntu-16-04-x64
location: ams3
private_networking: true
asx-salt-test:
extends: asx-base
size: s-1vcpu-2gb
tags:
- TEST
Steps to Reproduce Issue
salt-cloud -l debug -p asx-salt-test asx-jj-test
[DEBUG ] Reading configuration from /etc/salt/cloud
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: asx-salt
[DEBUG ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG ] Including configuration from '/etc/salt/cloud.providers.d/digitalocean.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.providers.d/digitalocean.conf
[DEBUG ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/digitalocean.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/digitalocean.conf
[DEBUG ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO ] salt-cloud starting
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG ] LazyLoaded parallels.avail_locations
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] Could not LazyLoad digital_ocean.optimize_providers: 'digital_ocean.optimize_providers' is not available.
[DEBUG ] The 'digital_ocean' cloud driver is unable to be optimized.
[DEBUG ] Could not LazyLoad digital_ocean.list_nodes_min: 'digital_ocean.list_nodes_min' is not available.
[DEBUG ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG ] LazyLoaded parallels.avail_locations
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG ] https://api.digitalocean.com:443 "GET /v2/droplets/?page=1&per_page=200 HTTP/1.1" 200 None
[DEBUG ] https://api.digitalocean.com/v2/droplets/?page=1&per_page=200
[DEBUG ] Generating minion keys for 'asx-jj-test'
[DEBUG ] LazyLoaded cloud.fire_event
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Sending event: tag = salt/cloud/asx-jj-test/creating; data = {u'profile': u'asx-salt-test', u'_stamp': '2019-09-01T15:18:12.006856', u'name': u'asx-jj-test', u'driver': u'digital_ocean', u'provider': u'do:digital_ocean', u'event': u'starting create'}
[INFO ] Creating Cloud VM asx-jj-test
[DEBUG ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG ] https://api.digitalocean.com:443 "GET /v2/sizes/?per_page=100 HTTP/1.1" 200 None
[DEBUG ] https://api.digitalocean.com/v2/sizes/?per_page=100
[DEBUG ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG ] https://api.digitalocean.com:443 "GET /v2/images/?page=1&per_page=200 HTTP/1.1" 200 None
[DEBUG ] https://api.digitalocean.com/v2/images/?page=1&per_page=200
[DEBUG ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG ] https://api.digitalocean.com:443 "GET /v2/regions/ HTTP/1.1" 200 None
[DEBUG ] https://api.digitalocean.com/v2/regions/
[DEBUG ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG ] https://api.digitalocean.com:443 "GET /v2/account/keys/?page=1&per_page=100 HTTP/1.1" 200 None
[DEBUG ] https://api.digitalocean.com/v2/account/keys/?page=1&per_page=100
[INFO ] ssh_interface: Setting interface for ssh to public
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Sending event: tag = salt/cloud/asx-jj-test/requesting; data = {u'event': u'requesting instance', u'_stamp': '2019-09-01T15:18:14.059552', u'ssh_keys': [u'17578289'], u'tags': [u'TEST'], u'region': u'ams3', u'ssh_interface': u'public', u'private_networking': True, u'size': u's-1vcpu-2gb', u'image': u'ubuntu-16-04-x64', u'name': u'asx-jj-test'}
[DEBUG ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG ] https://api.digitalocean.com:443 "POST /v2/droplets/ HTTP/1.1" 400 76
[ERROR ] Error creating asx-jj-test on DIGITALOCEAN
The following exception was thrown when trying to run the initial deployment: An error occurred while querying DigitalOcean. HTTP Code: 400 Error: '{"id":"bad request","message":"found unpermitted parameters: ssh_interface"}'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/digitalocean.py", line 449, in create
ret = create_node(kwargs)
File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/digitalocean.py", line 269, in create_node
node = query(method='droplets', args=args, http_method='post')
File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/digitalocean.py", line 577, in query
request.text
SaltCloudSystemExit: An error occurred while querying DigitalOcean. HTTP Code: 400 Error: '{"id":"bad request","message":"found unpermitted parameters: ssh_interface"}'
Error: There was a profile error: Failed to deploy VM
Versions Report
Salt Version:
Salt: 2018.3.4
Dependency Versions:
cffi: 1.11.5
cherrypy: 3.5.0
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: 0.2.4
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-134-generic
system: Linux
version: Ubuntu 16.04 xenial
johje349
All 12 comments
I have the exact same issue.
In DO API docs https://developers.digitalocean.com/documentation/v2/#droplets there is no ssh_interface attribute, it's only used internally by salt cloud right?
But now it's added to the request arguments for the API call, and DO rejects the request because of the unpermitted argument?
danielsarin
on 2 Sep 2019
We have the same issue here. Is there a workaround?
Salt Version:
Salt: 2018.3.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.5.3
docker-py: Not Installed
gitdb: 2.0.0
gitpython: 2.1.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.1
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.1
locale: UTF-8
machine: x86_64
release: 4.9.0-5-amd64
system: Linux
version: debian 9.1
davidgreisler
on 2 Sep 2019
We have the same issue here. Is there a workaround?
I ended up creating a server manually from the DO control panel, then installing the salt minion, editing the minion config and accepting the new key from the master. I think that's basically what salt cloud does when creating the minion, after that you can run all salt commands normally.
danielsarin
on 2 Sep 2019
We have the same issue here. Is there a workaround?
I ended up creating a server manually from the DO control panel, then installing the salt minion, editing the minion config and accepting the new key from the master. I think that's basically what salt cloud does when creating the minion, after that you can run all salt commands normally.
We create maybe 20 droplets per day so we cannot do it manually. My workaround was to comment the following line in /usr/lib/python2.7/dist-packages/salt/cloud/clouds/digitalocean.py:
kwargs['ssh_interface'] = ssh_interface
johje349
on 3 Sep 2019
I ended up creating a server manually from the DO control panel, then installing the salt minion, editing the minion config and accepting the new key from the master. I think that's basically what salt cloud does when creating the minion, after that you can run all salt commands normally.
Yeah, that works. But it becomes quite tedious if you have to start multiple servers. ;-)
The workaround from @johje349 works for us:
if ssh_interface in ['private', 'public']:
log.info("ssh_interface: Setting interface for ssh to %s", ssh_interface)
#kwargs['ssh_interface'] = ssh_interface
else:
We are using the public interface anyway (it's the default), though I suppose this might not work if you need to use the private interface:
If you need to perform the bootstrap using the local interface for droplets, this can be done by setting ssh_interface: private in your config. By default the salt-cloud script would run on the public interface however if firewall is preventing the connection to the Droplet over the public interface you might need to set this option to connect via private interface. Also, to use this feature private_networking: True must be set in the config.
davidgreisler
on 3 Sep 2019
Digitalocean have rolled back the change made on Friday. But this issue still needs to be fixed.
johje349
on 3 Sep 2019
we noticed this error over the weekend on our test suite as well for digital ocean. @johje349 can you point me to any sort of information that shows the details of what they are changing?
Ch3LL
on 3 Sep 2019
we noticed this error over the weekend on our test suite as well for digital ocean. @johje349 can you point me to any sort of information that shows the details of what they are changing?
I've had contact with their support. According to them a change was rolled out Friday to throw errors on requests with invalid parameters instead of silently accepting them. The invalid parameter sent by salt-cloud is "ssh_interface".
Now they've rolled back that change but they also said it may be implemented again in the future.
johje349
on 4 Sep 2019
thanks for the clarification. we will need to get this fixed up thanks
Ch3LL
on 4 Sep 2019
This was raised with the team at DigitalOcean. There had been a change deployed to the /v2/droplets endpoint to reject requests with extra, non-supported parameters. This was not communicated properly, and broke existing implementations. The change has been rolled back, and this should now work as it did previously. Sorry for the disruption, please let me know if you experience more issues.
andrewsomething
on 4 Sep 2019
thanks for the follow up @andrewsomething
do you know when these changes will be re-implemented so we can ensure our fix is in before then?
Ch3LL
on 6 Sep 2019
@Ch3LL We have no immediate plans to re-implement this behaviour. If we do so in the future, it will be with advanced warning (https://developers.digitalocean.com/documentation/changelog/) and proactive outreach. Sorry again for the disruption.
andrewsomething
on 6 Sep 2019
Related issues
golmaal
路
3Comments
allyunion
路
3Comments
qiushics
路
3Comments
Oloremo
路
3Comments
Arguros
路
3Comments
Most helpful comment
We create maybe 20 droplets per day so we cannot do it manually. My workaround was to comment the following line in /usr/lib/python2.7/dist-packages/salt/cloud/clouds/digitalocean.py:
kwargs['ssh_interface'] = ssh_interface