Salt: Salt-cloud create EC2 instance but failed to deploy salt-minion, but minion-key appear in salt-master

[commutecat]

Description of Issue/Question

Problem 1 : salt-cloud create EC2 instance and accept salt-minion key from no where even no salt-minion package deployed on the instance

I use salt-cloud command to create EC2 instance using my own profile

sudo salt-cloud -p "new-app" mywebserver 

Instance successfully created, I also notice a key name of mywebserver appear inside salt-key.
But test.ping. After ssh inside the server , I notice no salt-minion deployed on mywebserver.

Problem 2 : saltify failed even after delete the minion-key(that don't know where it came from) from salt-key.

# so I delete the key of the minion , since there is no package installed.
sudo salt-key -d mywebserver

# then I try to saltify it
sudo salt-cloud -p "saltify-my-web"  mywebserver 

it just show me this kinda error

mywebserver:
    ----------
    Error:
        mywebserver  already exists under my-aws:ec2

Where my-aws:ec2 is my cloud provider . HOWEVER, there is no such entry of mywebserver inside my cloud.profiles.d

I just issue IP address and saltify just run through without issues.
e.g.

sudo salt-cloud -p "saltify-my-web"  172.16.100.22 

Versions Report

Salt Version:
            Salt: 2015.8.8.2

Dependency Versions:
 Apache Libcloud: 0.15.1
          Jinja2: 2.7.2
        M2Crypto: Not Installed
            Mako: 0.9.1
          PyYAML: 3.10
           PyZMQ: 14.0.1
          Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
            RAET: Not Installed
         Tornado: 4.2.1
             ZMQ: 4.0.4
            cffi: Not Installed
        cherrypy: Not Installed
        dateutil: 2.5.1
           gitdb: 0.5.4
       gitpython: 0.3.2 RC1
           ioflo: Not Installed
         libgit2: Not Installed
         libnacl: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.3.0
    mysql-python: 1.2.3
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
    python-gnupg: Not Installed
           smmap: 0.8.2
         timelib: Not Installed

System Versions:
            dist: Ubuntu 14.04 trusty
         machine: x86_64
         release: 3.13.0-74-generic
          system: Ubuntu 14.04 trusty

[jfindlay]

@commutecat, thanks for reporting. salt-cloud can be used to create and manage more than just minion VMs. See, for example, the --no-deploy option. Do you see anything in the debug log of salt cloud (salt-cloud -l debug --script-args='-D') related to the minion not getting installed? I think salt cloud generates the minion's key before the minion is installed, so that you don't have to accept the key of a VM you know you created.

[commutecat]

This is setup that I use . Another comment will show my attempt trying to fix, but failed (but with many actions)

#/etc/salt/cloud.providers.d/aws-saltify.conf
vpc-saltify:
  minion:
    master: 172.16.55.100
  driver: saltify


#/etc/salt/cloud.profiles.d/saltify-host.conf
saltify-my-web:
  ssh_host:  mywebserver
  ssh_username: ubuntu
  key_filename: '/home/ubuntu/.ssh/mywebserver.pem'
  provider: vpc-saltify


#/etc/hosts
172.16.55.15  mywebserver 

# execution :
sudo salt-cloud -p saltify-my-web mywebserver  -l debug --script-args='-D' 

[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltmaster
[DEBUG   ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/aws-saltify.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/aws-saltify.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/saltify-host.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/saltify-host.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Could not LazyLoad saltify.optimize_providers
[DEBUG   ] The 'saltify' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Using AWS endpoint: ec2.eu-central-1.amazonaws.com
[DEBUG   ] AWS Request: https://ec2.eu-central-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[INFO    ] Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com
[DEBUG   ] Setting read timeout to None
[DEBUG   ] "GET /?Action=DescribeInstances&Version=2014-10-01 HTTP/1.1" 200 None
[DEBUG   ] AWS Response Status Code: 200
[ERROR   ] mywebserver already exists under dnt-aws:ec2
[DEBUG   ] LazyLoaded nested.output
mywebserver:
    ----------
    Error:
        mywebserver already exists under dnt-aws:ec2

[commutecat]

So I try to workaround the issue since salt-cloud refuse to saltify an "existing" server.
I change the server hosts from mywebserver to mywebminion

#/etc/salt/cloud.profiles.d/saltify-host.conf
saltify-my-web:
  ssh_host:  mywebminion
  ssh_username: ubuntu
  key_filename: '/home/ubuntu/.ssh/mywebserver.pem'
  provider: vpc-saltify


#/etc/hosts
172.16. 55.15  mywebminion

# execution :
sudo salt-cloud -p saltify-my-web mywebminion  -l debug --script-args='-D' --keep-tmp

[DEBUG   ] sftp> put  /tmp/tmpEGm9EX /tmp/.saltcloud-8e5a97c4-f93a-4623-ab03-7d5861e8b692/minion
Uploading /tmp/tmpEGm9EX to /tmp/.saltcloud-8e5a97c4-f93a-4623-ab03-7d5861e8b692/minion
/tmp/tmpEGm9EX                                                               100% 4726     4.6KB/s   00:00 ETA
[DEBUG   ] Not removing deployment files from /tmp/.saltcloud-8e5a97c4-f93a-4623-ab03-7d5861e8b692/
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'host': '172.16.55.15', 'name': 'mycrawler', '_stamp': '2016-04-15T10:07:10.183463', 'event': 'mycrawler has been deployed at 172.16.55.15'}
[INFO    ] Salt installed on mywebminion   
[DEBUG   ] LazyLoaded nested.output
mywebminion   :
    ----------
    deployed:
        True

However, salt-minion packages is NOT installed in my EC2 instance mywebminion 172.16.55.15 . When I inspect the /tmp folder inside mywebminion ,

I also notice there is other .saltcloud-* folder inside /tmp folder.

[jfindlay]

@commutecat, I'm not sure if salt-cloud can be used only to install the minion. There are other ways to do this, such as with bootstrap or the package manager on the system. You can even use salt-ssh to do this remotely if desired.

[choopooly]

Same issue for me on 2015.8.8.2, salt-cloud deploy an ec2 instance and no errors are mentioned.
In the minion however nothing has been done, except creating the /tmp/.saltcloud with the minion config and the keys.
I also created the instance using the "-l debug --script-args='-D' --keep-tmp" mentioned by @jfindlay.

I even got these logs from salt-cloud debug:

[INFO    ] Salt installed on test-01
[INFO    ] Created Cloud VM 'test-01'
[DEBUG   ] 'test-01' VM creation details:

Here is my report:

Salt Version:
           Salt: 2015.8.8.2

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: 0.9.1
         PyYAML: 3.10
          PyZMQ: 14.0.1
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-74-generic
         system: Ubuntu 14.04 trusty

[commutecat]

@jfindlay : I notice saltify is a 6 months old code, it works for me once last week.
I know the boostrap.sh and salt-ssh.
It is just funny that : a infrastructure management system cannot deploy its own infrastructure in seamless ways. And salt-cloud are suppose to fill in the blank Or perhaps something is missing in cloud.py ?

[choopooly]

Not sure if that helps but I tried to downgrade salt-cloud to versions '2015.8.5+ds-1', while keeping salt-master at '2015.8.8+ds-2'. Nothing changed, salt-cloud is still not able to bootstrap the minion node...

The workaround for this issue would be to downgrade every salt pkg to '2015.8.5+ds-1'.

[commutecat]

Another solution is totally forget about salt-cloud, just tell everyone to use the baseline salt-master + self-minion, with boto_lc. https://docs.saltstack.com/en/latest/ref/states/all/salt.states.boto_lc.html

[johje349]

Any workaround for this problem? I have the same issue when trying to deploy a digital ocean droplet. The droplet is created but not bootstrapped, i.e. salt-minion is not installed.

[johje349]

The problem was introduced by commit a743778e98796d98181ad94f7cbebee20546def2.
Most likely because config.py moved to a separate directory in the develop branch.

It was later fixed by commit 5a9f4e947e906c5ac5d3b06e0c59298e66abb271.

So the workaround is to update /usr/lib/python2.7/dist-packages/salt/config.py on the salt-master until a release with the fix arrives.

[jfindlay]

@johje349, thanks for your research on this. The new releases will come out in the next week or two.

[johje349]

This is fixed in 2015.8.10.

[jfindlay]

@johje349, thanks for verifying. @commutecat, can you also confirm?

[commutecat]

@johje349 :+1:
@jfindlay : :+1: So it is config.py . Updated, tested and confirm "salt-cloud automatically deployed minion again".

Update my current salt-master from 2015.8.8+ds-2 to 2016-3-0+ds-1

sudo apt-get install salt-master 

sudo salt-cloud -p "new-app" mywebserver 

I get back the long long apt-get update from my minion, test.ping successful and everything now in order.

I didn't test the saltify method, but I suspect it inherit the same problem. Once the detection of config fix, I assume salt cloud saltify driver will works as well.

[jfindlay]

@commutecat, thanks for confirming. Do you think this issue could be closed?

[commutecat]

@jfindlay : I think it is safe to close for 2016-3-0+ds-1.
Should this fix also push to 2015.8.8+ds-2 ?

[jfindlay]

@commutecat, @johje349 has confirmed the fix on the 2015.8 branch, so I think we are good to close.