salt-cloud no longer picking up master setting from provider file

Created on 21 Dec 2017 · 9Comments · Source: saltstack/salt

Description of Issue/Question

With salt-master and salt-cloud 2017.7.2, salt-cloud is no longer picking up the config for minion:master:dns defined in a salt-cloud provider file. This previously worked in salt 2016.11.X

Setup

provider.conf file

myprovider:
  minion:
    master: mymaster.dns.com
  ssh_interface: private_ips
  id: 'redacted'
  key: 'redacted'
  private_key: /etc/salt/mykey.pem
  keyname: mykey
  location: us-west-2
  availability_zone: us-west-2a
  ssh_username:
    - ubuntu
  driver: ec2

profile.conf file

myprofile:
  ssh_interface: private_ips
  block_device_mappings:
    - DeviceName: /dev/sda1
      Ebs.VolumeSize: 30
      Ebs.VolumeType: gp2
  provider: myprovider
  image: ami-myamiid
  script_args: stable 2017.7.2
  size: t2.micro
  ssh_username: ubuntu
  network_interfaces:
    - DeviceIndex: 0
      SubnetId: subnet-mysubnetid
      SecurityGroupId:
        - sg-mysgid
  minion:
    grains:
      env: myenv
      roles:
        - myrole
      team: myteam
  tag:
    env: myenv
    org: myorg
    roles: myrole
    team: myteam

Steps to Reproduce Issue

salt-cloud -p myprofile myprofile-host-1
[WARNING ] There's no master defined on the 'myprofile-host-1' VM settings.

Minion does not connect to master or receive the appropriate master setting after salt-cloud spins it up and bootstraps it.

Versions Report

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: 1.5.2
       cherrypy: 3.2.3
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Nov 20 2017, 18:23:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1041-aws
         system: Linux
        version: Ubuntu 16.04 xenial

Workaround

If I use the salt-cloud root config file located at /etc/salt/cloud instead of the provider file, then it works fine.

minion:
  master: mymaster.dns.com

Bug P2 Salt-Cloud fixed-pending-your-verification severity-medium team-cloud

Source

wes-novack

All 9 comments

I think I have noticed this as well.

I am going to try and see if I can bisect this and find the problem.

gtmanfred on 21 Dec 2017

❤1

It looks like merging configurations between the provider and the profile was accidentally ripped out when we deprecated the keyword provider for driver

https://github.com/saltstack/salt/commit/88936c2c84ba08b041c5b9f31a03944bf22bf1f4

I am only able to replicate this if I also have a minion dictionary in the profile as well.

Thanks for reporting.
Daniel

gtmanfred on 21 Dec 2017

@techhat do you remember where these configs used to get merged?

Thanks,
Daniel

gtmanfred on 21 Dec 2017

I don't. I believe @rallytime did the conversion from provider to driver, she might have a better idea.

techhat on 21 Dec 2017

Sure, but do you remember where in the code the provider and profile configs are merged?

gtmanfred on 22 Dec 2017

To have it also here:

Release of 2017.7.0 was "15 Aug 2017" which breaks this basic functionality... so now over ~ 7 months ago.

my bug report was done 2 weeks after the release 2017.7.0
this bug report was done 4 months after the release 2017.7.0
And even "Bug High Severity labels on 21 Dec 2017" was added on this issue 2,5 months later ( ~ 6,5 months after release 2017.7.0) there is still no commit/test done for this important (?) bug yet.

Also a little wondering how few people seems to use latest version/use enhanced provider file for their configuration setup...

I hope we have not to celebrate birthday for this bug - normally git history should have all changes inside ;).
And as I can check without proper Python test environment the salt-cloud states are very compact.
There is sadly no big jinja templating possible (only in map file some rare access but not in provider / profile ones).

Reiner030 on 9 Mar 2018

I am waiting for this as well. Installing latest salt-master on a new system, 2017.7.4.
We auto-generate our providers, profiles and cloud maps. This is a serious bug IMO.