Salt: Add Recursive Functionality to Salt-Cloud Extends Options

[rterbush]

Possibly related to #8574

I've taken a working setup from where it had evolved through my learning phases and have attempted to clean up and make this a bit more DRY. When attempting to deploy a profile that previously worked with one extends: key in the final profile statement, and instead create a profile definition that builds with multiple extends: keys, I get the following error:

[root@manage etc]# salt-cloud -l debug -p nat-gateway-primary nat.0.staging.uswest2a.lhn
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: aws.compute.amazonaws.com
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/cloud.providers.aws.useast1.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/cloud.providers.aws.useast1.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/cloud.providers.aws.uswest2.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/cloud.providers.aws.uswest2.conf
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/cloud.profiles.aws.uswest2.primary.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/cloud.profiles.aws.uswest2.primary.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/cloud.profiles.aws.uswest2.secondary.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/cloud.profiles.aws.uswest2.secondary.conf
[ERROR   ] 'provider'
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/salt/utils/parsers.py", line 169, in parse_args
    process_option_func()
  File "/usr/lib/python2.6/dist-packages/salt/utils/parsers.py", line 451, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python2.6/dist-packages/salt/utils/parsers.py", line 2638, in setup_config
    return config.cloud_config(self.get_config_file_path())
  File "/usr/lib/python2.6/dist-packages/salt/config/__init__.py", line 1868, in cloud_config
    providers_config)
  File "/usr/lib/python2.6/dist-packages/salt/config/__init__.py", line 2031, in vm_profiles_config
    return apply_vm_profiles_config(providers, overrides, defaults)
  File "/usr/lib/python2.6/dist-packages/salt/config/__init__.py", line 2111, in apply_vm_profiles_config
    if ':' not in extended['provider']:
KeyError: 'provider'
Usage: salt-cloud

salt-cloud: error: Error while processing <bound method SaltCloud.process_config_dir of <salt.cloud.cli.SaltCloud object at 0x30081d0>>: Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/salt/utils/parsers.py", line 169, in parse_args
    process_option_func()
  File "/usr/lib/python2.6/dist-packages/salt/utils/parsers.py", line 451, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python2.6/dist-packages/salt/utils/parsers.py", line 2638, in setup_config
    return config.cloud_config(self.get_config_file_path())
  File "/usr/lib/python2.6/dist-packages/salt/config/__init__.py", line 1868, in cloud_config
    providers_config)
  File "/usr/lib/python2.6/dist-packages/salt/config/__init__.py", line 2031, in vm_profiles_config
    return apply_vm_profiles_config(providers, overrides, defaults)
  File "/usr/lib/python2.6/dist-packages/salt/config/__init__.py", line 2111, in apply_vm_profiles_config
    if ':' not in extended['provider']:
KeyError: 'provider'

cloud.providers.d/cloud.providers.aws.uswest2.conf:

aws-uswest2:
  minion:
    master: salt
    keysize: 2048
  private_key: /srv/etc/.ssh/salt-provisioning
  keyname: salt-provisioning.pub
  ssh_interface: private_ips
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  location: us-west-2
  iam_profile: ServerRole
  driver: ec2

cloud.profiles.d/cloud.profiles.aws.uswest2.primary.conf

#####################################################################
### base instance configurations
base-ec2-uswest2a:
  provider: aws-uswest2
  availability_zone: us-west-2a
  del_root_vol_on_destroy: True
  rename_on_destroy: True

base-ec2-uswest2a-linux:
  extends: base-ec2-uswest2a
  image: ami-d5c5d1e5
  ssh_username: ec2-user

#####################################################################
### instance roles
nat-gateway-primary:
  extends: base-ec2-uswest2a-linux
  image: ami-69ae8259
  private_key: /srv/etc/.ssh/mykey.pem
  keyname: mykey.pem.pub
  iam_profile: ProxyRole
  network_interfaces:
    - DeviceIndex: 0
      PublicIpAddresses:
        - Primary: True
      AssociatePublicIpAddress: True
      allocate_new_eip: False
      SubnetId: subnet-xxx
      SecurityGroupId: sg-xxx
      SourceDestCheck: False
  tag: {'Monitored': 'Yes', 'Role': 'nat'}

salt version info:

[root@manage etc]# salt --versions
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.3.1
         Python: 2.6.9 (unknown, Apr  1 2015, 18:16:00)
           RAET: Not Installed
        Tornado: 4.1
            ZMQ: 3.2.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist:
        machine: x86_64
        release: 3.14.48-33.39.amzn1.x86_64

[jfindlay]

@rterbush, thanks for the report.

[bogdanr]

Happens for me too on Salt: 2015.5.3

[rallytime]

Though that would appear like an intuitive thing to do with extends, this is currently expected behavior. There is a note in the salt-cloud docs about the extends functionality not being recursive: https://docs.saltstack.com/en/latest/topics/cloud/config.html#extending-profiles-and-cloud-providers-configuration

That being said, this is a great feature request! I've edited the title of this report to reflect this feature addition.

[rterbush]

It occurred to me after reporting that this is likely a YAML parser issue.

I would not consider this to be "recursive", but instead, to be "chaining". Which in most other languages I have worked with, is supported. Violates rule of least astonishment. :-)

Thanks for considering.

[rallytime]

Fair point on the use of the word recursive. We'll go with chaining. :)

I don't think it's a YAML parser issue. I think it's the way salt-cloud itself has to handle parsing all of the supported cloud configuration file formats, and the way that the extends behavior was initially written. I've spent some time in this area of the code and it's clear that it wasn't written to be able to chain that many extends statements together.

That being said, I don't see a reason why that can't be done offhand. It just needs to be refactored to support this feature. Thanks again for the request!

[rowedonalde]

@rallytime Necroposting to this issue as we're finally trying to move to core salt-cloud away from our fork of the original repo to which we had added support for chained extends. Is it a deliberate design decision not to allow chained extends in salt-cloud profiles, or is it a technical limitation/bug?

[rallytime]

@rowedonalde It is not a design decision, it just wasn't written originally with that idea in mind. That functionality just needs to be refactored in order to support chaining multiple extends together.

[mlalpho]

+1 this would cut a great deal of clutter from my profiles :)

[birengoodco]

+1 Would help clean up all my profiles and facilitate less redundancy. Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[cmclaughlin]

I would still appreciate this functionality

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[mlalpho]

Please reopen 😬

Sent from Ninehttp://www.9folders.com/

---

From: "stale[bot]" [e-mail]
Sent: Wednesday, January 8, 2020 5:10 PM
To: saltstack/salt
Cc: Michael L Alphonso; Manual
Subject: Re: [saltstack/salt] Add Recursive Functionality to Salt-Cloud Extends Options (#27993)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/saltstack/salt/issues/27993?email_source=notifications&email_token=AB2KXZ7AH6SDHM77TFCC7N3Q4ZFNDA5CNFSM4BR7BLR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIOFGJQ#issuecomment-572281638, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB2KXZ7B4ZM6JHYZXAD3HG3Q4ZFNDANCNFSM4BR7BLRQ.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.