Salt: Salt Could should have a way to define an entire infrastructure

[ryanwalder]

At present the map file is pretty limited in what it can do in the fact it will only spin up instances not other resources (unless I'm missing something) and overstate seems to be dead, ideally you should be able to define an entire infrastructure (cross provider) through a mapfile/overstate style system.

My thinking is something similar to how you would write states would be a really powerful way of doing things, especially if you have a reasonably static infrastructure:

test-network:
  gce_network.present:
    - name: test
    - addresses: 10.0.0.1/8
    - gateway: 10.0.0.1

office-firewall:
  gce_firewall.present:
    - name: all-from-office
    - network: test
    - source_filter: ip
    - source_range: 123.123.123.123/32
    - rules: "udp:0-65535; icmp; tcp:0-65535"
    - require:
      - gce_network: test-network

web-firewall:
  gce_firewall.present:
      - name: web
      - network: test
      - source_filter: any
      - rules: "tcp:80,443"
      - target_tags: "web"
      - require:
        - gce_network: test-network

web-instance-disk:
    gce_disk.present:
      - name: web-data
      - type: ssd
      - image: None
      - delete_with_instance: False

web-instance-machine:
  gce_instance.present:
    - name: web-01
    - type: n1-standard-1
    - disk_name: web-01
    - delete_disk: True
    - disk_image: ubuntu-1404
    - network: test
    - zone: europe-west1-b
    - tags:
      - web
      - foo
    - require:
      - gce_disk: web-instance-disk
      - gce_firewall: web-firewall
      - gce_network: test-network

aws-web-01-dns:
  aws_route53.present:
    - name: web-01
    - type: A
    - value: <web-01.gce_instance.ip>

The main issue I can see here is self referencing things which haven't been created yet (as seen in the DNS block at the bottom) and that rather than defining full instances using a profile may be better (assuming a definition could override things if needed)

[jfindlay]

@ryanwalder, the overstate system has been replaced by the orchestrate runner, see http://docs.saltstack.com/en/latest/topics/tutorials/states_pt5.html. This may do all or most of what you are thinking of, although the actual VM creation would still be a separate step.

[cachedout]

@jfindlay FWIW, the cloud.py state could be used to roll VM provisioning into a single orchestrate runner.

[ryanwalder]

@jfindlay That was more my point, being able to define an entire infrastructure in a salt state/formula style and have salt-cloud be able to provision and configure and entire infrastructure would be extremely powerful.

At present cloud map files are limited to spinning up VMs (as far as I understand) which is only one piece of the puzzle, for example a map file for a Cassandra cluster will get the VMs up and running but anything that isn't a VM you need to manually call salt-cloud from the master which leads to 5ish manual steps when provisioning a cluster (create network, create firewall rules, create disks, call a salt-cloud map to setup cluster, cluster gets configured by salt states/orchestrate is run to configure).

Ideally you should be able to create anything salt-cloud could from within a state like system with some way of grabbing data (kind of like mine) from one bit to another, so I could have a state to spin up a VM (perhaps from a profile?), then a state to set the DNS which requires the VM then grabs the IP hen sets the DNS for that VM.

[jfindlay]

@ryanwalder, yes, I think the intent is to start moving in that direction, but I do not know what the specific details are going to be. @techhat and @UtahDave may be able to give you more information.

[techhat]

@ryanwalder is dead on with what needs to be done. There is already a cloud state module to move in this direction, but it really only supports creating nodes. The next step is to add generic cloud states to that module to handle other features. And to be Salty, they really do need to be as generic as possible (i.e., cloud_dns.a_record/aaaa_record instead of cloud.route53 or route53.a_record).

Some resources offhand that are offered by multiple cloud providers:

• Resource tags
• Disks/volumes
• DNS records

[jfindlay]

@techhat, does an issue exist to document and track the work on this project, or should this be it? :-)

[techhat]

@jfindlay, if one exists, either I don't know about it or I've forgotten.

[DanyC97]

has an issue been created on this already ? the suggestion from @ryanwalder is spot on :+1:

[jfindlay]

@DanyC97, it is looking like this is that issue. :-)

[aphor]

Here's an idea for salt-cloud architectural direction: What about the proxy minion interface? https://docs.saltstack.com/en/latest/topics/proxyminion/index.html
Example: https://docs.saltstack.com/en/latest/topics/proxyminion/demo.html

• @thatch45 @cro (who seem to be behind the design of proxy minion)

This sounds like a very intuitive way to implement salt-cloud providers. Drivers would be reimplemented as proxy and execution modules. The Amazon boto stuff is already implemented as execution modules. Rather than defining cloud providers in /etc/salt/cloud/providers.d, providers would be implemented as proxy minions defined using pillars. Instead of defining cloud profiles in /etc/salt/cloud/profiles.d, profiles would be implemented as proxy minion states and grains. Beacons and reactors and mines? Yes! I feel a Mesos integration coming on... a Mesos pillar?

Using states (with the support of pillars and grains) neatly solves the problems of tracking and dealing with the ancillary configuration items that depend on VMs like DNS records or other orchestration facilities which advertise capabilities, floating IP addresses, perstistent bulk storage containers, etc.

We could keep the salt-cloud CLI, and slowly deprecate the drivers as proxy-minion replaces it. We could file issues to implement individual cloud-provider execution modules (hey, Amazon is already mostly supported through boto), matching proxy-minions, and grains.

[DanyC97]

great idea @aphor

[zlanich]

For clarity, did this thread die for no reason, or because Salt Orchestrate Runner can now do everything mentioned above?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.