Salt: How to use runtime generated variables in states

[vorstenbosch-pwc]

Description of Issue/Question

I have requirements to encrypt a password and place it into a properties file. This encryption is done via proprietary command line tool. In my state file I am using the file.line module to place the password and I want to use the result from the proprietary command line tool as input for the file.line command.

I can use jinja to run a cmd.run invoking the proprietary command line tool and insert the encrypted password as a variable. However when I run salt minion state.apply all the jinja elements get rendered first before executing and therefor the proprietary command line tool is not yet installed resulting in a compile error before the state is even applied.

The only way to work around this (that I know of) is to use a 'two-step' deployment. Where I first run the salt minion state.apply (without the password encryption part) and afterwards run salt minion state.apply password-encryption-part that place the encrypted password.

Is there a way to use salt states, e.g. salt['cmd']['run']('command') on runtime as variables? Or let the jinja templating be rendered state by state? Or something fancy that I completely overlooked? The 'two-step' deployment makes my deployment procedure more complex that need be. Especially if I run into more parts of the deployment that require this kind of functionality.

Setup

• Have a top.sls file stating 2 states to be executed for _minion_.
• State 1 places a script on the file system that returns some text.
• State 2 calls this script (e.g. {{ salt['cmd']['run']('the/script/from/state/one.sh') }}) to fill a variable needed in state 2.

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)
Run salt minion state.apply
and you will get a compile error on the {{ salt['cmd']['run']('the/script/from/state/one.sh') }} because it returns 'no such file or directory' at the moment of rendering the jinja.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
Salt: 2016.3.4

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 1.5
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 0.9.1
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 14.0.1
RAET: Not Installed
smmap: 0.8.2
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist: Ubuntu 14.04 trusty
machine: x86_64
release: 3.13.0-101-generic
system: Linux
version: Ubuntu 14.04 trusty

[gtmanfred]

Unfortunately, there is not something that you overlooked. All the jinja is rendered first, and then the yaml is converted to low state data, this is how we can handle the requisites across different files in the system.

You could use an orchestrate file for your deployment, where the first state puts the file on there, and then you render the whole deploy state.

https://docs.saltstack.com/en/latest/topics/orchestrate/orchestrate_runner.html#orchestrate-runner

[curiositycasualty]

In order of "I'd love":

• a global option for states that forces the state to lazy evaluate at runtime (ala: http://stackoverflow.com/questions/24345180/saltstack-lazy-variable-evaluation-when-rendering-template)
• a clear, coherent example of how to solve this problem using an orchestrate runner

For reference sake, this problem is solved in Chef via: https://docs.chef.io/resource_common.html#run-in-compile-phase

[gtmanfred]

There is no lazy rendering yet, but we have been talking about it. https://github.com/saltstack/salt/pull/38469 is the beginning work in progress of the runtime evaluation.

For an orchestrate, you would just run a state.orchestrate that runs your state that makes the vm first, and then runs the state that does the postgres stuff.
assumine the docker state file basically get docker installed and started up and creates the docker0 bridge, and the docker.postgres state file creates the docker container in your issue, you would remove the include from the postgres state file and use this orchestrate.

orchestrate.docker

setup_docker:
  salt.state:
    - tgt: 'dockerhost'
    - sls:
      - docker

setup_postgres:
  salt.state:
    - tgt: 'dockerhost'
    - sls:
      - docker.postgres

Then it will be run with salt-run state.orch orchestrate.docker

[thenoid]

Since this issue is still open, I don't feel bad posting on it a year later. Looks like salt's [evil] engineering overlord closed #38469. We are running into the need for lazy evaluation of variables over and over again when dealing with AWS.

EG. First you need to create the security group, then you need to get the security group ID for use in ec2 state, then you need the ec2 instance id for state X, etc, etc.

Without the ability to lazily evaluate these variables, we end up with two solutions so far: write our own state function wrapping all the important stuff + adding logic to get the id's or writing our own module wrapping all the important stuff + adding logic to get the id's. Either solution seems ugly and frankly unnecessary.

Evaluating variables at runtime is essential for dealing with dynamic infrastructure.

[gtmanfred]

That got closed by a different one was implemented and will be included in oxygen https://github.com/saltstack/salt/pull/44866

[thenoid]

what release was oxygen? https://docs.saltstack.com/en/latest/topics/releases/version_numbers.html

[gtmanfred]

It hasn't been released yet, it is our next major release

[gtmanfred]

https://docs.saltstack.com/en/latest/topics/slots/index.html exists in
2018.3

On Thu, Jan 10, 2019 at 1:34 PM wongster80 [e-mail] wrote:

Hello, I'm on version 2017.7.2 has this feature to have jinja variables
evaluating at runtime or a workaround implemented yet?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/38072#issuecomment-453224821,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAssob9Jwmo4Z7qthhXYdu1sCJvX0cokks5vB5XUgaJpZM4LENne
.

[nnsense]

Regarding slots, honestly I cannot find it easing the process. As far as I could get, you need to turn a state into a function and squeeze it into another function. Am I right?
If so, a complex state with many arguments would result in a single long not so easy to read line.

My dream would be:

My State:
  - download.this: https://somethingmarvellous.com/onepackage.rpm
  - environ.setval:
    - myvar: rpmname.rpm

Now install:
  pkg.installed:
    - name: {{ salt['environ.getval']('myvar') }}

..yes, I dream in pseudocode..

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.