Salt: Fetching require data during salt states run, after previous states executed.

[hackerwin7]

HI,
I am writing some salt states into SLS file and encounter a simple common case is that:
In a sls file, if state.2 need state.1 generated data.
such as:

download-pkg:
  cmd.run:
    - name: curl -s -o {{ pkg_pah }} {{ pkg_url }}

{%- set dir_name = salt['cmd.run'](' tar tf ' + pkg_path + ' | sed -e "s@/.*@@" | uniq') %}

other-states:
  need {{ dir_name }} value

_other-states_ need _{{ dir_name }}_ value to continue to execute.
After viewing the following links:
Jinja variables are set before the state will be executed

12470

Is there any way to fit this case by using salt states or other flexible solutions for this.

[garethgreenaway]

@hackerwin7 I think what you might want is to use a combination of include and require inside the second state file and have it require the first state file sls. Something like this:

include:
  - state_1

other-states:
  something:
    - require:
      - sls: state_1

[hackerwin7]

@garethgreenaway Thanks your reply, the key problem is how can I get the run-time and dynamic result that state_1 produced ?
Such as following case:
```
state_1

jinja-call-salt-modules-get-produced-result, the jinja must call after the state_1 executed

{% set data = saltmodules %}

other-states:
something:
- some-args: {{ data }}
- other-args
`` Thedatais run-time and dynamic, we can not previously set the value in pillar or grains, thedatais produced by state_1, so we must get the run-time value after thestate_1executed and before the other states run. however, the execution order above is Jinja evaluated before states executed. I test you solution forincludeandrequire. any way, jinja still evaluated beforestate_1` run.
My puzzle is that how can I get the run-time result between the states execution and use it for subsequent execution of states?

[garethgreenaway]

@hackerwin7 ahh that makes more sense. Perhaps the orchastrate module will accomplish what you're trying to do?

[hackerwin7]

@garethgreenaway Thanks your advice very much!
After a simple glance of orchestration, I changed my states file, such as following:

state_1

salt.function

other-states:
  something:
    - some-args: {{ data }}
    - other-args

However, a few puzzles appears:

1. How can I get the result that salt.function produced ? such as {%- set data = salt['cmd.run'](some command)%} ,while jinja render before states run.
2. Orchestration generally applied in the case between the minions, while I just only execute these states in one single minion and one single sls file.
3. Is there any solution for one minion and single sls which implement the following case in the order literally:

state_1

# execute a cmd.run or file.exists or other execution modules and retrieve return data
result = execution_modules

# return data as subsequent states arg
state_2:
   args: {{ result }}

As you see, first I want to use jinja to implement result = execution_modules, but failed because of render order for jinja and states.

[hackerwin7]

After reviewing my question, my key problem is that get a run-time value produced by the states in a single minion and single sls file.
There is a instance for this:

# such as download file or other
download-tar-file:
  cmd.run:
     - name: curl -s -o /path/filename.tar.gz http:/file.url.xxx/filename.tar.gz

# get downloaded tar archive file items
# first, I want to use Jinja, but failed that jinja evaluated before states download-tar-file run
{#  {%- set count = salt['cmd.run']('tar tvf /path/filename.tar.gz | wc -l') %} #}

count is a run-time value, I can not previously set in pillar or grains, this value is produced by the states doanload-tar-file.
Why not single sls file support this combination of states and execution modules in the literally order ?

[hackerwin7]

@garethgreenaway Is there any flexible solution for this? I find this is very similar to me.
Maybe I Should use _writing custom python modules_ (_modules/) for this case. even though I still want to do this by salt state.

[nuharaf]

I have similar problem to this, tough I don't know if there is possible workaround for my problem.
I need to extract an archive, and the set the environment variable based on the top level directory of the archive.

[andreasnuesslein]

same here. I need to create a user, create their ssh-keys, then read the pubkey and use it in a http.query state. is there a nice solution to this?

[dgengtek]

Found this issue when I was faced with the same problem. Creating a user with a formula and querying user data of the same newly created user in another formula during a highstate.

Since the query happens when the highstate gets rendered the highstate will obviously fail with no data for the new user - since the user did not exist at that time.

Getting dynamic results which depend on a previous salt states result in the renderer is impossible since everything gets rendered before the states gets executed. This has to be done when executing a state. Namely doing this with custom states. https://docs.saltstack.com/en/latest/ref/states/writing.html

The only way to do this is by wrapping the states with your custom state modules defined in _states with the required logic.

All you would need to do is for example

# _states/somefilename.py
__virtualname__ = 'mystate'

def __virtual__():
    return __virtualname__

def wrap_whatever(name, username, **kwargs):
    user_data = __salt__.user.info(username) # or some other module which queries some dynamic data which was created in a previous run or query some central datasource
    # do whatever you need to with the dynamically created user data for the required state
    kwargs["name"] = name
    return __states__["somestate.function"](**kwargs)

then call the state in the formula + sync custom states beforehand

my custom wrapped state id:
  mystate.wrap_whatever:
    - name: 'somethingsomething'
    - username: 'someuser'
    - arg_of_wrapped_state1: value1
    - arg_of_wrapped_state2: value2
   ...

This would allow one to create a new user and query its data in the same highstate and update kwargs with whatever is needed which one would pass to the original state which requires this data.

If one does not want to wrap their states then the only sane way to achieve this is by having some central datasource which gets updated via custom external pillars so that all states in a highstate work with the same required data.

[prometheanfire]

I'm wondering if this is the same issue I'm seeing

FOO:
  location:
{% if grains['id'].endswith('.fmt1.foo.com') %}
    site: fmt1
{% else %}
    site: unknown
{% endif %}

nameservers:
{% if salt['pillar.get']('FOO:location:site', 'unknown') == 'fmt1' %}
{% if salt['network.in_subnet']('1.2.12.0/23') %}
  - '1.2.12.16'
{% endif %}
  - '1.2.13.16'
  - '1.2.14.16'
{% else %}
  # WARNING we don't know which sites name servers to use.
  - '8.8.8.8'
  - '8.8.4.4'
{% endif %}

and querying

# salt 'bar.fmt1.foo.com' pillar.get nameservers
bar.fmt1.foo.com:
    - 1.2.8.16
    - 8.8.8.8
    - 8.8.4.4
# salt 'bar.fmt1.foo.com' pillar.get FOO:location:site
bar.fmt1.foo.com:
fmt1

[nnsense]

Same here: a custom exec module is fetching a file from internet, and it returns its name and location. I should use the returned data to install such package during the next state.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[prometheanfire]

reopen, still seems like an issue

on another note, not sure how ansible solves it, but I know you can use output of previous ansible commands in subsequent commands.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[andreasnuesslein]

reopen, still seems like an issue

on another note, not sure how ansible solves it, but I know you can use output of previous ansible commands in subsequent commands.

Yeah ansible is designed different that way. Afaik Salt lacks such a feature and it would be really helpful to have it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[prometheanfire]

been another month, please reopen...

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[prometheanfire]

been another month, please reopen...

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[andreasnuesslein]

don't bother stalebot, this is still an issue.

[sooraj2589]

any update on this? facing the same issue