Salt: Import jinja template in py renderer

[kiemlicz]

Description of question

Typically using salt, we define map.jinja file that defines some defaults (e.g. per OS basis)
Such file can easily be used in *.sls files like so:

{% from "some/state/map.jinja" import state_data with context %}
some_state:
  x.y:
    - name: {{ state_data.something_fancy }}

How to achieve the same using py renderer?

#!py


def run():
  # how to import some/state/map.jinja ???

[gtmanfred]

You can use a defaults.yaml, and just read in the yaml file like you would for anything else in python.

https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.slsutil.html#salt.modules.slsutil.renderer

Or you can render the jinja, and make sure you actually print out the dictionary at the end, and use the slsutil.renderer . It will look different slightly than the map.jinja where you actually import with jinja.

It would also be possible to use jinja directly in the py file.

#!jinja | py

# Import Python lib

{% from 'some/state/map.jinja' import state_data with context %}

def run():
    data = json.loads({{state_data|json}})
    ...
    do stuff with data
    ...
    return whatever

[kiemlicz]

Thank you

1. Using your example as-is doesn't work.
   Yields syntax error: {% from 'some/state/map.jinja' import state_data with context %}
   I found this to be related: https://github.com/saltstack/salt/issues/18487

2. I've tried using slsutil.renderer in my #!py files.
   Example:

#!py

def run():
  data = __salt__.slsutil.renderer('some/state/map.jinja')
  masters = data["masters"]

It fails with error that "Template does not exist"
I kind of worked around this like so:

#!py

def run():
  cachedir = __opts__['cachedir']
  data = __salt__.slsutil.renderer('{}/files/base/some/state/map.jinja'.format(cachedir))
  masters = data["masters"]

It loads the map.jinja but if map.jinja itself loads other jinjas etc - they fail due to same reason (not found)

[gtmanfred]

You can specify salt://some/state/map.jinja and it will pull it from the fileserver.

Or you can use __salt__.cp.cache_file('salt://some/state/map.jinja') and that will cache the file, and also return where it was cached on the minion.

But slsutil.renderer can take a salt:// prefixed string, and it will pull it directly from the fileserver.

[kiemlicz]

Unfortunately neither of the aforementioned solution works
I've started to debug the py renderer.
Main use-case: #!jinja|py working.

Turns out that the first renderer (jinja) works properly, outputs nicely formatted string output, but the py renderer doesn't care and still loads the initial file (not pre-rendered)

def render(template, saltenv='base', sls='', tmplpath=None, **kws):
    '''
    Render the python module's components

    :rtype: string
    '''
    template = tmplpath # <============= overrides pre-rendered template
    if not os.path.isfile(template):
        raise SaltRenderError('Template {0} is not a file!'.format(template))

My thought: does the piping jinja to py is intended to work at all?
Otherwise it is a bug

[danlsgiga]

Also interested in this use case.

@gtmanfred I noticed that your suggestion on using salt:// prefixed string does not work when using slsutil.rendered directly. It works fine when using __salt__.cp.cache_file.

Moreover, rendering map.jinja files do not work, only defaults.yml and sls files

I rely a lot on map.jinja as my core dict for my formulas and would love to have the ability to import it into a python client custom state.

The deficiency of using the defaults.yml file is that any pillar data is not merged.

[kiemlicz]

If you want desperately jinja|py maybe check this very simple renderer
https://github.com/kiemlicz/envoy/blob/master/salt/base/_renderers/stringpy.py
#!jinja|stringpy
It seems to work, but treat it more like a PoC

[xchgaur]

Thank you

1. Using your example as-is doesn't work.
   Yields syntax error: {% from 'some/state/map.jinja' import state_data with context %}
   I found this to be related: #18487
2. I've tried using slsutil.renderer in my #!py files.
   Example:

#!py

def run():
  data = __salt__.slsutil.renderer('some/state/map.jinja')
  masters = data["masters"]

It fails with error that "Template does not exist"
I kind of worked around this like so:

#!py

def run():
  cachedir = __opts__['cachedir']
  data = __salt__.slsutil.renderer('{}/files/base/some/state/map.jinja'.format(cachedir))
  masters = data["masters"]

It loads the map.jinja but if map.jinja itself loads other jinjas etc - they fail due to same reason (not found)

Hey I am trying to import jinja the same way you did(the one you worked around). But I always get null return. Can you help me point out if I am doing something wrong.

/srv/salt# salt --version
salt 2016.11.9 (Carbon)

map.jinja:

#

{% set appbinary = salt'grains.filter_by' %}

Py renderer

#

!py

def run():
cachedir = __opts__['cachedir']
cached_yaml = '{}/files/base/map.jinja'.format(cachedir)
ret = __salt__.slsutil.renderer(cached_yaml)

return {
    'unknown_version': {
        'test.show_notification': [
            {'name': 'unknown version'},
            {'text' : ret }
        ]
    }
}

O/p

#

---

      ID: unknown_version
Function: test.show_notification
    Name: unknown version
  Result: False
 Comment: An exception occurred in this state: Traceback (most recent call last):
            File "/usr/lib/python2.7/site-packages/salt/state.py", line 1765, in call
              **cdata['kwargs'])
            File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1705, in wrapper
              return f(*args, **kwargs)
            File "/usr/lib/python2.7/site-packages/salt/states/test.py", line 263, in show_notification
              raise SaltInvocationError('Missing required argument text.')
          SaltInvocationError: Missing required argument text.
 Started: 12:50:23.698672
Duration: 2.951 ms
 Changes:

Summary for

Succeeded: 0

Failed: 1

Total states run: 1
Total run time: 2.951 ms
ERROR: Minions returned with non-zero exit code

[kiemlicz]

I think your minion didn't pull the map.jinja file to the /var/cache/salt/minion/files.
In my case:
I must have previously used plain sls files without py renderer (which must have pulled that file to minion cache) and then tried using py renderer
Checking your example on fresh (masterless) installation indeed doesn't pull the map.jinja to cache when py renderer is used. I think the use of cp.cache_file is needed (which makes this solution even less pleasant)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[kiemlicz]

not stale

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[TheJJ]

bad bot

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[kiemlicz]

not stale

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[kiemlicz]

not stale

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[sagetherage]

looks like this is stuck so pulling it through triage again and will review status again this week. I will also review the status of the PR linked.

[TheJJ]

Thanks. I've given up setting up the salt dev environment locally and tried to rely on the CI, but it had some failures not caused by my patches, and some are by me, and I've not had motivation fixing them yet. Also I'm not experienced with salts testing practice and rather hacked some test that might do the right coverage. Anyway, that's all in #55390.

[sagetherage]

If you want help with tests or going over this in a deep dive, our engineers will make time for a 30-45 min call or video conference for some 1-to-1 time. I followed up here and tomorrow this will go through a re-triage with Tyler. I can follow up again after that or if you want to schedule some time, do reply!

[sagetherage]

Oh, yes, there are changes in master for the pre-commit and isort, we can get you with an engineer, so do let me know if you would like that, I will make arrangements.

[TheJJ]

I've rebased my branch now, and of course I'd be glad if somebody helped with writing a proper test. I'm not sure though if a video conference would be the right approach, just an actual review and comments on my pull request would help a lot.

[sagetherage]

I hit rebase again as it looked it needed it and asked the assigned reviewer to tackle the review of the PR. I will follow up again later today and tomorrow to try to see this through to completion.

[sagetherage]

still checking on this

[sagetherage]

The PR linked needs work and therefore removing it from the Magnesium release, but keeping this and the PR open to continue work for a future release.