Salt: Pyobjects does not honor reload modules during render

[tyhunt99]

Description of Issue/Question

I am using pyobjects to render my sls files. I am trying to setup RAID on a newly deployed server. The distro I am using (ubuntu 14.04.4 LTS) does not have mdadm (raid tool we are using) installed by deafult. In my state the first thing I do is a Pkg.installed for mdadm and then use the Raid state module to actually create the raid. However the ssl fails to render with a NameError because the Raid state module is not loaded because it technically doesn't have the required dependencies installed even though the state is installing them.

Setup

Here is a slightly simplified version of my state. I made sure to set reload module to true.

with Pkg.installed('pde_disk_required_packages',
    pkgs=['mdadm','lvm2'],
    reload_modules=True):
            raid = '/dev/md0'        
            devices = grains('raids:'+raid+':devices',[])
            Raid.present('create_raid '+raid,
                name=raid,
                level='10',
                devices=devices,
                chunk=256,
                run=True)

Steps to Reproduce Issue

Running this state results in the following render error.

 Rendering SLS pde_disks.lvm failed, render error: name 'Raid' is not defined
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2672, in render_state
    sls, rendered_sls=mods
  File "/usr/lib/python2.7/dist-packages/salt/template.py", line 95, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/renderers/pyobjects.py", line 460, in render
    exec_(final_template, _globals)
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 679, in exec_
    exec("""exec _code_ in _globs_, _locs_""")
  File "<string>", line 1, in <module>
  File "<string>", line 18, in <module>
NameError: name 'Raid' is not defined

Versions Report

Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.21.1
           Mako: 0.9.1
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2
           cffi: 1.5.0
       cherrypy: 3.2.2
       dateutil: 2.4.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libgit2: 0.23.4
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 1.2.3
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.23.1
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-71-generic
         system: Ubuntu 14.04 trusty

My minion is running salt 2015.8.8.2

[jfindlay]

@tyhunt99, thanks for reporting.

[DBassel]

+1

[tyhunt99]

I upgraded to a newer version of salt (2016.3.1) and am still seeing this issue.

There also now appears to be what I believe is a related issue involving dockerng module. It now appears to require a restart of salt minion to detect that docker has been installed.

Starting with a fresh machine

salt 'test' state.apply docker #installs docker and all prereqs
salt 'test' state.apply mystate reload_modules=True #this does not seem to see docker is installed

mystate.sls:

#!pyobjects

Dockerng.image_present('autodns_image',
    name='364535168108.dkr.ecr.us-east-1.amazonaws.com/autodns')

Dockerng.running('autodns',
    detach=True,
    name='autodns',
    tty=True,
    binds=['/var/run/docker.sock:/var/run/docker.sock'],
    image='xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/autodns',
    port_bindings='53:53/udp',
    hostname='autodns',
    working_dir='/app',
    require=Dockerng('autodns_image'))

Results in the following error:

test:
    Data failed to compile:
----------
    Rendering SLS docker-host failed, render error: name 'Dockerng' is not defined
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2776, in render_state
    sls, rendered_sls=mods
  File "/usr/lib/python2.7/dist-packages/salt/template.py", line 95, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/renderers/pyobjects.py", line 464, in render
    exec_(final_template, _globals)
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 679, in exec_
    exec("""exec _code_ in _globs_, _locs_""")
  File "<string>", line 1, in <module>
  File "<string>", line 30, in <module>
NameError: name 'Dockerng' is not defined

But as soon as I restart the salt-minion salt 'test' service.restart salt-minion it works just fine. Not sure if these issues are actually related or not.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[dgengtek]

This issue is still relevant. It is not possible to require and reload modules with pyobjects renderer since like mentioned above the objects do not exist when pyobjects renders the state and fails with the NameError.

salt version: 2018.3.1

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[rallytime]

Thanks for updating this @dgengtek!

[jfrederickson]

Oof, I just ran into this myself... was hoping to use the pyobjects renderer for a few things, but this might be a blocker.

This is an interesting chicken and egg problem though; salt.loader.minion_mods seems to only load execution modules that are usable on the current system, which it determines by calling __virtual__() in each of them. But some of them might not be usable until other states run!

It feels like a reasonable thing to do here would be to build objects for every execution module and not just ones that are currently runnable. I'll have to dig into the Salt codebase and see if there's a simple way to do that.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[einsiedlerkrebs]

I am observing a similar behaviour when trying to use zpool state on a FreeBSD salt-sshtarget, called from within a Debian "master". Same thing I could observe for using the service states modules.
Salt Version: Salt: 3000.2

@jfrederickson:

This is an interesting chicken and egg problem though; salt.loader.minion_mods seems to only load execution modules that are usable on the current system, which it determines by calling __virtual__() in each of them. But some of them might not be usable until other states run!

This looks like a good explanation for this behaviour...
It would be nice, if PyObjects would get a better priority as renderer, since it is best in readability and flexibility.