Salt: strftime filter not found in 2018.3.0

[oeuftete]

Description of Issue/Question

The strftime filter seems to be missing somehow in 2018.3.0. (Other internal filters like to_bool or exactly_one_true work fine.)

SaltRenderError: Jinja syntax error: no filter named 'strftime'

Setup

Run a dockerized masterless minion.

Steps to Reproduce Issue

salt-call -l debug slsutil.renderer string='{{ "2002/12/25"|strftime("%y") }}'

Output:

[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/minion
[INFO    ] Could not determine init system from command line: (bash)
[DEBUG   ] The `lspci` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] Please install 'virt-what' to improve results of the 'virtual' grain.
[DEBUG   ] All tools for virtual hardware identification failed to execute because they do not exist on the system running this instance or the user does not have the necessary permissions to execute them. Grains output might not be accurate.
[INFO    ] Although 'systemd-detect-virt' was found in path, the current user cannot execute it. Grains output might not be accurate.
[DEBUG   ] Loading static grains from /etc/salt/grains
[DEBUG   ] Determining pillar cache
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded slsutil.renderer
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: :string:
[DEBUG   ] Jinja search path: [u'/var/cache/salt/minion/files/base']
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] Jinja2 environment lstrip_blocks was set to True by jinja_env
[DEBUG   ] Jinja2 environment trim_blocks was set to True by jinja_env
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/local/lib/python2.7/dist-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
    tmplstr)
SaltRenderError: Jinja syntax error: no filter named 'strftime'; line 1

---
{{ "2002/12/25"|strftime("%y") }}    <======================
---
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
SaltRenderError: Jinja syntax error: no filter named 'strftime'; line 1

---
{{ "2002/12/25"|strftime("%y") }}    <======================
---
Traceback (most recent call last):
  File "/usr/local/bin/salt-call", line 11, in <module>
    sys.exit(salt_call())
  File "/usr/local/lib/python2.7/dist-packages/salt/scripts.py", line 400, in salt_call
    client.run()
  File "/usr/local/lib/python2.7/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/local/lib/python2.7/dist-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/local/lib/python2.7/dist-packages/salt/cli/caller.py", line 212, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/salt/modules/slsutil.py", line 146, in renderer
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/salt/template.py", line 93, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/salt/renderers/jinja.py", line 70, in render
    **kws)
  File "/usr/local/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/local/lib/python2.7/dist-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
    tmplstr)
SaltRenderError: Jinja syntax error: no filter named 'strftime'; line 1

---
{{ "2002/12/25"|strftime("%y") }}    <======================
---
Traceback (most recent call last):
  File "/usr/local/bin/salt-call", line 11, in <module>
    sys.exit(salt_call())
  File "/usr/local/lib/python2.7/dist-packages/salt/scripts.py", line 400, in salt_call
    client.run()
  File "/usr/local/lib/python2.7/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/local/lib/python2.7/dist-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/local/lib/python2.7/dist-packages/salt/cli/caller.py", line 212, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/salt/modules/slsutil.py", line 146, in renderer
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/salt/template.py", line 93, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/salt/renderers/jinja.py", line 70, in render
    **kws)
  File "/usr/local/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/local/lib/python2.7/dist-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
    tmplstr)
salt.exceptions.SaltRenderError: Jinja syntax error: no filter named 'strftime'; line 1

---
{{ "2002/12/25"|strftime("%y") }}    <======================

Versions Report

(I installed dateutil and timelib after seeing this problem, and nothing changed.)

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 17.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.1.6

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.13.0-37-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[gtmanfred]

@mirceaulinic can you take a look at this when you get a minute?

[root@e12e29582184 /]# salt-call --versions-report
Salt Version:
           Salt: 2018.3.0-n/a-3923571

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.9.87-linuxkit-aufs
         system: Linux
        version: CentOS Linux 7.4.1708 Core

[root@e12e29582184 /]# cat /srv/salt/test.sls
{% do salt.log.error("2002/12/25"|strftime("%y")) %}
[root@e12e29582184 /]# salt-call --local state.apply test
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
    tmplstr)
SaltRenderError: Jinja syntax error: no filter named 'strftime'; line 1

---
{% do salt.log.error("2002/12/25"|strftime("%y")) %}    <======================
---
[CRITICAL] Rendering SLS 'base:test' failed: Jinja syntax error: no filter named 'strftime'; line 1

---
{% do salt.log.error("2002/12/25"|strftime("%y")) %}    <======================
---
local:
    Data failed to compile:
----------
    Rendering SLS 'base:test' failed: Jinja syntax error: no filter named 'strftime'; line 1

---
{% do salt.log.error("2002/12/25"|strftime("%y")) %}    <======================
---

Thanks,
Daniel

[mirceaulinic]

It looks like the strftime filter is injected through the jinja_filter decorator which I'm not entirely sure how it works: https://github.com/saltstack/salt/blob/2018.3/salt/utils/dateutils.py#L63
Maybe @lorengordon has a better knowledge?

[oeuftete]

The date_format filter has the same issue (though looking at the code, maybe that's out of date docs and it's been replaced by strftime?)

[lorengordon]

The jinja_filter decorator just creates a dictionary of <filter_name>: <callable>, which is then added to the jinja env using the dictionary .update() method. If the filter is not available, then all I can imagine is that there is some problem loading the dateutils.py module. @terminalmage has the latest changes in history there.

[terminalmage]

Should already be fixed for 2018.3.1 via https://github.com/saltstack/salt/pull/46744

[terminalmage]

Note: That PR mentions salt-ssh but this wouldn't have be confined to salt-ssh.

[oeuftete]

Thanks everyone. I should have found that related issue. Also glad to see #46848 is in the works to add tests.

Should I open a new issue for https://github.com/saltstack/salt/issues/46834#issuecomment-378319423. It looks like the date_format util was deprecated, but not removed from the filter docs. Or maybe it should be left in the filter docs and decorated to still be a filter. In any case, the docs do not currently reflect reality, AFAICT.

https://github.com/saltstack/salt/blob/ec973104afd2a7b68460f3a48258315ba5dd948d/salt/utils/__init__.py#L1796-L1808

[noelmcloughlin]

@terminalmage when will 2018.3.1 be available? thanks.
salt-minion is already the newest version (2018.3.0+ds-1).

[terminalmage]

@oeuftete Thanks, I've added it back in https://github.com/saltstack/salt/pull/46853.

In 2017.7 both date_format and strftime filters reference the same filter, but it seems that when we moved to a decorator-based approach, only the strftime filter was added as a decorator. It doesn't even look like there was a decision to deprecate the filter, it seems to have been just an oversight by the community member who created the @jinja_filter decorator.

@noelmcloughlin The tentative release date for 2018.3.1 is April 30th, roughly 4 weeks from now.

[PeterS242]

Is there any sort of work-around available for strftime or date_format for the current 2018.3.0 release?

[terminalmage]

@PeterS242 strftime still should work in 2018.3.0, it's just date_format that does not. In 2017.7 they both were the same, it's just that date_format was not carried over when the method of creating jinja filters was moved to the decorator approach.

[terminalmage]

Oh I misread, sorry, the lack of a dateutils import means that strftime did not work in 2018.3.0. The only workaround would be to add that import to salt/utils/templates.py

[oeuftete]

@PeterS242 If your platform supports it, you can work around it replacing something like None|strftime("%a") with salt['cmd.run']("date '+%a'").