Salt: `file.get_managed` does not update the cached template after update

[mirceaulinic]

Description of Issue/Question

I am using file.get_managed in order to render a template, physically stored under /etc/salt/templates/ntp.jinja.
Under file_roots I have appended /etc/salt/templates.
When specifying the source argument using the absolute path (i.e.: /etc/salt/templates/ntp.jinja), I don't see any issues.
But when specifying the source as salt://ntp.jinja, it caches the template after the first run and it does not update it. So if the template is updated and file.get_managed is executed again, it is going to use the old version.

Because of this, I need to go under the cache dir and remove the files:

root@ip-172-31-13-136:/var/cache/salt# rm ./master/roots/hash/base/ntp.jinja.hash.sha256
root@ip-172-31-13-136:/var/cache/salt# rm ./proxy/device1/files/base/ntp.jinja

After removing the files above, it caches the newest version and so on.

This happens in both Carbon and develop branch:

Salt Version:
           Salt: 2016.11.0-787-gfc59d5e

Dependency Versions:
           cffi: 1.9.1
       cherrypy: Not Installed
       dateutil: 2.5.3
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.9.5
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.0
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.13 (default, Dec 18 2016, 20:19:42)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.1

System Versions:
           dist: debian 8.6
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Linux
        version: debian 8.6

[ninja-]

@mirceaulinic I don't get it. Could you paste the relevant .sls file part?

[mirceaulinic]

Sure - here you go @ninja-

# cat /etc/salt/templates/ntp.jinja
{%- if grains.vendor|lower == 'juniper' -%}
system{
  replace:
  ntp {
    {%- for peer in peers -%}
    peer {{ peer }};
    {%- endfor -%}
    {%- for server in servers -%}
    server {{ server }};
    {%- endfor -%}
  }
}
{%- endif -%}

[ninja-]

@mirceaulinic but how exactly are you executing this? salt command line or .sls file? Could you paste the command?

[mirceaulinic]

More specifically the template is rendered inside this function called net.load_template: https://github.com/saltstack/salt/blob/develop/salt/modules/napalm_network.py#L1171-L1182. At this level it only forwards the arguments.

Running from the CLI directly, it has the same behaviour:

# salt device1 file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash=None source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.4']"

One can easily reproduce by:

• run the command above
• modify the template
• rerun the command
• boom :)

[mirceaulinic]

Here are my steps:

• run the command... all good:

root@ip-172-31-13-136:~# cat /etc/salt/templates/ntp.jinja
{%- if grains.vendor|lower == 'juniper' -%}
system{
  replace:
  ntp {
    {%- for peer in peers -%}
    peer {{ peer }};
    {%- endfor -%}
    {%- for server in servers -%}
    server {{ server }};
    {%- endfor -%}
  }
}
{%- endif -%}
root@ip-172-31-13-136:~# sudo salt device1 file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash=None source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.4']"
device1:
    - /tmp/__salt.tmp.gmobAi
    |_
      ----------
      hash_type:
          sha256
      hsum:
          eac8947e462887f22477a907718f2ff5a6573677ac857b5ea814f412f856949e
root@ip-172-31-13-136:~# cat /tmp/__salt.tmp.gmobAi
system{
  replace:
  ntp {peer 172.17.17.1;server 1.2.3.4;}
}

• update the template:

root@ip-172-31-13-136:~# vi /etc/salt/templates/ntp.jinja
root@ip-172-31-13-136:~# cat /etc/salt/templates/ntp.jinja
{%- if grains.vendor|lower == 'juniper' -%}
system{
  replace:
  ntp {
    {%- for peer in peers -%}
    peer {{ peer }};
    {%- endfor -%}
  }
}
{%- endif -%}

• rerun the command... and file.get_managed is still using the old template:

root@ip-172-31-13-136:~# sudo salt device1 file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash=None source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.4']"
device1:
    - /tmp/__salt.tmp.nt2tnA
    |_
      ----------
      hash_type:
          sha256
      hsum:
          eac8947e462887f22477a907718f2ff5a6573677ac857b5ea814f412f856949e
root@ip-172-31-13-136:~# cat /tmp/__salt.tmp.nt2tnA
system{
  replace:
  ntp {peer 172.17.17.1;server 1.2.3.4;}
}

• remove the cache

root@ip-172-31-13-136:~# find /var/cache/salt/ | grep ntp.jinja
/var/cache/salt/master/roots/hash/base/ntp.jinja.hash.sha256
/var/cache/salt/proxy/device1/files/base/ntp.jinja
root@ip-172-31-13-136:~# rm /var/cache/salt/master/roots/hash/base/ntp.jinja.hash.sha256
root@ip-172-31-13-136:~# rm /var/cache/salt/proxy/device1/files/base/ntp.jinja

• rerun after deleting the cache... all good again:

root@ip-172-31-13-136:~# sudo salt device1 file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash=None source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.4']"
device1:
    - /tmp/__salt.tmp.JXs3y5
    |_
      ----------
      hash_type:
          sha256
      hsum:
          47a8b8efed44b3244d8d66a87053a2feebef96b048a93d4c0dd0063505824956
root@ip-172-31-13-136:~# cat /tmp/__salt.tmp.JXs3y5
system{
  replace:
  ntp {peer 172.17.17.1;}
}
root@ip-172-31-13-136:~#

Hope this exposes better the issue.

[gtmanfred]

I have confirmed this.

Also, it looks like the source_hash is only used to check if the hash matches the cached file, and not the file on the remote, so it isn't being updated if you specify the source_hash either.

[root@salt ~]# salt-call file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash='md5=70947f359f58410688f179f6b45d4cd7' source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.5']"
local:
    - /tmp/__salt.tmp.ISOsmR
    |_
      ----------
      hash_type:
          sha256
      hsum:
          15aab43045eba84c4c2b7736b959924b525b2bf4087a3f4801177bc438425ae2
[root@salt ~]# cat /tmp/__salt.tmp.ISOsmR
system{
  replace:
  ntp {peer 172.17.17.1;server 1.2.3.5;}
}
[root@salt ~]# md5sum /srv/salt/ntp.jinja
0d86303d31656d8e6e5c13b3976052c8  /srv/salt/ntp.jinja
[root@salt ~]# salt-call file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash='md5=0d86303d31656d8e6e5c13b3976052c8' source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.5']"
local:
    - /tmp/__salt.tmp.QX_AdJ
    |_
      ----------
      hash_type:
          sha256
      hsum:
          15aab43045eba84c4c2b7736b959924b525b2bf4087a3f4801177bc438425ae2
[root@salt ~]# cat /tmp/__salt.tmp.QX_AdJ
system{
  replace:
  ntp {peer 172.17.17.1;server 1.2.3.5;}
}

Thanks for reporting,
Daniel

[gtmanfred]

As always anything you could contribute would be greatly appreciated. 😄

I would say that the work around for now might be to check the hash of the file on the master.

https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.cp.html#salt.modules.cp.hash_file
and then cp.cache_file to update the cache if the hash is different.

Other than that, we just need time to get to this to fix it, and there is a lot ahead of it unfortunately :/

Thanks!
Daniel

[mirceaulinic]

Sounds good, I will submit tomorrow a PR trying to address it.

Cheers,
Mircea

[mirceaulinic]

Submitted https://github.com/saltstack/salt/pull/39438 to fix that.

[mirceaulinic]

Closed via https://github.com/saltstack/salt/commit/e816d6c23ec71c9b0d85e71d2e19cefe95da3d99

[mirceaulinic]

This is borked again in 2016.11.4:

root@ip-172-31-15-232:/etc/salt# cat > templates/ntp.jinja
{% for server in servers %}
ntp server {{server}}
{% endfor %}
^C
root@ip-172-31-15-232:/etc/salt# sudo salt device1 file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash=None source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.4']"
device1:
    - /tmp/__salt.tmp.8llSBr
    |_
      ----------
      hash_type:
          sha256
      hsum:
          efec430883160b3dc28cc8b5c2665eb7a17bbf88849fb896157de0f1c6745c2c
root@ip-172-31-15-232:/etc/salt# cat /tmp/__salt.tmp.8llSBr

ntp server 1.2.3.4

root@ip-172-31-15-232:/etc/salt# cat > templates/ntp.jinja
{% if (servers is defined) and servers %}
system {
  ntp {
    {% for server in servers %}
      {% if server %}
        server {{server}};
      {% endif %}
    {% endfor %}
  }
}
{% endif %}
^C
root@ip-172-31-15-232:/etc/salt# sudo salt device1 file.get_managed name=/tmp/stuff source=salt://ntp.jinja source_hash=None source_hash_name=None user=root group=root mode='755' template=jinja saltenv=base skip_verify=True context=None defaults=None peers="['172.17.17.1']" servers="['1.2.3.4']"
device1:
    - /tmp/__salt.tmp.SssGL9
    |_
      ----------
      hash_type:
          sha256
      hsum:
          efec430883160b3dc28cc8b5c2665eb7a17bbf88849fb896157de0f1c6745c2c
root@ip-172-31-15-232:/etc/salt# cat /tmp/__salt.tmp.SssGL9

ntp server 1.2.3.4

[mirceaulinic]

@gtmanfred I have re-opened this ticket. Would you recommend me to close this one and open a separate issue, as this happens now in 2016.11.4, while the initial description corresponds to 2016.11.2?

[gtmanfred]

I don't see a need to open a new issue if the issue is the same.

[mirceaulinic]

@gtmanfred false alarm :(
It was me, using the skip_verify=True. Apologies for the noise.