Salt: ENH: Read Pillar files into OrderedDict to preserve source order

[westurner]

It would be great if .iteritems of pillar data read out in source order.

Example:

Pillar Input: https://github.com/saltstack-formulas/openssh-formula/blob/master/pillar.example

sshd_config:
  Port: 22
  Protocol: 2
  HostKey:
    - /etc/ssh/ssh_host_rsa_key
    - /etc/ssh/ssh_host_dsa_key
    - /etc/ssh/ssh_host_ecdsa_key
  UsePrivilegeSeparation: yes
  KeyRegenerationInterval: 3600
  ServerKeyBits: 768
  SyslogFacility: AUTH
  LogLevel: INFO
  LoginGraceTime: 120
  PermitRootLogin: yes
  StrictModes: yes
  RSAAuthentication: yes
  PubkeyAuthentication: yes
  IgnoreRhosts: yes
  RhostsRSAAuthentication: no
  HostbasedAuthentication: no
  PermitEmptyPasswords: no
  ChallengeResponseAuthentication: no
  X11Forwarding: yes
  X11DisplayOffset: 10
  PrintMotd: no
  PrintLastLog: yes
  TCPKeepAlive: yes
  AcceptEnv: "LANG LC_*"
  Subsystem: "sftp /usr/lib/openssh/sftp-server"
  UsePAM: yes

Template: https://github.com/saltstack-formulas/openssh-formula/blob/master/openssh/files/sshd_config

{% set sshd_config = pillar.get('sshd_config', {}) %}
# {...}
{% for keyword, argument in sshd_config.iteritems() %}
    {%- if argument is sameas true %}
{{ keyword }} yes
    {%- elif argument is sameas false %}
{{ keyword }} no
    {%- elif argument is string or argument is number %}
{{ keyword }} {{ argument }}
    {%- else %}
        {%- for item in argument %}
{{ keyword }} {{ item }}
        {%- endfor %}
    {%- endif %}
{%- endfor %}
# {...}

Output: /etc/ssh/sshd_config

StrictModes yes
IgnoreRhosts yes
PermitEmptyPasswords no
PermitRootLogin no
UseDNS no
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
LogLevel INFO
X11DisplayOffset 10
AcceptEnv LANG LC_*
TCPKeepAlive yes
X11Forwarding yes
KeyRegenerationInterval 3600
Subsystem sftp /usr/lib/openssh/sftp-server
ServerKeyBits 768
UsePrivilegeSeparation yes
SyslogFacility AUTH
RhostsRSAAuthentication no
PrintLastLog yes
ChallengeResponseAuthentication no
LoginGraceTime 120
PubkeyAuthentication yes
UsePAM yes
Protocol 2
PrintMotd no
HostbasedAuthentication no
RSAAuthentication yes
Port 22

[basepi]

As far as I know, we are not using OrderedDicts for basic template data, only for the topfile for pillar data. We'll investigate using them for all pillar data to get ordered iteritems.

[aboe76]

I got almost the same kind of issues with my own formula of h2o web server:
where file.serialize is using dataset_pillar which in turn uses pillar.get

init.sls:

{% from "h2o/map.jinja" import h2o with context -%}

h2o:
  pkg.installed:
    - name: {{ h2o.pkg }}
  service.running:
    - name: {{ h2o.service }}
  file.serialize:
    - name: /etc/h2o/{{ h2o.config_file }}
    - user: root
    - group: root
    - mode: 644
    - formatter: yaml
    - dataset_pillar: h2o:config
    - watch_in:
      - service: h2o

pillar: h2o.sls

h2o:
  lookup:
    pkg: h2o-git
  config:
    listen: 8080

    user: http

    access-log: /var/log/h2o-access.log

    hosts:
      "127.0.0.1.xip.io:8080":
        paths:
          /:
            file.dir: /srv/http
            file.dirlisting: 'ON'
            expires: 1 day

rendered config: /etc/h2o/h2o.conf

access-log: /var/log/h2o-access.log
hosts:
  127.0.0.1.xip.io:8080:
    paths:
      /:
        expires: 1 day
        file.dir: /srv/http
        file.dirlisting: 'ON'
listen: 8080
user: http

but it should be:/etc/h2o/h2o.conf

listen: 8080

user: http

access-log: /var/log/h2o-access.log

hosts:
  "127.0.0.1.xip.io:8080":
     paths:
       /:
         file.dir: /srv/http
         file.dirlisting: 'ON'
         expires: 1 day

[westurner]

seeAlso: CPython enhancement 16991 (Python 3.5) https://bugs.python.org/issue16991#msg232825 "Add OrderedDict written in C"

[uvsmtid]

Should this issue be closed because issue #11599 is already closed?
See comment that OrderedDict is already used.

[basepi]

Good catch! This has indeed been implemented.

[westurner]

Thanks!
On Nov 24, 2015 6:10 PM, "Colton Myers" [email protected] wrote:

Closed #12161 https://github.com/saltstack/salt/issues/12161.

—
Reply to this email directly or view it on GitHub
https://github.com/saltstack/salt/issues/12161#event-473949354.

[genuss]

I'm terribly sorry to say that, but this issue isn't still fixed. #11599 was addressed to order in pillar top file, not pillar values.
There are several issues which come from this one: https://github.com/saltstack-formulas/nginx-formula/issues/40 https://github.com/saltstack-formulas/nginx-formula/issues/102 https://github.com/saltstack-formulas/nginx-formula/issues/122 https://github.com/saltstack-formulas/php-formula/issues/17

Can we move to using OrderedDicts in pillar and states?

[nicksloan]

I would love to see OrderedDicts everywhere as well. Ordered dicts are useful in a lot of cases where sequence matters, but you want to preserve the ability to override the value. I would think that this would be valuable for formula authors and end-users alike.

[mpibpc-mroose]

Me too. OrderedDicts make the behaviour mor predictive and intuitive.

[kungfoome]

Can this issue be re-opened?

[gabriel8fm]

Someone have any news about this issue? I have the same on our company salt environment and already tried to disable sort_keys on config.sls but doesn't work.

[gabriel8fm]

FYI We had problems when we tried to apply nginx state (with passenger) in Ubuntu 18.04 Bionic. The nginx.conf was created with the include /etc/nginx/modules-enabled/*.conf; at the end of file ordering alphabetically, causing nginx: [emerg] unknown directive error message. With this in mind we solve our problem puting the below block on nginx.conf template file:

{% if 'include' in config.keys() %}
{{ nginx_block(config.pop('include'), 'include') }}
{%- endif -%}

These setup can be used to another part of formula like pid, user, worker_processes or what you want.

Thanks for the moment.

Regards,

Gabriel Miranda

[noelmcloughlin]

What is best solution for nginx issues?
https://github.com/saltstack-formulas/nginx-formula/pull/150
https://github.com/saltstack-formulas/nginx-formula/issues/102
https://github.com/saltstack-formulas/nginx-formula/issues/40
https://github.com/saltstack-formulas/nginx-formula/pull/120
https://github.com/saltstack-formulas/nginx-formula/pull/101

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[mitar]

Unstale.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.