Salt: Postgres module regression on 2016.11

Created on 29 Nov 2016 · 14Comments · Source: saltstack/salt

Description of Issue/Question

Setup

$ sudo salt-call postgres.user_list
'postgres.user_list' is not available.

Steps to Reproduce Issue

Not sure if it's relevant, but I'm running Postgres 9.4 from the Postgres' PGDG repo. The command above works in in 2015.8.1 and shows as not available in 2016.11.

$ sudo dpkg -l | grep postgres
ii  pgdg-keyring                     2014.1                            all          keyring for apt.postgresql.org
ii  postgresql-9.4                   9.4.10-1.pgdg16.04+1              amd64        object-relational SQL database, version 9.4 server
ii  postgresql-client-9.4            9.4.10-1.pgdg16.04+1              amd64        front-end programs for PostgreSQL 9.4
ii  postgresql-client-common         177.pgdg16.04+1                   all          manager for multiple PostgreSQL client versions
ii  postgresql-common                177.pgdg16.04+1                   all          PostgreSQL database-cluster manager
ii  postgresql-contrib-9.4           9.4.10-1.pgdg16.04+1              amd64        additional facilities for PostgreSQL

Versions Report

$ sudo salt-call --versions
Salt Version:
           Salt: 2016.11.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-45-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Bug Execution Module P2 RIoT severity-medium

Source

ipmb

👍8

Most helpful comment

I've done some digging into this and a workaround is to set:

postgres.bins_dir: '/usr/lib/postgresql/<version>/bin/'

on Ubuntu in your /etc/salt/minion.d/your.conf. This will allow Salt to find the initdb binary inside Postgres. This won't work, however, if you don't have the initdb variable anywhere (as in @r3gis3r's case).

A quick fix could be to remove the check for initdb in salt.modules.postgres. In the longer term, the Salt docs will need to be updated to reflect this.

scott-w on 29 Nov 2016

👍3

All 14 comments

neilferreira on 29 Nov 2016

Seeing this as well. I think the issue is wider than just postgres.user_list. All of our postgres_user.present states are failing as well. Eg, one like:

ccnmtl-postgres-user:
  postgres_user.present:
    - name: ccnmtl
    - password: {{password}}
    - user: postgres
    - require:
      - service: postgresql

Fails with:

          ID: ccnmtl-postgres-user
    Function: postgres_user.present
        Name: ccnmtl
      Result: False
     Comment: State 'postgres_user.present' was not found in SLS 'postgresql'
              Reason: 'postgres_user' __virtual__ returned False
     Changes:

Versions here:

Salt Version:
           Salt: 2016.11.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 4.5.5-x86_64-linode69
         system: Linux
        version: Ubuntu 14.04 trusty

thraxil on 29 Nov 2016

👍2

Same problem here : looks like the postgres module now requires initdb to be installed (which on my distrib is installed only if I install a postgres server).

I think that the need of initdb binary makes not necessarily sense for a salt-master server : I use postgres to provide pillar extension and the postgres server is not installed on the salt-master server.

https://github.com/saltstack/salt/blob/2016.11/salt/modules/postgres.py#L115

It would be better if initdb binary is checked at runtime when datadir_init is launched.

r3gis3r on 29 Nov 2016

👍1

I've done some digging into this and a workaround is to set:

postgres.bins_dir: '/usr/lib/postgresql/<version>/bin/'

A quick fix could be to remove the check for initdb in salt.modules.postgres. In the longer term, the Salt docs will need to be updated to reflect this.

scott-w on 29 Nov 2016

👍3

Hi! I have exact situation as @r3gis3r described previously. salt-master doesn't have postgresql package.

2015.* works great, 2016.* have an error.

jfoboss on 29 Nov 2016

Seeing this issue as well. Worked on builds prior to 2016.11.0.

state:

{% from "postgres/map.jinja" import postgres with context %}

# Create Postgres users and databases.

{% for k, v in postgres.users.items() %}
create_pg_user_{{ k }}:
  postgres_user.present:
    - name: {{ k }} # the postgres db user
    - user: postgres # the user account running the postgres commands.
    - password: {{ v.password }}
    - createdb: True
    - replication: True
    - inherit: True
{% endfor %}

Running locally on a minion. Also doesn't work on salt-master either.
```root@minion1:~# salt-call state.apply postgres.users
[ERROR ] State 'postgres_user.present' was not found in SLS 'postgres.users'
Reason: 'postgres_user' __virtual__ returned False

local:

      ID: create_pg_user_teamcity
Function: postgres_user.present
    Name: teamcity
  Result: False
 Comment: State 'postgres_user.present' was not found in SLS 'postgres.users'
          Reason: 'postgres_user' __virtual__ returned False
 Changes:

Summary for local

Succeeded: 0

Failed: 1

Total states run: 1
Total run time: 0.000 ms
```

sdemura on 29 Nov 2016

Can someone give the fix in #37946 a try?

Ch3LL on 29 Nov 2016

Actually my distrib is ubuntu xenial (16.04) and due to https://github.com/saltstack/salt/pull/37946/commits/d33d4039695093b39860d363328263d1f6ccc7d1 revert during patch process I'm pretty sure it will not help to add the bin folder config since my
/usr/lib/postgresql/9.5/bin/ only contains :

pg_config  createdb    createuser  droplang  pg_basebackup  pg_dumpall  pg_receivexlog  pg_restore  reindexdb
clusterdb  createlang  dropdb      dropuser  pg_dump        pg_isready  pg_recvlogical  psql        vacuumdb

I have no binary for initdb installed if I do not install postgres-server aka on ubuntu the package postgres-9.5 : cf http://manpages.ubuntu.com/manpages/xenial/man1/initdb.1.html.
I only installed postgresql-client-9.5 on my salt master node (and I obviously don't want to install a postgres server on this node ;) ).

Note that initdb purpose is to init the database cluster on a volume. Unlike psql it does not take an argument to specify a remote host server. That's why, it's installed in the server package.
And I think that the salt module about postgres commands (with almost all that takes a host as parameter) should not require to have a server installed locally.

r3gis3r on 29 Nov 2016

@Ch3LL

Can someone give the fix in #37946 a try?

Doesn't really help here… I didn't set the option postgres.bins_dir before (in 2016.3) and it stopped working with 2016.11.
The problem seems to be most likely, that the check for the binaries isn't done using the postgres user which would add the required location to the user's PATH.
If this doesn't work due to the PATH only being defined for Shell environments, it should still be covered by Salt to handle this transparently without having to manually keep the postgres paths in Salt's configuration in sync with the deployed version which is guaranteed to break in upgrade situations in a lot of beautiful ways… ;)