Salt: Consul backend minion cache does not work

[djhaskin987]

Description of Issue/Question

I just confirmed that grain targeting fails regardless of whether or not salt '*' saltutil.sync_all is called, when the cache backend is consul.

Setup

1. Download attached zip file --> saltbug.tar.gz
2. Unzip it, enter the directory, and type vagrant up
3. vagrant ssh master
4. apt-get install -y vim unzip
5. cd /vagrant
6. unzip consul*.zip
7. mv consul /usr/bin

Steps to Reproduce Issue

Steps to reproduce, using the vagrant environment I attached above:

I just confirmed that this problem exists regardless of whether or not salt '*' saltutil.sync_all is called, when the cache backend is consul.

Steps to reproduce, using the vagrant environment I attached above:

1. Start Consul on the salt master:

mkdir -p /var/lib/consul
mkdir -p /etc/consul.d
consul agent -server -bootstrap-expect=1 \
    -data-dir=/var/lib/consul -node=agent-one -bind=127.0.0.1 \
    -datacenter=dc1 -config-dir=/etc/consul.d

1. Configure consul cache on the salt master:

cache: consul
consul.host: 127.0.0.1
consul.port: 8500
consul.token: None
consul.scheme: http
consul.consistency: default
consul.dc: dc1
consul.verify: True

1. Restart salt master
2. Run rm -rf /var/cache/salt/master/minions/*
3. Run salt '*' saltutil.sync_all
4. To verify that salt did indeed sync minion cache data, run consul kv get -recurse -detailed minions
5. Run salt -C 'G@examplegrain:faraway' test.ping

You will see minion a returning with "True". Minion a is the only minion which has this grain set; the other minions should not be targeted. These other minions return erroneously with "Minion did not return". This means that salt consulted the minion data cache, but couldn't find any entries in the cache associated with those minions.

Versions Report

root@saltmaster:~# salt --versions
Salt Version:
           Salt: 2016.11.3

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-21-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[djhaskin987]

This problem was moved over from #35439 and relates to it.

[gtmanfred]

Your saltbug.tar.gz needs the master: in the saltstack/etc/minion to be changed to saltmaster

Testing this now.

[djhaskin987]

Thanks!

[gtmanfred]

It looks like you have an error in the vagrant setup somewhere, when I setup this using docker-compose and this repo https://github.com/gtmanfred/salt-docker-demo/tree/issue/40748

[root@saltmaster /]# consul kv get -recurse minions
[root@saltmaster /]# salt \* saltutil.sync_all
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'saltmaster_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/a3/c7b7c7abfe1854d8a9f35047d4a905f95396b40f4d29623344c91dcbcdfa71/.minions.p
[DEBUG   ] get_iter_returns for jid 20170419235252749676 sent to set(['minion1', 'minion2']) will timeout at 23:52:57.792739
[DEBUG   ] jid 20170419235252749676 return from minion1
[DEBUG   ] LazyLoaded nested.output
minion1:
    ----------
    beacons:
    engines:
    grains:
    log_handlers:
    modules:
    output:
    proxymodules:
    renderers:
    returners:
    sdb:
    states:
    utils:
[DEBUG   ] jid 20170419235252749676 return from minion2
[DEBUG   ] LazyLoaded nested.output
minion2:
    ----------
    beacons:
    engines:
    grains:
    log_handlers:
    modules:
    output:
    proxymodules:
    renderers:
    returners:
    sdb:
    states:
    utils:
[DEBUG   ] jid 20170419235252749676 found all minions set(['minion1', 'minion2'])
[root@saltmaster /]# consul kv get -recurse minions
minions/minion1/data:��pillar��grains�=�kernel�Linux�domain��uid�zmqversion�4.1.6�kernelrelease�3.10.0-514.10.2.el7.x86_64�pythonpath��/usr/bin�/src/salt�/usr/lib64/python27.zip�/usr/lib64/python2.7� /usr/lib64/python2.7/plat-linux2�/usr/lib64/python2.7/lib-tk�/usr/lib64/python2.7/lib-old� /usr/lib64/python2.7/lib-dynload�"/usr/lib64/python2.7/site-packages� /usr/lib/python2.7/site-packages�pidT�ip_interfaces��groupname�root�fqdn_ip6��mem_total�t�saltversioninfo���
                                                             �SSDs��mdadm��id�minion1�osrelease�7.3.1611�ps�ps -efH�server_id�VY�fqdn�minion1�lsb_distrib_id�CentOS Linux�ip6_interfaces��num_cpus�hwaddr_interfaces��osfullname�CentOS Linux�ip4_interfaces��init�systemd�gid�master�saltmaster�virtual_subtype�Docker�dns��domain��sortlist��nameservers��127.0.0.11�ip4_nameservers��127.0.0.11�search��c.gtmanfred-1263.internal�google.internal�ip6_nameservers��options��ndots:0�ipv6��cpu_flags�2�fpu�vme�de�pse�tsc�msr�pae�mce�cx8�apic�sep�mtrr�pge�mca�cmov�pat�pse36�clflush�mmx�fxsr�sse�sse2�ss�ht�syscall�nx�pdpe1gb�rdtscp�lm�constant_tsc�rep_good�nopl�xtopology�nonstop_tsc�eagerfpu�pni�pclmulqdq�ssse3�cx16�sse4_1�sse4_2�x2apic�popcnt�aes�xsave�avx�hypervisor�lahf_lm�tsc_adjust�xsaveopt�localhost�minion1�ipv4��username�root�fqdn_ip4��shell�/bin/sh�nodename�minion1�saltversion�2016.11.4-28-gd8f7855�systemd��version�219�features��+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN�saltpath�/src/salt/salt�osmajorrelease�7�os_family�RedHat�oscodename�CentOS Linux 7 (Core)�osfinger�CentOS Linux-7�pythonversion��final�num_gpus�virtual�gce�disks��sda�cpu_model�Intel(R) Xeon(R) CPU @ 2.60GHz�host�minion1�pythonexecutable�/usr/bin/python2�osarch�x86_64�cpuarch�x86_64�lsb_distrib_codename�CentOS Linux 7 (Core)�osrelease_info��K�locale_info��detectedencoding�UTF-8�defaultlanguage�en_US�defaultencoding�UTF-8�gpus��path�</usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/bin:/sbin�machine_id� e9691150d2ea2108017ff7ecf11ead35�os�CentOS
minions/minion2/data:��pillar��grains�=�kernel�Linux�domain��uid�zmqversion�4.1.6�kernelrelease�3.10.0-514.10.2.el7.x86_64�pythonpath��/usr/bin�/src/salt�/usr/lib64/python27.zip�/usr/lib64/python2.7� /usr/lib64/python2.7/plat-linux2�/usr/lib64/python2.7/lib-tk�/usr/lib64/python2.7/lib-old� /usr/lib64/python2.7/lib-dynload�"/usr/lib64/python2.7/site-packages� /usr/lib/python2.7/site-packages�pidZ�ip_interfaces��groupname�root�fqdn_ip6��mem_total�t�saltversioninfo���
                                                             �SSDs��mdadm��id�minion2�osrelease�7.3.1611�ps�ps -efH�server_id�VY�fqdn�minion2�lsb_distrib_id�CentOS Linux�ip6_interfaces��num_cpus�hwaddr_interfaces��osfullname�CentOS Linux�ip4_interfaces��init�systemd�gid�master�saltmaster�virtual_subtype�Docker�dns��domain��sortlist��nameservers��127.0.0.11�ip4_nameservers��127.0.0.11�search��c.gtmanfred-1263.internal�google.internal�ip6_nameservers��options��ndots:0�ipv6��cpu_flags�2�fpu�vme�de�pse�tsc�msr�pae�mce�cx8�apic�sep�mtrr�pge�mca�cmov�pat�pse36�clflush�mmx�fxsr�sse�sse2�ss�ht�syscall�nx�pdpe1gb�rdtscp�lm�constant_tsc�rep_good�nopl�xtopology�nonstop_tsc�eagerfpu�pni�pclmulqdq�ssse3�cx16�sse4_1�sse4_2�x2apic�popcnt�aes�xsave�avx�hypervisor�lahf_lm�tsc_adjust�xsaveopt�localhost�minion2�ipv4��username�root�fqdn_ip4��shell�/bin/sh�nodename�minion2�saltversion�2016.11.4-28-gd8f7855�systemd��version�219�features��+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN�saltpath�/src/salt/salt�osmajorrelease�7�os_family�RedHat�oscodename�CentOS Linux 7 (Core)�osfinger�CentOS Linux-7�pythonversion��final�num_gpus�virtual�gce�disks��sda�cpu_model�Intel(R) Xeon(R) CPU @ 2.60GHz�host�minion2�pythonexecutable�/usr/bin/python2�osarch�x86_64�cpuarch�x86_64�lsb_distrib_codename�CentOS Linux 7 (Core)�osrelease_info��K�locale_info��detectedencoding�UTF-8�defaultlanguage�en_US�defaultencoding�UTF-8�gpus��path�</usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/bin:/sbin�machine_id� 8e6d36d5080d051b83d27dca6f02a13e�os�CentOS

I cleared out the cache, and then did a sync_all and it is working.

Can you check your vagrant, and set the log_level to debug, and see if the master is throwing errors for some reason?

I was unable to get it to work in ubuntu, i think there was an issue with getting the consul cache module loaded.

Thanks,
Daniel

[gtmanfred]

Ahh, i see the problem, i removed the /var/cache/salt/master/minions/ and it is not checking the cache for the matching stuff.

[gtmanfred]

For whoever works this, if you use the docker-compose environment listed above, run

docker-compose up -d
docker-compose exec saltmaster bash

then

rm -rf /var/cache/salt/master/minions
salt \* saltutil.sync_all
salt -G 'os:CentOS' test.ping

and you will see the error

(need to run the docker compose on a linux vm that has /sys/fs/cgroup mounted)

Thanks,
Daniel

[djhaskin987]

Your prompt responses on this ticket has been impressive 👍

[garethgreenaway]

@djhaskin987 When you get a moment could you see if the changes made in PR #41287 fix the issue for you? Thanks!

[djhaskin987]

Thanks for the fix, @garethgreenaway ! I'll look @ it.

[djhaskin987]

Looks a lot better, thanks!

[garethgreenaway]

@djhaskin987 Great news! If you consider the issue resolve, if you wouldn't mind closing out the issue. Thanks!