Salt: Thorium deletes keys after master or minion restart

[yannispanousis]

Description of Issue/Question

Hi,

I'm attempting to implement the basic use case of Thorium to cleanup dead minion keys.

From a clean slate, everything works fine the first time.

However, when the master or the minion is restarted, then the minion's key is rejected soon after, with messages:

2017-03-08 22:52:43,308 [salt.master][INFO][13854] Rotating master AES key
2017-03-08 22:52:44,311 [salt.state][INFO][13852] Running state [startreg] at time 22:52:44.311613
2017-03-08 22:52:44,311 [salt.state][INFO][13852] Executing state status.reg for startreg
2017-03-08 22:52:44,312 [salt.state][INFO][13852] No changes made for startreg
2017-03-08 22:52:44,312 [salt.state][INFO][13852] Completed state [startreg] at time 22:52:44.312167 duration_in_ms=0.555
2017-03-08 22:52:44,312 [salt.state][INFO][13852] Running state [keydel] at time 22:52:44.312332
2017-03-08 22:52:44,312 [salt.state][INFO][13852] Executing state key.timeout for keydel
2017-03-08 22:52:44,312 [salt.state][INFO][13852] No changes made for keydel
2017-03-08 22:52:44,312 [salt.state][INFO][13852] Completed state [keydel] at time 22:52:44.312874 duration_in_ms=0.542
2017-03-08 22:52:51,755 [salt.master][WARNING ][13865] Salt minion claiming to be ip-10-1-22-105.ec2.internal attempted to communicate with master but key could not be read and verification was denied.
2017-03-08 22:52:51,755 [salt.master][WARNING ][13865] Minion id ip-10-1-22-105.ec2.internal is not who it says it is!

It seems related to the AES key rotation that is happening shortly before.

I would really appreciate any ideas!

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

I'm using the configuration described in the docs linked above.

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

• Install master with the Thorium config described in the docs for minion cleanup
• Install minion with the Thorium config for minion cleanup
• Restart minion or master
• Wait for key to be removed

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Master:

Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-48-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Minion:

Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: 1.9.1
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: 2.17
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-48-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[yannispanousis]

I also tried 2016.11.3 but had the same issue :(

[techhat]

Hi @yannispanousis. I had the same issue a few days ago. This is resolved with #39858.

[yannispanousis]

Great! @techhat Does this mean it should be in 2016.11.4?

[techhat]

I'm guessing so, but I'm not in the release loop.

[gtmanfred]

Yup, this will be in 2016.11.4!

Thanks @techhat.

I am going to close this issue.