Salt: Pillar-based Salt mine doesn't refresh mine on pillar_refresh

[JensRantil]

See https://groups.google.com/forum/#!topic/salt-users/Gp_3lTu89Ho.

I haven't checked whether the mine is submitted once or continously (every 60 minutes) after a restart.

[dlanderson]

@JensRantil Did you send a saltutil.refresh_pillar command to the minion after adding the mine_functions to the pillar? That should trigger a pillar refresh and reload the scheduler modules.

[JensRantil]

@dlanderson No, I never tried that. However, I did execute state.highstate which should reload pillars right?

[basepi]

I would also recommend trying a saltutil.sync_all, as there seem to be some circumstances where refresh_pillar is not currently doing the whole job.

[JensRantil]

@dlanderson I just executed saltutil.refresh_pillar and on the host and have verified that pillar.items contains mine_functions. Querying mine directly after this does not yield any result.

I'll wait 65 minutes and will try again. If the mine hasn't been submitted by then I will issue refresh_all and see if that works. I'll get back with my investigation shortly.

[JensRantil]

Good news, executing saltutil.refresh_pillar eventually led to the minion submitting its information (I checked after 65 minutes). I suggest it submits the information on pillar_refresh instead to more easily force an update. Would you agree?

[basepi]

Ah, that would make some sense, it wouldn't update the mine until the next mine update that was scheduled. We should make it send a mine update any time the mine information in pillar changes, as you mentioned.

[basepi]

I updated the title to be more descriptive of the problem.

[rca]

I am running into a similar issue where I have restarted the minion in order to get mine data to populate on the master. I have tried saltutil.sync_all + mine.update with no success. Is there a known code issue here?

[basepi]

One thing to note is that pillar is not actually refreshed on a saltutil.sync_all. @rca does it still not update even after a saltutil.refresh_pillar call?

[rca]

@basepi I will check, will report back. However I'm sure mine.update did not update even after state.highstate, which I'm almost certain refreshes pillars.

[whiteinge]

Quick note, I can confirm this behavior on 2014.1.10.

[samos123]

I think I'm seeing the same behaviour, filed another ticket in https://github.com/saltstack/salt/issues/18505

[rallytime]

Can anyone confirm (or deny) that this is still a bug on more recent versions of salt? (Current stable is 2015.5.3.) I know that we've had several issues along these lines filed, but I thought we'd cleaned most of them up.

[ghost]

This issue is still present. I have tested this using v2015.5.0 on Ubuntu 15.04. If I run salt-call mine.update on minion, it updates the mine properly. But if I run salt '*' mine.update on master, it does not update the mine. If you fix #26599, I can test it on latest saltstack.

[timcharper]

I'm seeing this as well. I have to run a saltutil.refresh_pillar, followed by a mine.update.

[DanyC97]

has anyone seen it in 2015.8.x too ?

[surry]

@rallytime I am seeing this issue or something similar with 2015.5.5. I have a pillar-based mine function:

mine_functions:
  network.ip_addrs:
    interface: eth0

I'm using the orchestrate runner to run a mine.update before calling the highstate via salt.state in my orchestration state. The mine.update reports that it ran successfully on all minions, however there is no data in the mine when I query it.

If I restart one of my minions, then the IP address shows up for that minion in the mine.

If I run salt-call mine.update on a minion, it does update the mine. Running salt 'minion-name' mine.update on the master does not result in the mine updating.

[basepi]

@surry what happens if you do a saltutil.refresh_pillar and then a mine.update?

[JensRantil]

I just hit this again. FYI, restarting the minion process automatically submits the mine data to the master. My version: 2015.5.3

[kaihowl]

Also hit this problem. saltutil.refresh_pillar and then mine.update did the trick after all.

Nevertheless, when I set new pillar data, it is not synced for a highstate:

salt 'testnode' pillar.get testpillar # Returns empty
# I set the testpillar
salt 'testnode' state.highstate
salt 'testnode' pillar.get testpillar # Still empty
salt 'testnode' saltutil.refresh_pillar
salt 'testnode' pillar.get testpillar # Now finally returning my data

I was under the impression that a highstate would refresh the pillar data before doing the actual state application. Could this be the root cause for the whole confusion with the saltmine going on in this issue?

My version: 2015.8.1

[surry]

@basepi It looks like a saltutil.refresh_pillar followed by a mine.update is working as others have indicated above. I should be able to put these steps into my orchestration state to make sure the mine is updated before my states are applied on my minions.

[basepi]

Sounds like a good workaround for the moment. This is still on our list to get fixed.

[gibsonje]

Just encountered this on 2014.1.12.

saltutil.refresh_pillar followed by mine.update was the only solution.

[JensRantil]

This is still an issue in 2016.3.0. Restarting the minions made them submit immediately.

[sokratisg]

Is this still on the list for fixing?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[JensRantil]

@stale This is still an issue.