Vault: Reload template on Change in Vault agent.

[woosley]

Is your feature request related to a problem? Please describe.
in Kubernetes, we use vault agent as sidecar to render the template, which is mounted to vault agent's local disk via configmap, to the real application configuration file. When we update the configmap, it looks like vault agent can not re-render vault template when the configmap content get changed. we are required to restart the application Pod

Describe the solution you'd like
Watch the template file change on the disk, whenever it changes, reload it from disk. In this way, we can integrate with other solutions to automatically refresh the applications configuration.

[tyrannosaurus-becks]

Hi! I was wondering if the wait command documented here might help you with this? It's obviously more of a polling approach than an event-driven approach, but might help in the meantime?

[CommanderTso]

Watching for template changes and reloading would be excellent. We are trying to use our internal Vault instance, with Vault Agent on the client machine, to fill in ElastAlert rule templates. We're having to dive through all sorts of hoops (and still haven't made it work) to get Vault to fill in the secrets when the templates get refreshed on the client machine.

[woosley]

will this be included in some roadmap ?

[ncabatoff]

Hi @woosley,

As far as I know Vault Agent is already watching files on disk and re-rendering templates when they change. If that's not working, it may be a bug, or it may be something more on the kubernetes side. I know it can take several minutes for a configmap change to appear in a pod's mount. Can you confirm that you saw the file updated in the pod and that Agent failed to subsequently re-render the template?

[woosley]

when is this feature added? I can double check but based on the feedback from other users in this channel, re-rendering template is not implemented.

[woosley]

below is a screenshot about this not working, configmap is updated to test123 but in the rendered file, it is still test.

[ncabatoff]

Just to double check, when you say "config map is updated to test123", you mean you verified that the configmap mounted in the pod is updated, right?

[woosley]

yes, please see the the screenshoot above, configmap is mounted to /etc/template0files/settings.json and the highlighted red box is "ES_USERNAME_test123", but the rendered settings json is still "ES_USERNAME_test“

[ncabatoff]

Perhaps this is a kubernetes-specific issue. I haven't used k8s in a while, maybe the fsnotify code consul-template uses doesn't work with the configmap mount fs? What does mount report if you get a shell in the pod? Or maybe it's as described here, the issue is that the configmaps are symlinks? Could you also show me the ls -l output for one of your mounted configmaps?

I'm adding a test (in #9097) because although consul-template already verifies the re-render behaviour, it's good to also validate that Agent does the same. So I'm confident this works on "normal" filesystems with regular files.

[woosley]

@ncabatoff can you point me the code where vault agent implement this notify and reload mechanism?

[woosley]

here is an output of config map file. it is indeed a syslink

[ncabatoff]

@ncabatoff can you point me the code where vault agent implement this notify and reload mechanism?

I'm afraid I cannot. It would be within consul-template, which we use as a library. It's not something I've ever looked at. I took a look just now to see if I could help you, but it's not obvious. One thing I did see is that I may have misrepresented how it works: I said it uses fsnotify, but I found comments suggesting the opposite, like

    // FileQuerySleepTime is the amount of time to sleep between queries, since
    // the fsnotify library is not compatible with solaris and other OSes yet.

So I think it's actually polling, and the fix would probably be to switch an os.Stat for an os.Lstat. But searching for os.Stat I wasn't able to definitively say which code is relevant.

[woosley]

by reading https://golang.org/pkg/os/#Lstat, os.Lstat does not follow link, but os.Stat does, so the correct one should be os.Stat?

I am looking at https://github.com/hashicorp/consul-template/blob/d3b6838a3ad8c82b60b5416c733595d6db4f7ea1/dependency/file.go#L101 , it seems consul template is already using the correct implementation.

[ncabatoff]

Hmm, you're right, I was confused. Perhaps the issue isn't symlink related at all. Could you try running the agent with log level debug and share the logs?

[woosley]

it is definitely a bug, below Debug logs showing that the file is re-rendered

However, if you look at the file content, the content is not updated