Helmfile: Temporary value files not deleted

[mukgupta]

I am using helmfile for deploying our charts. On our jenkins machines, we see lot of temporary (/tmp/values*) files in /tmp folder. From what i understand, these are rendered value files generated when we run helmfile apply. The problem is that these files contain sensitive information like passwords, api keys in plain text which is a security risk. It'd be ideal if these files get deleted after helmfile apply operation is complete.

[mumoshu]

@mukgupta Thanks for reporting!

That's odd - I thought we had the cleanup logic in place not to leave sensitive info like that.

Would you mind confirming if it happens always, or only when helmfile apply succeeds, or failed, or else?

[mukgupta]

@mumoshu : Just verified this. Value files persist even after a successful diff/apply operation. I also noticed that files are created twice for a single apply operation. That's probably because apply operation runs diff as well.

[mumoshu]

@mukgupta Thanks again for your support ☺️

I dig deeper, and found that helmfile had been missing cleaning up the generated file when the values entry was a file path. That is:

values:
- foo.yaml

and

values:
- foo.yaml.gotmpl

resulted in remaining files.

Inline values under values: and secrets are unaffected. Fortunately, it wasn't a security issue, as long as you have never put sensitive info in raw values.yaml files(and you 'd never!).

So just to confirm,

Value files persist even after a successful diff/apply operation. I also noticed that files are created twice for a single apply operation. That's probably because apply operation runs diff as well.

You are definitely correct on these points!

diff and sync generates temporary values files. A temp values file is generated even when it isn't suffixed with .gotmpl. It isn't ideal but it's how helmfile is implemented at the moment. apply is basically diff and then sync so you are also correct that it generates temp files twice.

[mumoshu]

Closing this as fixed via #520

[mukgupta]

@mumoshu : Just checked version: v0.51.0 . The issue is not completely resolved. Value files still persist if I run just the diff operation.

[mumoshu]

I'll take a look into that. Thanks for reporting!
2019年4月1日(月) 16:58 Mukesh Gupta [e-mail]:

@mumoshu https://github.com/mumoshu : Just checked version: v0.51.0 .
The issue is not resolved. Value files still persist after running
diff/apply operation

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/roboll/helmfile/issues/503#issuecomment-478475619,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-QcdqTIUYl_wChuL2t79mqX4NzEUks5vcbwJgaJpZM4cFAe1
.

[mumoshu]

I believe #525 fixes it. Would you mind confirming?

[mukgupta]

@mumoshu Yes this works. Thanks for the fix

[mumoshu]

@mukgupta Nice! Thanks for confirming ☺️

[guillomep]

@mumoshu it looks like the 0.79.1 version is still affected by this bug.
What I observed is when you use files for values and secrets, some temporary files still exist at the end:

• When using template, secret and value files are still here after
• When using sync, destroy, apply or diff, secret files are still here after

So secret files are always present, and value files just when using the template command.
It doesn't matter for values since it doesn't contains sensible data, but it is a problem for secret files.

[mumoshu]

@guillomep Thanks for reporting! I'll take a look asap

[guillomep]

@mumoshu the problem still exists today with the last version of helmfiles (v0.116.0)

[savar]

and it also exists on secrets.. I have the sops secured secrets in plaintext in /tmp/secret*

[savar]

secrets from releases seem to be cleaned.. as they are appended to release.generatedValues which is being cleaned on

// Clean will remove any generated secrets
func (st *HelmState) Clean() []error {

but secrets on the environment level seem to stay where they are
.. also they seem to exist multiple times.. a diff on a specific release name produced at least on one of the secret files 2 versions of it

[mumoshu]

@savar Thanks! I had no quick way to test this but anyway #1304 should fix that.

Would you mind testing it out, if possible? FYI, checking the code out and then running make build will produce ./helmfile for testing.

[savar]

# helmfile.yaml
# shortened so there are more secret files in the `environments` in reality,
# so please don't count  `/tmp/secret*` results in the shell output and
# compare it to the `secrets:` of the environment

environments:
  myenv:
    values:
      - values/clusters/global/values.yaml
      - values/clusters/non-prod/values.yaml
    secrets:
      - values/clusters/non-prod/secrets.yaml
      - values/clusters/non-prod/stage/secrets.yaml

- name: elasticsearch-curator
  namespace: logging
  version: 1.0.0
  chart: ./charts/{{`{{ .Release.Name }}`}}
  values:
    - releases/{{`{{ .Release.Name }}`}}/values.yaml.gotmpl
  secrets:
    - releases/{{`{{ .Release.Name }}`}}/secrets.yaml

# releases/elasticsearch-curator/secrets.yaml
test: the helmfile cleanup

Test run with version v0.118.5:

$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/secret*': No such file or directory
ls: cannot access '/tmp/value*': No such file or directory
$ helmfile -e myenv -l chart=elasticsearch-curator diff --skip-deps
...
$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/value*': No such file or directory
 /tmp/secret284287480   /tmp/secret354416812   /tmp/secret504619585   /tmp/secret523281655   /tmp/secret546566122   /tmp/secret572061117
$ grep 'the helmfile cleanup' /tmp/secret*; echo $?
1
$ rm /tmp/secret* /tmp/value*
$ helmfile -e myenv -l chart=elasticsearch-curator apply --skip-deps
...
$ ls /tmp/secret* /tmp/value*
/tmp/secret008771460  /tmp/secret482589459  /tmp/secret755320441  /tmp/secret791765103
/tmp/secret332955984  /tmp/secret646731010  /tmp/secret782093048  /tmp/values722865341
$ grep 'the helmfile cleanup' /tmp/secret*; echo $?
/tmp/secret782093048:test: the helmfile cleanup
0

so actually the cleanup of "releases" seems to not happen on apply properly as well but on diff (weird enough). Neither for values nor for secrets. To mention is: there is no diff nor change happening. Just in case this could change the flow within the helmfile so that the cleanup isn't triggered on an apply without a change or something like that :man_shrugging:

Now the cleanup-decrypted-env-secrets-files branch version:

$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/secret*': No such file or directory
ls: cannot access '/tmp/value*': No such file or directory
$ path/to/selfcompiled/helmfile -e myenv -l chart=elasticsearch-curator diff --skip-deps
...
$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/secret*': No such file or directory
ls: cannot access '/tmp/value*': No such file or directory
$ path/to/selfcompiled/helmfile -e myenv -l chart=elasticsearch-curator apply --skip-deps
...
$ ls /tmp/secret* /tmp/value*
/tmp/secret115335158  /tmp/values139239475
$ grep 'the helmfile cleanup' /tmp/secret*; echo $?
test: the helmfile cleanup
0

so it looks your changes fixes the environment related issues (nice! :smiley: ) but as it seems I was assuming
too much on the releases.. it works on diff but not on apply (the cleanup).. do you need more information or is this enough for you to dig deeper (as I didn't test other things like template, lint, sync or test or any other which might generate the secrets/values) for the other issue as well?

[mumoshu]

@savar Hey! Thank you so much for your support.

After digging deeper based on your feedback, I realized that the source of the issue has been there since early days of helmfile. I changed the way how the cleanup is done, and it's now live at #1304.

I'll try to test it myself in my spare time, but I'd also appreciate it if you could test it out! Thanks again for your support.

[guillomep]

It seems to work with all the commands for me, but in my test I only have environment secrets

[savar]

I will redo my test run the same way and report back ASAP

[savar]

$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/secret*': No such file or directory
ls: cannot access '/tmp/value*': No such file or directory
$ path/to/selfcompiled/helmfile -e myenv -l chart=elasticsearch-curator diff --skip-deps
...
$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/secret*': No such file or directory
ls: cannot access '/tmp/value*': No such file or directory
$ path/to/selfcompiled/helmfile -e myenv -l chart=elasticsearch-curator apply --skip-deps
...
$ ls /tmp/secret* /tmp/value*
ls: cannot access '/tmp/secret*': No such file or directory
ls: cannot access '/tmp/value*': No such file or directory

also tested template, sync and test and none showed any secret or value files in /tmp .. thanks so much @mumoshu