Salt: gpg renderer includes new line when decrypting data

Created on 7 Mar 2016 · 17Comments · Source: saltstack/salt

Description of Issue/Question

When rendering gpg data in pillars a new line is included in the pillar data

Setup

#!yaml|gpg
testdata: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v2

    hQEMA/Vz8pLFN8VCAQf/fICBeA4s05X6MRPajqxKn8+PrSLMCF57xcE9//zhuAre
    vgBZCuHGzUK4TqoF7nrRPqwkJNFYdbXzIduekE5WzLnWgcidlQAnMrkdsp6reTTA
    afxoU5Q17v2mdQNe0aR30aKJJpf/lex03y8clCu9FKFCedbP6C6woqhzPJEInJVA
    fmQK8M7yfkM0U8kPHwKkOA2r5uo2NcIeVGRXVAeC/cxs2HwGgopTbvcrY3viPccy
    +00eRAkiBbYmjDm02FdJffXUO1P1vBNfbn3l6xZ86Bocwz6QIWWxnyj1fxs+nqkU
    WR5icGB6Su05Hh4p4qWXgCMAaPdFd1eRhQrUqNi9nYUBDANQxr5cKZY/aAEIAJnS
    scRsbpZXluCFY5oJv/4QDIGbzNtJYKlh/SZAvNLXlJE2GXDXva7jkxv19fUbnr3S
    Kss6YXNBPGzDhljH3tGmJFToIX3vlOpFUQMxDQASqIuUsfIIs92exIenu3nLvH9H
    fEfQygHhGoG9nutgrvObsXSQuwnv/E7rkfQpD8vhr1ES3ovgXHF8XkYXP800Pbpz
    XQJute1gIKI3sEK/319XTSXTpJrXTEGbCgmrrBSAB+qzZ850NEjFQsFPK3H7uYn5
    sADQ3oEXJpov3hNdlVR6zrboaH45wy/uRmSvbtUkIKhQjtiwHDMOKED7SisCkbaH
    tz0wykplVynqzqTRK6aFAQwDyq8jBy8IDsgBB/4lxptOxIO6LH6ifscmQMxDcSpT
    8CNHYgHZH/u0nJPHT5PL+GAaXu7h7WI7k14Gx9rHGA6x3E32XrF87BDBiJMV6gz5
    yRno+oTeFYKIxeBwiC9yT+/gYL/Fttrj/M+ugseucSp9BtA3ZGg6IPwRn7RbCZ7P
    0gRkhu/u+IYJb8FnAVBMfhigmi+a9RUVp/oa4Y4uBQFu6Qg2eR6+zDL769nhY3mk
    3t45Mc5bNelmMbFmL5Gqwy9ag1Lbey1qgtCbBOw3tI1Ul7zp6sJ7q/NqR5tNnWO2
    TV1BVtGBEHE76EYiDZR31AMYb6EEvikzjpDGbQUKlBqBfwHxocM0yXLcjF9f0kQB
    ibUSh0zIVNrNTshIbJHwn0bsfJFPSsu8ZWbuq9lJ92SlfhPJEzYVjgo+jvLLVdTG
    a0+twD7xErdX4iC7MJ7/2os02w==
    =fdB2
    -----END PGP MESSAGE-----

Steps to Reproduce Issue

salt-call --local --output=yaml -l debug  pillar.get testdata

local: 'testdata

'

_Include debug logs if possible and relevant._

Versions Report

Salt Version:
Salt: 2016.3.0rc1

Dependency Versions:
Jinja2: 2.7.3
M2Crypto: 0.21.1
Mako: 1.0.3
PyYAML: 3.11
PyZMQ: 15.2.0
Python: 2.7.11 (default, Dec 6 2015, 15:43:46)
RAET: 0.6.5
Tornado: 4.3
ZMQ: 4.1.3
cffi: 1.5.2
cherrypy: 4.0.0
dateutil: 2.5.0
gitdb: Not Installed
gitpython: 0.1.7
ioflo: 1.5.0
libgit2: Not Installed
libnacl: 1.4.4
msgpack-pure: Not Installed
msgpack-python: 0.4.7
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pygit2: Not Installed
python-gnupg: 0.3.8
smmap: Not Installed
timelib: Not Installed

System Versions:
dist:
machine: x86_64
release: 4.1.18-1-lts

_Provided by running_ salt --versions-report

Bug Needs Testcase P1 Pillar Platform severity-medium stale

Source

jagguli

Most helpful comment

This once again shows that a utility to handle the encryption of values in Pillar SLS would be quite handy, e.g. start with a plain Pillar SLS like this one in your Pillar repo:

foo/secret.sls:

foo:
  bar: some super secret

Run a command like: salt-dev pillar.encrypt_gpg key=0x0A52AC sls=foo.secret pillar=foo:bar
…and the SLS will be transformed to this:

#!yaml|gpg
foo:
  bar: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v2

    hQEOA0p+wNBgxEAxEAP/ZBYQUbTN2XyEXQPoiPlXJNcHKmas7xcjVKlAK5H/b0A+
    FICekhhM5RbW8iDjwRuVi6s/yQdF0wJjSaqG3o6JYywXUo3MGtvt2VDIswRu7mdK
    TVuj8VijrwnNVspPafw9icf16nSHqtlCh8zhkDrJzmueml7mR3wCQHw2fbXqXYQE
    AImaqcVQ5JTKGir3U/JxsrvKLTjC3ixmG7DewosFae8vP/Kr3nEr9fwJmJRzmJao
    Dg1LX3dYOr6W1UtSTw80EKJAP1IL62tqTIsA+/c1RzEMFKaOnU4KD5fkPofRQ0cc
    ye9QjjNwj/WruXBvteZc+Jn3KRX4wt17QU2NZCxI4FFGhQIMAzfltYjGxBdxAQ/+
    NmePQc2pjHxdMRuyxnT00vJq8S14QgO2CR+eZY40acnkGX2tf6lFMgeZIn801iif
    rFYumtKzcZ/B+ngtpv7dsDe5qfEbmrOx8YpRCC6gPzXvvNnwXlTAi8ktxs3VpRbS
    Bu95/RnRbKIRh6IlYkGqs7ItAkDWZZDBLCmovEP5DJyGZilXVNNwVX9qayVq7fMD
    snzzmhbpumlqztssto89RdqQg9CYxD2vKSlKaeNPwPvagQU65t/Umomut59XxwRF
    X9E8BnoKUAKR66xuJTm/BhhjUD1RK/lSt2ZNQC5TXpEJcDR25HiurAKUGVh6D1V1
    bztw6HRMlEr+29900USN8fB0baA0SaYRIRlcIyAeL2jL+9o5whnH0G7Mmuwlk5tz
    AGKKq/Co0k5gm1KnqYhjhijjM2cZWOugTCMssz6dbk4XtI4GGFJfi1w//LxhZsJc
    mSRf7DD8K3W3VAQPUWIo+XAYN34igi9JtMKaiiUlROM/DkxEB/hHdtzjAyfKSZCs
    WRetKgdBytaGHS9IcauJBWIkffVvwmQAFAfhbBvjutilsFYhYExvG6sBPb0VJq8e+m29K
    /gVPwQthQdp5GSkaDF4jmf82I5oQY6Vmegl/tkFhrrMSylNOnBrlBePzyBjj/a46
    0MMIcGmncdGH8wB8fUBO6SOM9jNReQyX3UczxYnrtgXSRgFMGVlRgjfOQtD2YbXl
    WgRekz+3eIXPELRJKh7z0U2k/SxvtVTeyRx1eNzDlREYu6RVniHe52LlibYTu6Zt
    qxBmeg6XjoA=
    =MV/8
    -----END PGP MESSAGE-----

eliasp on 15 May 2016

👍2

All 17 comments

/salt/renderers/gpg.py:Line 273

return str(decrypted_data).strip()

fixes the issue, but im not sure if thats right. looks like Popen call is returning with a new line at the end

jagguli on 7 Mar 2016

~:» gpg --version                                                                                                                                                                                                     [130] [master**] 09:07:56
gpg (GnuPG) 2.1.11
libgcrypt 1.6.5
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

jagguli on 7 Mar 2016

@jagguli, thanks for reporting.

jfindlay on 7 Mar 2016

I can replicate this behavior. I'm unclear as to why this is only impacting one particular Pillar value rather than across the board, however,.

QuinnyPig on 11 Apr 2016

fwiw, the last time we experienced a bug like this, it's because someone forgot the -n flag to echo when creating gpg blobs for pillar.

e.g.: echo "this is my rifle" | gpg --recipient [email protected] --armor --output supersecret.txt --encrypt

gpg dutifully encrypted it as-is, line-break intact. Adding back -n resolved the issue. (Granted, this was in 2015.5.3) ymmv.

vectorsigma on 13 Apr 2016

👍1

You are correct. In this case, 'cat' also had a newline appended. Crisis averted!

QuinnyPig on 13 Apr 2016

Wondering if @jagguli had the same situation?

vectorsigma on 13 Apr 2016

This once again shows that a utility to handle the encryption of values in Pillar SLS would be quite handy, e.g. start with a plain Pillar SLS like this one in your Pillar repo:

foo/secret.sls:

foo:
  bar: some super secret

Run a command like: salt-dev pillar.encrypt_gpg key=0x0A52AC sls=foo.secret pillar=foo:bar
…and the SLS will be transformed to this:

#!yaml|gpg
foo:
  bar: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v2

    hQEOA0p+wNBgxEAxEAP/ZBYQUbTN2XyEXQPoiPlXJNcHKmas7xcjVKlAK5H/b0A+
    FICekhhM5RbW8iDjwRuVi6s/yQdF0wJjSaqG3o6JYywXUo3MGtvt2VDIswRu7mdK
    TVuj8VijrwnNVspPafw9icf16nSHqtlCh8zhkDrJzmueml7mR3wCQHw2fbXqXYQE
    AImaqcVQ5JTKGir3U/JxsrvKLTjC3ixmG7DewosFae8vP/Kr3nEr9fwJmJRzmJao
    Dg1LX3dYOr6W1UtSTw80EKJAP1IL62tqTIsA+/c1RzEMFKaOnU4KD5fkPofRQ0cc
    ye9QjjNwj/WruXBvteZc+Jn3KRX4wt17QU2NZCxI4FFGhQIMAzfltYjGxBdxAQ/+
    NmePQc2pjHxdMRuyxnT00vJq8S14QgO2CR+eZY40acnkGX2tf6lFMgeZIn801iif
    rFYumtKzcZ/B+ngtpv7dsDe5qfEbmrOx8YpRCC6gPzXvvNnwXlTAi8ktxs3VpRbS
    Bu95/RnRbKIRh6IlYkGqs7ItAkDWZZDBLCmovEP5DJyGZilXVNNwVX9qayVq7fMD
    snzzmhbpumlqztssto89RdqQg9CYxD2vKSlKaeNPwPvagQU65t/Umomut59XxwRF
    X9E8BnoKUAKR66xuJTm/BhhjUD1RK/lSt2ZNQC5TXpEJcDR25HiurAKUGVh6D1V1
    bztw6HRMlEr+29900USN8fB0baA0SaYRIRlcIyAeL2jL+9o5whnH0G7Mmuwlk5tz
    AGKKq/Co0k5gm1KnqYhjhijjM2cZWOugTCMssz6dbk4XtI4GGFJfi1w//LxhZsJc
    mSRf7DD8K3W3VAQPUWIo+XAYN34igi9JtMKaiiUlROM/DkxEB/hHdtzjAyfKSZCs
    WRetKgdBytaGHS9IcauJBWIkffVvwmQAFAfhbBvjutilsFYhYExvG6sBPb0VJq8e+m29K
    /gVPwQthQdp5GSkaDF4jmf82I5oQY6Vmegl/tkFhrrMSylNOnBrlBePzyBjj/a46
    0MMIcGmncdGH8wB8fUBO6SOM9jNReQyX3UczxYnrtgXSRgFMGVlRgjfOQtD2YbXl
    WgRekz+3eIXPELRJKh7z0U2k/SxvtVTeyRx1eNzDlREYu6RVniHe52LlibYTu6Zt
    qxBmeg6XjoA=
    =MV/8
    -----END PGP MESSAGE-----

eliasp on 15 May 2016

👍2

@techhat @cachedout fyi @eliasp suggestion which sounds great imho.

DanyC97 on 17 May 2016

Nope echo -n is not enough just tested against

salt/srv:» salt --versions-report                                                                                                                                                                                        [2] [master*] 01:49:34
/usr/local/lib/python2.7/dist-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
/usr/local/lib/python2.7/dist-packages/cffi/model.py:526: UserWarning: 'git_merge_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
Salt Version:
           Salt: 2016.3.0

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 6.0.1
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 1.2.3
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist: debian 8.5
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Linux
        version: debian 8.5

have an outdated fix here https://github.com/StreetHawkInc/salt/commit/86c9217054d787c1180109ce5cba02d2d5566153

jagguli on 9 Jun 2016

@jagguli I spent a while working on encrypted pillars with newlines last night as well. I haven't thoroughly tested this for all situations, but I was able to encrypt multiline and single line successfully with this script:

https://gist.github.com/WillPlatnick/1dbe859f552390b778f9453098dfd081

WilliamDurin on 9 Jun 2016

👍1

I faced the same issue but it turns out that I misused echo command. Thanks everyone for shared workarounds.

vitalyisaev2 on 4 Oct 2016

When using echo I run it as echo -n so that it doesn't add the new line... GPG simply encrypts what it is given and echo with the -n option will add a LF to the output.

jbouse on 4 Oct 2016

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] on 6 Jul 2018

In case anyone ever finds this issue, this utility would likely help a lot. It will take a fully unencrypted yaml pillar file, and gpgify it for you.

vectorsigma on 18 Nov 2018

It seems like 2018.3.4 still has the same (?) issue.

State example:

Slack Client Secret:
  file.managed:
    - name: /etc/xxx/oauth.slack.client.secret
    - user: xxx
    - group: xxx
    - mode: 600
    - contents_pillar: xxx:slack:client_secret
    - contents_newline: False

Pillar is stored in GPG format, generated by echo -n mysecret | gpg -ae (no newline in the end of string, checked by cat mysecret.asc | gpg | hexdump -C).

Applying the state leads to the change when 0x0a is being added at the end of existing file:

          ID: Slack Client Secret
    Function: file.managed
        Name: /etc/xxx/oauth.slack.client.secret
      Result: None
     Comment: The file /etc/xxx/oauth.slack.client.secret is set to be changed
     Started: 06:59:02.427698
    Duration: 1.616 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -mysecretcode+mysecretcode

Versions:

# salt --versions
Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 2.0.8
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1090-aws
         system: Linux
        version: Ubuntu 16.04 xenial

It seems like contents_newline does not have any effect in the above case. Trying to get more details now, appreciate any input.