Salt: External binary files appear to fail in 2018.3.0

Created on 4 Apr 2018 · 13Comments · Source: saltstack/salt

Description of Issue/Question

Binary files fetched from S3 urls result in a utf-8 error.

          ID: /path/to/file
    Function: file.managed
      Result: False
     Comment: Unable to manage file: 'utf8' codec can't decode byte 0x80 in position 24: invalid start byte
     Started: 04:06:01.292323
    Duration: 39.311 ms
     Changes:

Setup

/path/to/file:
  file.managed:
    - source: "http://bucket.s3.amazonaws.com/file"
    - source_hash: "sha1=341f42c69efe03532c492968f9d5696503f8af9b"
    - mode: 755
    - user: root
    - group: root

Note that this worked in 2017.3 but started failing right when we upgraded to 2018.3.0

Steps to Reproduce Issue

highstate with the given config.

Versions Report

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1050-aws
         system: Linux
        version: Ubuntu 16.04 xenial

Bug Core P4 fixed-pending-your-verification severity-medium team-core

Source

liquidgecka

Most helpful comment

Ok, the problem is that this is trying to do a diff of a binary file, when it probably just shouldn't do that.

if you set show_changes: False on the state, it will skip trying to get a diff between the two, and will succeed.

I have marked this as a bug.

Thanks!
Daniel

gtmanfred on 4 Apr 2018

👍4

All 13 comments

Can you provide me an example file that this error happens with so that I can replicate this issue?

Thanks,
Daniel

gtmanfred on 4 Apr 2018

This is happening with every single binary (ELF mostly) file we have in our configs but I created a simple go program (https://play.golang.org/p/FKAbM-DHFi9) and uploaded it to S3 and ended up with this config:

/tmp/main:
  file.managed:
    - source: https://s3-us-west-2.amazonaws.com/mobcrush-temp/main
    - source_hash: "sha1=815220b0bfa5b530cfaa14ca03f5e04d5255cba1"
    - mode: 755
    - user: root
    - group: root

results in:

[INFO    ] Running state [/tmp/file] at time 15:36:25.160939
[INFO    ] Executing state file.managed for [/tmp/file]
[DEBUG   ] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 2569, in managed
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 5158, in manage_file
    real_name, sfn, show_filenames=False)
  File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 4921, in get_diff
    for x in fp_.readlines()])
  File "/usr/lib/python2.7/dist-packages/salt/utils/stringutils.py", line 114, in to_unicode
    return _normalize(s.decode('utf-8', errors))
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 24: invalid start byte

[ERROR   ] Unable to manage file: 'utf8' codec can't decode byte 0x80 in position 24: invalid start byte
[INFO    ] Completed state [/tmp/file] at time 15:36:25.183537 (duration_in_ms=22.598)

----------
          ID: /tmp/main
    Function: file.managed
      Result: False
     Comment: Unable to manage file: 'utf8' codec can't decode byte 0x80 in position 24: invalid start byte
     Started: 15:33:35.314459
    Duration: 129.774 ms
     Changes:

liquidgecka on 4 Apr 2018

Ok, the problem is that this is trying to do a diff of a binary file, when it probably just shouldn't do that.

if you set show_changes: False on the state, it will skip trying to get a diff between the two, and will succeed.

I have marked this as a bug.

Thanks!
Daniel

gtmanfred on 4 Apr 2018

👍4

@gtmanfred I can't get your workaround to work with contents_pillar.

       ID: /srv/server/session-ticket-keys/0.key
    Function: file.managed
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1878, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1823, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 2267, in managed
                  isinstance(use_contents, six.string_types) and '\0' in use_contents
              UnicodeDecodeError: 'ascii' codec can't decode byte 0xc6 in position 0: ordinal not in range(128)
     Started: 14:41:06.189828

I also tried forcing the system locale to utf-8 but that didn't work.

default_locale:
  locale.system:
    - name: en_US.UTF-8

Version info

Salt Version:
               Salt: 2018.3.0

    Dependency Versions:
               cffi: Not Installed
           cherrypy: Not Installed
           dateutil: 2.6.0
          docker-py: 2.3.0
              gitdb: 0.5.4
          gitpython: 0.3.2 RC1
              ioflo: Not Installed
             Jinja2: 2.10
            libgit2: Not Installed
            libnacl: Not Installed
           M2Crypto: Not Installed
               Mako: 0.9.1
       msgpack-pure: Not Installed
     msgpack-python: 0.5.6
       mysql-python: 1.2.3
          pycparser: Not Installed
           pycrypto: 2.6.1
       pycryptodome: Not Installed
             pygit2: Not Installed
             Python: 2.7.6 (default, Nov 23 2017, 15:49:48)
       python-gnupg: Not Installed
             PyYAML: 3.12
              PyZMQ: 17.0.0
               RAET: Not Installed
              smmap: 0.8.2
            timelib: Not Installed
            Tornado: 4.5.3
                ZMQ: 4.1.6

    System Versions:
               dist: Ubuntu 14.04 trusty
             locale: ascii
            machine: x86_64
            release: 4.4.0-96-generic
             system: Linux
            version: Ubuntu 14.04 trusty

epelc on 5 Apr 2018

That is a different error from the one I was mentioning in the work around above, because the one above was erroring in the get_diff function. This one is having problems picking up the file at all, and that is because no locale is set for salt.

Have you tried restarting salt after verifiying that locale is set? If that doesn't work, please open a new issue with steps to replicate this error.

gtmanfred on 5 Apr 2018

@gtmanfred It's still ascii after restarting both the minion and master. I'll make a new issue

epelc on 5 Apr 2018

@epelc also, look at your System Versions stuff, it is ascii.

    System Versions:
               dist: Ubuntu 14.04 trusty
             locale: ascii

That needs to be a unicode locale.

gtmanfred on 5 Apr 2018

@gtmanfred I saw that but I figured the locale.system was supposed to change it. It only changes it in the grains though. test.versions didn't update after changing it.

epelc on 5 Apr 2018

If you are using salt-call, you will need to log out and back in to make sure that your locale gets updated. But that is the problem, if the grains is correct, but you are still seeing the error as ascii, i am betting you are using salt-call? in which case, you do have to log out and back in to get the new local set. Otherwise, stop and start the minion, and run the command using salt * state.apply, so that it uses the daemon.

Daniel

gtmanfred on 5 Apr 2018

Hi @gtmanfred ,

Do you think you can fix-it before 2018.3.1 release ?

Many Thanks
Yann