Salt: [python3] Uncaught exception while updating s3fs fileserver cache

Created on 27 May 2019 · 6Comments · Source: saltstack/salt

Description of Issue/Question

During startup, the following error appears:

[ERROR   ] Uncaught exception while updating s3fs fileserver cache
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/master.py", line 459, in _do_update
    update_func(*args)
  File "/usr/lib/python3.7/site-packages/salt/fileserver/s3fs.py", line 126, in update
    metadata = _init()
  File "/usr/lib/python3.7/site-packages/salt/fileserver/s3fs.py", line 366, in _init
    metadata = _refresh_buckets_cache_file(cache_file)
  File "/usr/lib/python3.7/site-packages/salt/fileserver/s3fs.py", line 551, in _refresh_buckets_cache_file
    pickle.dump(metadata, fp_)
TypeError: write() argument must be str, not bytes

Setup

Relevant part of master config:

s3:
  keyid: myuser
  key: mykey
  buckets:
    base:
      - mybucket
  service_url: minio.example.com
  path_style: True
  https_enable: True

fileserver_backend:
  - rootfs
  - s3fs

Steps to Reproduce Issue

Install salt-master (latest, on alpine edge, with python3.7).
Set up s3 (as per above).
Run salt-master.

Versions Report

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.28.1
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.28.1
         Python: 3.7.3 (default, Apr 17 2019, 11:48:37)
   python-gnupg: Not Installed
         PyYAML: 5.1
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 4.19.41-0-vanilla
         system: Linux
        version: Not Installed

Some minions are on version 2018.3.3 (latest non-edge), but we don't really get to that point.
Apologies on the edit spam - I accidentally clicked "enter" while writing the bug title, and figured it'd be less invasive to simply edit rather than close and open a new one.

Confirmed team-core

Source

CosmicToast

Most helpful comment

Diff to fix it:
_salt/fileserver/s3fs.py, line 550_

<     with salt.utils.files.fopen(cache_file, 'w') as fp_:
---
>     with salt.utils.files.fopen(cache_file, 'wb') as fp_:

jerem991 on 23 Jan 2020

👍4 🎉2

All 6 comments

Update: I'm seeing a similar failure on Ubuntu 18.04.2 LTS (this time at work). Same information set:

The error shown is a bit different (likely due to the older version of salt):
[ERROR ] Exception write() argument must be str, not bytes occurred in file server update

Setup

s3:
  buckets.base:
    - mybucket
  https_enable: True
  key_id: my_id
  key: mykey
  service_url: zone.digitaloceanspaces.com

fileserver_backend:
  - roots
  - s3fs

Versions Report

Salt Version:
           Salt: 2017.7.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.7 (default, Oct 22 2018, 11:32:17)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-50-generic
         system: Linux
        version: Ubuntu 18.04 bionic

This very much seems to be a python3 compatibility problem.

CosmicToast on 28 May 2019

@5paceToast Thanks for reproducing this on Ubuntu (officially supported platform). Agree appears to be a Python 3 compatibility issue, even thought 2017.7.4 does not have full support for Python 3, that came in 2018.3.0, the same issue is occurring on 2019.2.0 with Alpine.

Appear to have missed this file when updating to allow for Python 3 and encode/decode

dmurphy18 on 28 May 2019

We've run into this as well on Debian 9, Debian 10 and Windows Server 2016 (version 1607) on 2019.2.0. We've had to use the Python 2 version for Windows, and the Debian apt repository at repo.saltstack.com also packages the Python 2 version, which we've had to use in preference over the version packaged by Debian themselves (which uses Python 3 and is similarly broken in its support for s3fs).

As SaltStack know, Python 2 is EOL as of January 1, 2020. Please fix this issue before then so we aren't caught out with unpatched security vulnerabilities, as the UK NCSC have warned: https://www.ncsc.gov.uk/blog-post/time-to-shed-python-2

And hopefully your bot doesn't come along and close this issue as a "stale" issue either!

Prototyped on 29 Aug 2019

Diff to fix it:
_salt/fileserver/s3fs.py, line 550_

<     with salt.utils.files.fopen(cache_file, 'w') as fp_:
---
>     with salt.utils.files.fopen(cache_file, 'wb') as fp_:

jerem991 on 23 Jan 2020

👍4 🎉2

Any chance we can get the fix proposed by jerem991 checked in? This seems to work for a new Py3 installation on 3000.

shorawitz on 26 Feb 2020

I just wanted to note that I just ran into this too.
Same error message, and the same fix.
Would love to see this merged in.

Running Python3.7 on 3000.2:

Salt Version:
           Salt: 3000.2

Dependency Versions:
           cffi: 1.14.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: 0.28.4
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 1.0.3
         Python: 3.7.6 (default, Feb 26 2020, 20:54:15)
   python-gnupg: Not Installed
         PyYAML: 4.2
          PyZMQ: 17.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.3

System Versions:
           dist:   
         locale: UTF-8
        machine: x86_64
        release: 4.14.173-137.229.amzn2.x86_64
         system: Linux
        version: Not Installed