salt-master on CentOS 7 using the Python 3 packages cannot use pygit2 via normal means

[pstengel]

Description of Issue/Question

As has been dealt with in a few different issues, EPEL recently changed from Python 3.4 to Python 3.6. Salt packages have been updated, and new installs of Salt using PY3 instructions here work.

However, pygit2 is not provided by EPEL or salt-py3-repo for Python 3.6. Additionally, the Python 3.6 version of Salt installed by salt-py3-repo on CentOS 7 specifies an interpreter of #!/usr/bin/python3.6 -s. The -s argument avoids loading the user's site directory to syspath and as such, using pip to install pygit2 for Python 3.6 will not work.

It seems that salt-py3-repo should provide a python36-pygit2 package as it does some other python36-* packages, or the -s argument should be removed so that users can install required packages not provided by the repositories.

Setup

• Bring up a clean install of CentOS 7
• Install EPEL following the instructions here
• Install the latest salt-master release (2019.2.0 as of this report) using the instructions for "Redhat / CentOS 7 PY3" here
• Install pip for Python 3.6 (yum install python36-pip)
• Install pygit2 via pip for Python 3.6 (pip3.6 install pygit2)

Steps to Reproduce Issue

• Run salt-master -V and observe that pygit2 is unavailable
• Observe that python36-pygit2 is not available from EPEL or salt-py3-repo
• head -1 "$(which salt-master)" shows the restrictive interpreter that does not allow the version of pygit2 installed by pip to function for this particular version of Salt

Versions Report

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.6 (default, Mar 29 2019, 00:03:27)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

[dmurphy18]

@pstengel There have been requests that SaltStack package pygit2 and its associated libgit2 in the past and they have been rejected due to nature of pygit2 and libgit2 versions having to be tied together, and how that can be comprised when updates are made to pygit2 and libgit2 outside of the versions that SaltStack would provide.

As for removing -s, I cannot comment on that.

[Ch3LL]

@dmurphy18 isnt the python3.6 installed by epel and not salt so we cannot manage that either right? Am i understanding it correctly as to why you cannot comment on removing -s?

[pstengel]

I can confirm that python3.6 is installed/managed by EPEL. I'm not sure of the reason for the inclusion of the -s in the interpreter. I'm having trouble even locating where that's done during the install of Salt from the packages.

[dmurphy18]

@pstengel @Ch3LL The '#!/usr/bin/python3.6 -s' is generated by the build process on RHEL 7 from the spec file: the original line for example: scripts/salt-minion contains '#!/usr/bin/env python'.

The spec file is available for 2018.3.4 from https://github.com/saltstack/salt-pack-py3/blob/develop/file_roots/pkg/salt/2018_3_4/rhel7/spec/salt.spec , see Line 586.

[Marwel]

I have 2019.2.0 running before the update to python3.6 and it is working with the -s flag for python3.4. I also have installed pygit2 via pip. As soon as i do the update to the latest version of 2019.2.0, then it pulls in python3.6 and salt-master dies on startup, claiming to have no pygit2 available.

[thuhak]

I met this problem today.

You may rewrite systemd startup service file ,add environ settings
to fix it temporarily.

/etc/systemd/system/salt-master.service

.include /usr/lib/systemd/system/salt-master.service

[Service]
Environment=PYTHONPATH=/usr/lib/python3.6/site-packages:/usr/lib64/python36.zip:/usr/lib64/python3.6:/usr/lib64/python3.6/lib-dynload:/usr/lo

then

systemctl disable salt-master && systemctl enable salt-master

[Marwel]

@thuhak
your service description ends at "/usr/lo", which looks like it is not complete.
This addin for systemd could be a simple workaround for now. But i don't understand, why it is not needed for 3.4.

[wildk1w1]

This is also an issue on the salt-minion when using pip installed packages such as pyinotify under CentOS 7.
It is required to use a pip installed package for this, as the CentOS supplied one is too old a version for the inotify beacons in Salt.

CentOS7 sets python36 PIP up to install to /usr/local/lib/python3.6/site-packages

Python 3.6 with salt 2018.3.4
Python 3.6 with salt 2019.2.0

On the minion the following files contain the -s command line option in the first line of the files.
/usr/bin/salt-minion
/usr/bin/salt-call
/usr/bin/salt-proxy

[wildk1w1]

Would it be useful to have the _-P_ bootstrap installer script flag, remove the -s from the first line in the /usr/bin/salt-* scripts?
After all it is an indicateion that PIP will be used.

[lorengordon]

Following. Anyone got a way ahead on this? Is this indeed a bug in salt's py3 packaging, or is it intentional? pip3 installs into /usr/local/lib/python3.6/site-packages, but ... 😢

[root@ip-172-31-41-91 ~]# python3 -c 'import sys; print("\n".join(sys.path))'

/usr/lib64/python36.zip
/usr/lib64/python3.6
/usr/lib64/python3.6/lib-dynload
/usr/local/lib/python3.6/site-packages
/usr/lib64/python3.6/site-packages
/usr/lib/python3.6/site-packages

[root@ip-172-31-41-91 ~]# python3 -s -c 'import sys; print("\n".join(sys.path))'

/usr/lib64/python36.zip
/usr/lib64/python3.6
/usr/lib64/python3.6/lib-dynload
/usr/lib64/python3.6/site-packages
/usr/lib/python3.6/site-packages

[Ch3LL]

ping @dmurphy18 can you clarify on @lorengordon 's question "Is this indeed a bug in salt's py3 packaging, or is it intentional?"

[ArthurREGNARD]

Hello,

UP : #55037

Best regards,

[dmurphy18]

@lorengordon Sorry for the delay in answering this, seeing it now, I missed a lot of September due to personal reasons. The '-s' is generated by the py3_install macro used when building the packages. This is the recommended macro to use from Redhat when building packages for their platform. SaltStack is following their recommendation when building packages.

As for libgit2 and pygit2, I refer to the answer given above as to why SaltStack does not package them. I note that I don't see a Python 3 version of python-pygit2 from either Redhat or EPEL, but SaltStack has no current plans for porting python-pygit2 to Python 3.

[lorengordon]

@dmurphy18 The issue affects the usage of pip.installed also. The state will install packages to /usr/local/lib/python3.6/site-packages by default. But due to the -s those packages are not available to salt, so modules that depend on those packages cannot be loaded. Overriding the target works, but not sure it's a good idea. E.g...

install pip packages:
  pip.installed:
    - pkgs:
      - pygit2
    - target: /usr/lib/python3.6/site-packages
    - reload_modules: True
    - ignore_installed: True

[dmurphy18]

@lorengordon FYI with the next release, shall look at not using the Redhat recommended macro py3_install and instead different command to perform the Py 3 install and lose the '-s'. The next release should be soon too.

[dmurphy18]

FYI The '-s' flag is getting added due to use of py3_build macro using the rpmbuild tools from Redhat. There are plans to not use py3_build but it's expanded equivalent without the '-s' in the next point release when building for the Redhat family (which includes Amazon Linux 1 & 2). BTW: The next point release will be soon.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[lorengordon]

not stale

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[dmurphy18]

@pstengel This should be fixed in the Neon release

[sagetherage]

@pstengel can you verify the fix?

[sagetherage]

@lorengordon are you able to verify the fix? @dmurphy18 can you provide the details to verify, please?

[pstengel]

@sagetherage Are there development RPMs somewhere, or would I need to build them myself?

[sagetherage]

@pstengel we were a little late on updating the packages to repo.saltstack.com, but they are now available, apologies! we are working on that to be better and faster.

[wb529kq]

Anyone have a workaround for this until its officially fixed ?

[pere3]

@wb529kq
i'm doing this dirty stuff, but it gets the job done:

00-ops.fix_python36_centos7_salt-file.replace:
  file.replace:
    - name: /usr/bin/salt-minion
    - pattern: '#!/usr/bin/python3[\.]?[6]? -s'
    - repl: '#!/usr/bin/env python3.6'
01-ops.fix_python36_centos7_salt-file.replace:
  file.replace:
    - name: /usr/bin/salt-call
    - pattern: '#!/usr/bin/python3[\.]?[6]? -s'
    - repl: '#!/usr/bin/env python3.6'
02-ops.fix_python36_centos7_salt-file.replace:
  file.replace:
    - name: /usr/bin/salt-proxy
    - pattern: '#!/usr/bin/python3[\.]?[6]? -s'
    - repl: '#!/usr/bin/env python3.6'
03-ops.fix_python36_centos7_salt-cmd.run:
  cmd.run:
    - name: |
        exec 0>&- # close stdin
        exec 1>&- # close stdout
        exec 2>&- # close stderr
        nohup /bin/sh -c 'salt-call --local service.restart salt-minion' &
    - onchanges:
      - file: 00-ops.fix_python36_centos7_salt-file.replace

[ArthurREGNARD]

@wb529kq

I get a python script:

#!/usr/bin/env python

# Python Libs
import os
import subprocess


SALT_BIN_PATH = "/usr/bin"
FIRST_LINE = "#!/usr/bin/python3"
SERVICES_TO_RESTART = ['salt-master', 'salt-minion']


files_to_patch = []
for file in os.listdir(SALT_BIN_PATH):
    if file.startswith('salt'):
        files_to_patch.append(os.path.join(SALT_BIN_PATH, file))

for file_to_patch in files_to_patch:
    print('patch : {}'.format(file_to_patch))
    new_file = [FIRST_LINE]
    with open(file_to_patch, 'r') as file:
        file_content = file.read().split(os.linesep)
        file_content.pop(0)
        new_file.extend(file_content)
    with open(file_to_patch, 'w') as file:
        file.write(os.linesep.join(new_file))

for service in SERVICES_TO_RESTART:
    try:
        print('restarting {} service'.format(service))
        subprocess.check_call(['systemctl', 'restart', service])
    except subprocess.CalledProcessError as e:
        ploter(messages=["Commande    : {}".format(e.cmd)], level='ERROR')
        ploter(messages=["Return code : {}".format(e.returncode)], level='ERROR')
        ploter(messages=["OutPut      : {}".format(e.output)], level='ERROR')
        exit(1)

print("Your salt bin was updated, and services restarted.")

On the cluster run:

sudo python <correct_install>.py

[dmurphy18]

@pstengel @wb529kq Just checked again Neon RC2 on Redhat 7, from the first few lines of salt-minion

root@localhost:/usr/bin# cat salt-minion

!/usr/bin/python3

'''
This script is used to kick off a salt minion daemon
'''

same for salt-call, salt-master, salt-api, etc.

root@localhost:/usr/bin# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
root@localhost:/usr/bin#

[pstengel]

@sagetherage I don't see development/Neon RPMs on repo.saltstack.com. Browsing through here, I only see 2019.2.3 as the latest available version. What am I missing?

In any case, it looks like @dmurphy18 confirmed the fix as long as they used the RPMs to install RC2. This interpreter issue was never a problem installing via non-RPM methods.

[dmurphy18]

@pstengel Can you recheck, Salt 3000 with the fix went live on Monday 10th Feb 2020. The RC's where in salt_rc off of repo.saltstack.com, the usual place for RC's, for example: http://repo.saltstack.com/salt_rc/py3/redhat/7/x86_64/

[pstengel]

[root@447def22f311 /]# salt-master -V
Salt Version:
           Salt: 3000.0.0rc2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug  7 2019, 17:28:10)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 14.7.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.7.1908 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.19.76-linuxkit
         system: Linux
        version: CentOS Linux 7.7.1908 Core

[root@447def22f311 /]# pip3.6 install pygit2
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting pygit2
  Downloading https://files.pythonhosted.org/packages/98/18/2dd7840bf1816f1b40ad6d07e5203acb34497aae11487a6a2b67661cc1e4/pygit2-1.0.3-cp36-cp36m-manylinux1_x86_64.whl (5.1MB)
    100% |################################| 5.1MB 280kB/s
Collecting cffi (from pygit2)
  Downloading https://files.pythonhosted.org/packages/f1/c7/72abda280893609e1ddfff90f8064568bd8bcb2c1770a9d5bb5edb2d1fea/cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (399kB)
    100% |################################| 399kB 3.4MB/s
Collecting pycparser (from cffi->pygit2)
  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
    100% |################################| 163kB 5.6MB/s
Installing collected packages: pycparser, cffi, pygit2
  Running setup.py install for pycparser ... done
Successfully installed cffi-1.14.0 pycparser-2.19 pygit2-1.0.3
[root@447def22f311 /]# salt-master -V
Salt Version:
           Salt: 3000.0.0rc2

Dependency Versions:
           cffi: 1.14.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.8.1
        libgit2: 0.28.4
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: 1.0.3
         Python: 3.6.8 (default, Aug  7 2019, 17:28:10)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 14.7.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.7.1908 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.19.76-linuxkit
         system: Linux
        version: CentOS Linux 7.7.1908 Core

[root@447def22f311 /]# head -1 "$(which salt-master)"
#!/usr/bin/python3

@dmurphy18 Looks good!

[dmurphy18]

@pstengel If you are happy, can you consider closing this issue. Thanks, and sorry for the delay in getting this out there, Neon was delayed further than I thought it would be