Salt: Salt Master won't start when using FreeBSD and PyGit2 (renamed)

Created on 9 Oct 2017  路  7Comments  路  Source: saltstack/salt

Description of Issue/Question

Trying to use a private bitbucket repo as the backend for gitfs causes the master to fail to start.

Setup

# .../salt/master
fileserver_backend:
  - git

gitfs_remotes:
  - ssh://[email protected]/$repo/sfu-salt.git:
    - pubkey: /root/.ssh/salt_id_rsa.pub
    - privkey: /root/.ssh/salt_id_rsa
    - root: states

Steps to Reproduce Issue

Master config above, try to start salt master

root@sfup-salt01:~ # salt-master -l debug
[DEBUG   ] Reading configuration from /usr/local/etc/salt/master
[DEBUG   ] Using cached minion ID from /usr/local/etc/salt/minion_id: sfup-salt01.$domain.com
[DEBUG   ] Configuration file path: /usr/local/etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Master
[DEBUG   ] Loaded master key: /usr/local/etc/salt/pki/master/master.pem
[PROFILE ] Beginning pwd.getpwall() call in masterarpi access_keys function
[PROFILE ] End pwd.getpwall() call in masterarpi access_keys function
[INFO    ] Preparing the root key for local communication
[DEBUG   ] Removing stale keyfile: /var/cache/salt/master/.root_key
[DEBUG   ] Created pidfile: /var/run/salt-master.pid
[INFO    ] Starting up the Salt Master
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] LazyLoaded git.envs
[CRITICAL] Exception caught while initializing gitfs remote 'ssh://[email protected]/$repo/sfu-salt.git': empty (sub)expression
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/salt/utils/gitfs.py", line 351, in __init__
    self.new = self.init_remote()
  File "/usr/local/lib/python2.7/site-packages/salt/utils/gitfs.py", line 1464, in init_remote
    str(self.ssl_verify).lower())
  File "/usr/local/lib/python2.7/site-packages/pygit2/config.py", line 186, in set_multivar
    check_error(err)
  File "/usr/local/lib/python2.7/site-packages/pygit2/errors.py", line 64, in check_error
    raise GitError(message)
GitError: empty (sub)expression
[CRITICAL] Failed to load gitfs
[CRITICAL] Master failed pre flight checks, exiting

Versions Report

Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: 1.7.0
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.5
        libgit2: 0.25.1
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.25.1
         Python: 2.7.14 (default, Oct  3 2017, 01:18:38)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.2.2

System Versions:
           dist:
         locale: US-ASCII
        machine: amd64
        release: 11.1-RELEASE-p1
         system: FreeBSD
        version: Not Installed
Bug P2 severity-medium stale
Source
picture mcarlton00

Most helpful comment

Been digging into this, and I found some more info, and a workaround:

First: Weirdness. It's not happening on all servers. I have another FreeBSD machine, same patch level, identical packages and versions installed, same salt version, and it works correctly there.

Second: For the devs, pygit2 seems to be erroring out while initializing the git repo locally. If I switch from pygit2 to gitpython, everything works correctly. After the repo has been checked out and initialized, if I switch back to pygit2, it continues to work. However, if I remove the /var/cache/salt/master/gitfs directory and try to re-initialize using pygit2, the master starts failing again.

Third: For @sylgeist, short instructions for switching to gitpython:

  1. shut down master
  2. pkg remove py27-pygit2 (assuming you used the package manger and not pip. Adjust accordingly)
  3. pkg install py27-gitpython
  4. Edit your gitfs remotes to just have the repo, no keys (gitpython doesn't support this). Also edit the gitfs_provider if you manually specified it, which usually isn't necessary.
gitfs_remotes:
  - ssh://[email protected]/$repo/sfu-salt.git:
  1. Create an entry in root's (or your salt user's) ~/.ssh/config to define the correct key file
host bitbucket.org
 IdentityFile    ~/.ssh/salt_id_rsa
 User git
  1. Make sure the website key is in the user's ~/.ssh/known_hosts file. A simple way to do this could be ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts
  2. Start master
picture mcarlton00 on 23 Oct 2017
👍2

All 7 comments

I'm having a similar problem with a git repo:

fileserver_backend:
  - roots
  - git

   - ssh://git@gitbox/data/git/saltmaster.git:
     - pubkey: /root/.ssh/id_rsa.pub
     - privkey: /root/.ssh/id_rsa


Salt Version:                                 
           Salt: 2017.7.1                     

Dependency Versions:                          
           cffi: 1.7.0                        
       cherrypy: Not Installed                
       dateutil: 2.6.1                        
      docker-py: Not Installed                
          gitdb: Not Installed                
      gitpython: Not Installed                
          ioflo: Not Installed                
         Jinja2: 2.9.5                        
        libgit2: 0.25.1                       
        libnacl: Not Installed                
       M2Crypto: Not Installed                
           Mako: Not Installed                
   msgpack-pure: Not Installed                
 msgpack-python: 0.4.7                        
   mysql-python: Not Installed                
      pycparser: 2.10                         
       pycrypto: 2.6.1                        
   pycryptodome: Not Installed                
         pygit2: 0.25.1                       
         Python: 2.7.14 (default, Sep 26 2017, 18:22:29)                                    
   python-gnupg: Not Installed                
         PyYAML: 3.11                         
          PyZMQ: 16.0.2                       
           RAET: Not Installed                
          smmap: Not Installed                
        timelib: Not Installed                
        Tornado: 4.5.2                        
            ZMQ: 4.2.2                        

System Versions:                              
           dist:                              
         locale: US-ASCII                     
        machine: amd64                        
        release: 11.1-RELEASE-p1              
         system: FreeBSD                      
        version: Not Installed

2017-10-10 16:10:21,246 [salt.utils.gitfs                         ][CRITICAL][93649] Exception caught while initializing gitfs remote 'ssh://git@gitbox/data/git/saltmaster.git': empty (sub)expression
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/salt/utils/gitfs.py", line 351, in __init__
    self.new = self.init_remote()
  File "/usr/local/lib/python2.7/site-packages/salt/utils/gitfs.py", line 1464, in init_remote
    str(self.ssl_verify).lower())
  File "/usr/local/lib/python2.7/site-packages/pygit2/config.py", line 186, in set_multivar
    check_error(err)
  File "/usr/local/lib/python2.7/site-packages/pygit2/errors.py", line 64, in check_error
    raise GitError(message)
GitError: empty (sub)expression 
2017-10-10 16:10:21,249 [salt.master                              ][CRITICAL][93649] Failed to load gitfs
2017-10-10 16:10:21,249 [salt.master                              ][CRITICAL][93649] Master failed pre flight checks, exiting
sylgeist picture sylgeist on 10 Oct 2017

Been digging into this, and I found some more info, and a workaround:

First: Weirdness. It's not happening on all servers. I have another FreeBSD machine, same patch level, identical packages and versions installed, same salt version, and it works correctly there.

Second: For the devs, pygit2 seems to be erroring out while initializing the git repo locally. If I switch from pygit2 to gitpython, everything works correctly. After the repo has been checked out and initialized, if I switch back to pygit2, it continues to work. However, if I remove the /var/cache/salt/master/gitfs directory and try to re-initialize using pygit2, the master starts failing again.

Third: For @sylgeist, short instructions for switching to gitpython:

  1. shut down master
  2. pkg remove py27-pygit2 (assuming you used the package manger and not pip. Adjust accordingly)
  3. pkg install py27-gitpython
  4. Edit your gitfs remotes to just have the repo, no keys (gitpython doesn't support this). Also edit the gitfs_provider if you manually specified it, which usually isn't necessary.
gitfs_remotes:
  - ssh://[email protected]/$repo/sfu-salt.git:
  1. Create an entry in root's (or your salt user's) ~/.ssh/config to define the correct key file
host bitbucket.org
 IdentityFile    ~/.ssh/salt_id_rsa
 User git
  1. Make sure the website key is in the user's ~/.ssh/known_hosts file. A simple way to do this could be ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts
  2. Start master
mcarlton00 picture mcarlton00 on 23 Oct 2017
👍2

Also on FreeBSD 11.1 with salt 2017.7.1

I have a similar issue with the same errors. I have a winrepo setup in my master conf with:
winrepo_remotes_ng:

  • file:///data/shares/winrepo.git/

I accidentally deleted, via rsync, the 'win' directory in my states/base folder. I restarted salt_master a few times before noticing and now when I run a: salt-run winrepo.update_git_repos I get the same error as above.

Following mcarlton00's advice I removed pygit2 and installed gitpython and it worked fine. Switching back the repo continues to work.

rhavenn picture rhavenn on 1 Nov 2017

Same for me as well. Switching to gitpython has solved the issue in my environment.

sylgeist picture sylgeist on 2 Nov 2017

This issue also goes away if a regexp is added to the set_multivar command at gitfs.py:1466. A keyboard mash does the job. Hopefully this is enough information for someone who actually understands libgit2.

backerman picture backerman on 18 Mar 2018

Also seeing this on NetBSD 7.1.2. Switching to gitpython worked. Note that gitpython isn't in pkgsrc so you'll have to install it with pip.

travispaul picture travispaul on 14 May 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] picture stale[bot] on 27 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

sfozz picture sfozz  路  3Comments
mooperd picture mooperd  路  3Comments
udf2457 picture udf2457  路  3Comments
lhost picture lhost  路  3Comments
seanacais picture seanacais  路  3Comments