Description of Issue/Question
I am getting lots of lock warnings for my gitfs remotes. These lock files seem to be preventing salt from automatically updating the available states.
[WARNING ] gitfs_global_lock is enabled and update lockfile /var/cache/salt/master/gitfs/SHA/.git/update.lk is present for gitfs remote '[email protected]:my/project.git'. Process PID obtained the lock but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master.
[WARNING ] Update lock file is present for gitfs remote '[email protected]:my/project.git', skipping. If this warning persists, it is possible that the update process was interrupted, but the lock could also have been manually set. Removing /var/cache/salt/master/gitfs/SHA/.git/update.lk or running 'salt-run cache.clear_git_lock gitfs type=update' will allow updates to continue for this remote.
Setup
fileserver_backend:
- roots
- gitfs
gitfs_remotes:
- [email protected]:my/project.git:
- mountpoint: salt://hello
Versions Report
Salt Version:
Salt: 2018.3.3
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: 2.0.5
gitpython: 2.1.11
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Jul 13 2018, 13:06:57)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.5.1804 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.14.4.el7.x86_64
system: Linux
version: CentOS Linux 7.5.1804 Core
firelizzard18
All 22 comments
Also of note is that I frequently get segfaults when I update:
> salt-run fileserver.update backend=gitfs
True
Segmentation fault
is there any additional information in the debug logs that you can include?
Also did you do something recently such as an upgrade of gitpython, salt, etc when this behavior started to show up?
Ch3LL
on 5 Feb 2019
I set up salt relatively recently. I think I've been seeing both of these issues since I added my first gitfs remote. I'll attach logs once I have them.
firelizzard18
on 5 Feb 2019
Hi,
I got the same logs on my infrastructure, same saltstack version : 2018.3.3
I've planed to upgrade to 2018.4 to see if this keeps happening.
Cheers,
Guillaume.
importepeu
on 4 Mar 2019
I've recently started seeing this issue as well. It seems to occur when the salt-master service restarts.
My best guess is that restarting the salt-master service while the gitfs repositories are being updated is what is causing the problem. When the service stops it doesn't seem to be cleaning up the locks on the repos, so when the service starts back up again it complains that something else has a lock on the repos.
jerrykan
on 5 Apr 2019
Seeing this as well.
wyardley
on 13 Apr 2019
@jerrykan can you verify this guess by looking at the log timestamp vs when the saltmaster service is restarted.
Ch3LL
on 25 Apr 2019
or better yet posting the sanitized version of your debug logs here
Ch3LL
on 25 Apr 2019
I have similar problem here, but it's not related to service restarts and probably the case is a bit different.
Salt Master has local git repo hosted on it, Syndics are connected to this repo using SSH. Sometimes when there is a network maintenance on the weekend and there is no connectivity between Master and Syndic, local Syndic installations get stuck with update lock on gitfs and only when the lock is deleted, it continues working properly. Btw, pygit2 is used.
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.10
libgit2: 0.26.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: 1.3.10
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.2
Python: 3.6.7 (default, Oct 22 2018, 11:32:17)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 4.15.0-42-generic
system: Linux
version: Ubuntu 18.04 bionic
akrus
on 4 Jun 2019
Is there an easy way to check this (other than looking at logs) in order to detect and fix a stuck repo?
OrangeDog
on 19 Aug 2019
I'm getting the same warnings on salt-run fileserver.update.
If I run the suggested salt-run cache.clear_git_lock gitfs type=update, running fileserver.update a second time causes the shell to hang.
If I skip the second fileserver.update and attempt to highstate, it fails saying there's no sls in env 'base' for the formula I'm trying to pull from the git repo.
Where can I look for debug logs? I'm brand new to Salt.
dalemartin
on 16 Sep 2019
@dalemartin : I've made up with this errors configuring this two directives in salt-master :
gitfs_global_lock: False
git_pillar_global_lock: False
It allow salt to remove the lock when it detect it without displaying errors. No side effects detected using thoose. This is what I used to do manually when dealing with this errors.
importepeu
on 17 Sep 2019
@importepeu as far as I can see that means there aren鈥檛 any locks.
The side effect will be you鈥檒l apply an inconsistent state if changes update between simultaneous/overlapping runs.
OrangeDog
on 17 Sep 2019
@OrangeDog I agree, but from my experience, i used to have state/pillar update problems with thoose regular locks (due to died salt processes) which I don't have now anymore. I don't apply states while editing my repos that's why I always use up to date states/pillars datas as far as I can observe. Using this two directives is probably not a best practice but at the moment I've no side effects.
importepeu
on 17 Sep 2019
@importepeu and @OrangeDog thanks for your timely responses.
As it turns out, I was trying to access a git repo with read-only credentials and had the wrong password. So maybe Salt gets stuck trying to authenticate and never releases the lockfile? Maybe there should be a more graceful way of handling this, like somehow indicating that is what's happening?
dalemartin
on 17 Sep 2019
Is there an easy way to check this (other than looking at logs) in order to detect and fix a stuck repo?
possibly check to see if the lock file is there for n length of time
@dalemartin can you share your master configs so i can try to replicate this?
Ch3LL
on 22 Oct 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
![stale[bot] picture](https://avatars.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 7 Jan 2020
As far as I am aware this issue hasn't been addressed.
jerrykan
on 8 Jan 2020
Thank you for updating this issue. It is no longer marked as stale.
stale[bot]
on 8 Jan 2020
For what its worth: We are facing the same error.
Now switching from gitfs_provider: pygit2 to gitfs_provider: gitpython
Please assume _all fine after that change_ in case we stay in lurking mode ...
stappersg
on 6 Feb 2020
To summarize this issue up, it seems that with pygit2 we need to have some sort of way to detect if a lock is stuck for a certain length of time and to clean up the lock, not sure if there is a better approach for solving this. ping @saltstack/team-core any ideas other than my approach?
Ch3LL
on 11 Feb 2020
Run into this issue aswell . @Ch3ll if you need some additional information , just ping me.
network-shark
on 28 Feb 2020
Related issues
sagetherage
路
3Comments
twangboy
路
3Comments
Inveracity
路
3Comments
zieba88
路
3Comments
udf2457
路
3Comments