Salt: Using git submodules in gitfs

[batulin-s]

Using git submodules in gitfs would be possible to separate access between independent states for different groups of minions. In this case We have one repo with the common states and some submodules for user-dependent states and We can allowing access to the submodule without access to all states.
As far as I could see this feature can be implemented in fileserver/gitfs.py
in git-python part. I can do it, but have a one question. Is submodules pulled in all cases or if defined special variable in config (like pull_with_submodules)?

[basepi]

Submodules....so many nightmares dealing with those.

It would probably be useful to get them in, but you should coordinate with @terminalmage on the implementation, as gitfs is mostly maintained by him these days. As far as submodules being pulled in all cases, I don't know. I wouldn't count on it...

[terminalmage]

This all depends on how the different git frontend modules (GitPython, pygit2, dulwich) expose submodules, something which I have not investigated. At any rate, we would want to implement it for _all_ frontends, not just GitPython, if possible.

[terminalmage]

Also, I just want to make sure I'm clear on what you are asking. Are you asking whether or not we should by default include submodules in gitfs? If so, then I think the we should make the default true, and I would suggest using a config parameter called gitfs_include_submodules.

Please note that per-remote equivalents must also be supported.

I'm much more familiar with this code since I re-wrote gitfs to support per-remote params and the pygit2 and dulwich frontends. I'm happy to look into this.

[batulin-s]

Thank You for responses, Sorry for delay, I'm trying to do it now.

[mnzaki]

+1 on this feature

[rminchev1]

+1

[eliasp]

:+1: for this…
I'd like to use submodules to have a repository containing stable/tested versions of public salt formulas.
Then I could easily keep them in sync with upstream after testing the latest commit in a dev branch.

[terminalmage]

I have an idea on how to implement this. I was investigating Pygit2's submodule support and unfortunately it wouldn't work for gitfs because it requires that a branch is checked out. We don't checkout a branch because we have to be able to support any branch/tag at any time as a fileserver environment, so we just fetch git objects and assemble files from the blob data. That way, if one minion wants a file from branch master and another wants one from branch dev, we don't run into a race condition.

A better solution for gitfs would be to read in .gitmodules and fetch each configured submodule into its own separate cachedir underneath /var/cache/salt/master/gitfs. Then, each GitProvider object would have an attribute listing its submodules, each of which would link to the separately-fetched repo.

Doing it this way _should_ make it possible to support submodules irrespective of the gitfs_provider being used, but I can't promise anything until I start playing around in a python shell and figuring out how to implement it.

I'm going to tentatively target this for the next feature release, but there are higher-priority issues I'm committed to doing for this release so it may not make it in until the one after next.

[Heiko-san]

+1 ... pleeeeeeaseeeeee , we really^n need this

[terminalmage]

@Heiko-san This is targeted for the feature release after the upcoming one.

[cliffrowley]

@terminalmage which release would that be, the 2016.x?

[terminalmage]

It would be targeted for this fall.

[terminalmage]

In other words, not the imminent release.

[mrichar1]

For info, an alternative approach to delegating access in this way can be achieved using the mountpoint option on a repo with different access:

gitfs_remotes:
  - https://git.example.com/top.git:
  - https://git.example.com/devolved.git
    - mountpoint: salt://devolved

You can then do things like:

include:
  - devolved.foo

Note however that this only works for gitfs, not for git_pillar (see #34210)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[akatrevorjay]

This issue was obviously not meant to be closed, can you reopen? Thanks!

[phineas42]

Agreed. This issue should be reopened.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[iavael]

Up

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[esilva-everbridge]

Is this still being worked on? It's something that has become important to us recently.

[esilva-everbridge]

If anyone can point to where in the codebase the functionality exists I'd love to take a look.

[terminalmage]

Yeah I'm sorry but I never got the opportunity to start this. All of the gitfs code is in salt/utils/gitfs.py. It's quite a large file, so you'd need to take some time to familiarize yourself with it. salt/fileserver.gitfs.py will create instances of the GitFS class, and each GitFS instance will have multiple Pygit2 or GitPython class instances, one per entry in gitfs_remotes.

You'll also want to know how to add a new configuration parameter to gitfs so that you can toggle whether or not salt tries to list submodules. This configuration loading happens in the GitProvider class (parent of both Pygit2 and GitPython) __init__ function.

[esilva-everbridge]

Thanks @terminalmage, I'll poke around and see if I can help out.

[nandhaece07]

could you please help me on this feature request? anyone working on it.

[esilva-everbridge]

I started to look at this but priorities at work have shifted, so I can't focus on it at the moment. @terminalmage gave a good overview of what and where to look to start working on it but I can't spare the time. :(