Git-subrepo: Performance with 1000+ subrepos

[udeved]

Hello,

this is not directly a bug report or feature request, rather than asking for advice.
We use subrepos for our package git tree, one subrepo package(PKGBUILD) for each.

This amounts to roughly 1100 subrepo in one main project.

Now the problem, a push, usually the first push after some subrepo has been cloned, depending on the commit count, tend to be painfully slow when subrepo rewrites history. If I enable -v switch, I see subrepo sequentially interating throught the commits.
In a repo with about 4000 commits and 1100 subrepos, it takes up to 4-5 mins to push a change into a subrepo.

My question is, if there is something that can be done about that, or if I might use subrepo command wrongly?

We use a simple git subrepo push $subrepo --clean. Is the clean flag responsible for the slow initial pushes? Documentation is somewhat sparse on that.

[grimmySwe]

Hi @udeved, yes, there are some operations that can take some time with subrepo. Are you using the latest master? or the "old" 0.3.0 release?

number of subrepos shouldn't be a problem in itself as you only "work" against one at the time, although if you have that many subrepos there will be a lot of commits just handling them in the main repo.

[udeved]

@grimmySwe

We are still stuck with https://github.com/chrisfair/git-subrepo fork, since it was already 0.4.1 long before 0.4.0 was relased. I may give it a try from upstream master.

Problem is, I need to mass update all Jenkinsfile in the subrepos, and many subrepos had no commit since clone.

[udeved]

another question on the --all option.

Is it viable to use a git subrepo push --all?
Is it faster that pushing each single one? Ie, does the rewrite with --all map all subrepos, or for each just as normal push does?

[udeved]

Regarding commit handling, our superrepos holding the subrepos are basically irrelevant. They are just a nice means to get all the package repos in one go. Our build operations etc are triggered by pushes in the subrepos, so a simple git prune in the superrepo does the job for us after each push.

A new "none" for method in .gitrepo would come handy, so the main repo gets only main commit history. Would be enough in our case.

[grimmySwe]

Ok, sounds like a cool use case. Nothing we have prepared for though.

Regarding your "all" question, there already is a an issue for this in #318.
Your "None" suggestions sounds interesting, it could certainly be helpful when you handle lots of subrepos. We'll see what 2019 brings :-)

[grimmySwe]

Note that if you never perform any useful operations within the main repo you could use "clone -f" to update, then you wouldn't need to perform any merge/rebase, it will simply rewrite the .gitrepo file to point to the new location.

[udeved]

@grimmySwe

Thank you for your answers, its appreciated. :)

I changed yesterday to subrepo upstream 0.4.0, there is unfortunately no improvement in regards to the rewriting history times, but that was expected.

Yes, our use case is not directly what subrepo was intended for, since we use it more or less in the subrepo direction, not main repo direction. I had been experimenting with submodules and subtree, and subrepo proved to be the best solution for the task.

The "none" method was just the idea to sort of have submodule style emulated if needed. Yep, in our case, it would be fully enough, if the main repo only held .gitrepo files. A squash method could work alternatively as well, like subtree does.

Little info on the setup, our mainrepo is configured to do a rebase on pull, works well so far.

[udeved]

In case you want to look at the described structure:

the main repos:
https://gitea.artixlinux.org/artixlinux/packages
https://gitea.artixlinux.org/artixlinux/community

And the subrepos ...
https://gitea.artixlinux.org/explore/organizations

Mainrepos are currently not updated yet, since I run a script to update the jenkinsfiles in each subrepo and only do a push in main repo when finished.

[udeved]

A little experience reporting after my update of the subrepos.

I kind of took benchmarks how log each push took.
In average around 3 minutes with the mapping min ~4000 and max ~7000 commits at rewrite.
My observation was, the time for the push depends much on how old the previous commit of the subrepo was, and how many other subrepo commits had been made in between.
A push could take around 8 minutes worst case, and best case 50 secs.

My conclusion is, the way we use it, might not be viable in the long run, with ever increasing commit count in the main repo and so many subrepo commits.

[udeved]

Some more thought after we settled with our new structure.
As it stands, there is really no good alternative to git-subrepo.

In order to solve the problem a bit, we can perhaps split the mainrepos in smaller task driven units or so.
However, the mentioned idea with a merge method "none" would cure the performance problems with many subrepos and commits significantly.

[keithn]

I'm not sure if this helps, but we have found dealing with many subrepos is generally painful. We have many shared libs across many projects. We are now consolidating our libs into, essentially, "library mono repos" which means some projects will have libs included in them that they will never use, but actually is way easier to work with.

[udeved]

@keithn

We have done kind of similar thing, wesplit the large main repo into smaller ones.

[devinrhode2]

Very interesting discussion here. I found that doing a git subrepo pull --all was just way too slow for my tastes. I am already quite experienced with git submodules and am going to try and figure out a little bit of scripting to help facilitate a sane pr workflow.

[admorgan]

There are a few patches that help quite a bit with this, especially https://github.com/ingydotnet/git-subrepo/pull/301

[admorgan]

For the PR workflow could you add it to https://github.com/ingydotnet/git-subrepo/issues/409?

[udeved]

@admorgan

This looks very helpful, what's the status with #301 ?
I would give it a try, is this a stable patch that can be applied to a production workflow?

[admorgan]

It is on the 0.4.1 branch, I believe it is very stable and am currently using in production at Garmin.

Austin

On Mon, Dec 23, 2019 at 08:53:39AM -0800, artoo wrote:

@admorgan

This looks very helpful, what's the status with #301 ?
I would give it a try, is this a stable patch that can be applied to a production workflow?

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/ingydotnet/git-subrepo/issues/389#issuecomment-568524922

[devinrhode2]

I was using the release/0.4.1 branch, so presumably was using that patch.

For anyone evaluating subrepo: I have 50 git "child repos" right now (currently setup as standard submodules right now). There are a million reasons for our project to not have 50+ git repos, and I am really dragging my feet on consolidating them, even though it's something I've "decided" that we should do multiple times already. There is great flexibility afforded by keeping them as is, but that's a different discussion. Subrepo worked great on windows 10, and one reason it could have been too slow for me was that I was really running windows 10 inside a parallels vm on a 2015 (maxed out) macbook pro. (I was evaluating on windows because my co-workers mostly use windows). While my parallels vm is slower than a normal machine, it's not that bad. Compared to what some of my co-workers are subjected to, a VDI running windows server 2008 R2, I'm sure my parallels vm is faster. Many places wouldn't have this situation, it would be nice if we could all run the latest and greatest available software and hardware.

TL;DR: I have a severely obtuse use-case, to anyone evaluating git-subrepo, it may perform quite well for you most of the time.

Regarding the PR workflow - I was referring to figuring out some scripts to aid in using _git submodules_ with a normal branching workflow. Therefore, I am not sure how I could constructively contribute to the discussion on #409. Feel free to email me if you are curious about the submodule scripts.

[udeved]

After one year and introducing grouping master repos that contain between 20 and 200 sub repos, it has acceptable push times. That said, this is on a system with fast nvme storage. There is already a noticeable difference on ssd for the same repo operations, it is slower at rewrite operation, ie when git tries to find/map the previous commit of the given sub repo.
Everything above 200+ can get pretty slow, the bottleneck remain the git native call to rewrite the master repo history.