Git-subrepo: Using a temporary directory for subrepo operations

[grimmySwe]

Using the current repo dir when performing subdir operations can have some implications:

1. If you have a large repository there might be some extra time spent checking out the entire repository.
2. Confusion with ignored files, as the content will change and so will the ignore files. And possible conflicts.
3. All files will be re-written, so editors like Emacs might warn you that things have changed under the hood.

Suggestion would be to investigate the possibility to somehow perform the file operations in a temporary repo. If there are conflicts during the merge steps, you end up in a temporary directory with the conflicts and you still have your main repo intact.

[ingydotnet]

@grimmySwe, I like this idea in theory. Do you have any spare cycles for it?

I would:

• Play with GIT_WORK_TREE and GIT_DIR env vars
• Probably put the work dir in ./.git/tmp/git-subrepo-$$/ or somesuch.
  
  
  
  • That's portable and always writable.
  
  

Getting the error workflows to feel good will be the hard part.

[stsquared99]

I'm starting some testing on this, since we need it for a repo that may have about ~90 subrepos within, very slow during pull/push operations.

[stsquared99]

I've submitted a PR at #173 that solves the issue for branch/init without having to use another work directory. Still testing for push/pull.

[stsquared99]

I've submitted an example at #175 of how I'm using the older git-new-workdir.

[grimmySwe]

@stsquared99 I might look into this soon, what is your latest status on this?

[stsquared99]

@grimmySwe I haven't looked at it since, I'm afraid.
I have had success in a standalone script I've been using by referencing a non-checked-out branch/ref using filter-branch, which speeds things up significantly. There's only a few commands that actually need the branch checked out, I think.

[grimmySwe]

This issue is a blocker for us (me and @jrosdahl) at work. We have a good size repo that contains a lot of built files occasionally. So it's a real killer if the entire tree is reset. So we actually have this on our todo-list...

I started looking at the GIT_DIR and GIT_WORK_TREE and it sure looks like that is one viable path. One thing though crossed our minds:
If there are no conflicts, it should be possible to perform the necessary operations entirely in the repo space (without touching the working copy). So I think I will start dig there, to see if it's actually possible to have most of the subrepo operations (init/clone/"pull no merge"/push) avoid resetting the working copy.

For the "pull merge" case I have an idea that might be good from a user perspective.
Lets say that you have a repo R with a subrepo S. If something fails the directory S will actually contain a working copy AND a repo copy of R at the "merge state" R(S). On the regular R level you will have an indication about this by the unknown files that are present in S. Instructions will tell you to "cd to S and solve the conflicts there" Then go back to R level and "git subrepo merge S" or something like that.

As an option it would be possible that the R(S) can be created in a temp directory somewhere, to prevent the reset of the directory S.

So in the default case S directory might reset if it needs to merge, but for users that want to prevent this loss at the cost of a little more strange instructions you can redirect this directory.

To be continued...

[grimmySwe]

Additional benefit of having it in-place is that you can actually test and/or compile with the main repo before you finalize the stuff and put them into your branch. So the current plan would be:

• Make sure that git-subrepo doesn't perform unnecessary operations that needs to remove the original location.
• If there are conflicts that needs to be solved, prepare the repo and then clone it into the subrepo dir and finally checkout the subrepo part into the subrepo directory to perform the merge.
• If necessary add a config option to make the merge outside in a temporary directory. (To prevent loss of things inside the subrepo dir and problems with .gitignore)

@ingydotnet Do you have any opinions against this?

[ingydotnet]

Seems like a good idea. Anyone want to try to implement it? Then we can review something real.

I would probably suggest putting the work dir under .git/. I think there's a standard place for doing things like that, but I forget.

[grimmySwe]

@ingydotnet Hi Ingy, I am about at trying out this now.
What is the status of merging issue/216 to master? by the activity I am guessing that more important things from real life has shown up.

[ingydotnet]

I'll try to look over subrepo stuff tonight.

[grimmySwe]

@ingydotnet Do you have an opinion on using git worktree, available from git 2.5.

I have been looking into this and I find that the GIT_DIR and the GIT_WORK_TREE might be a little dangerous if things go bad. Because you can end up having a status that differ between the GIT_DIR and the working copy. The main idea being not checking out a complete repository again.

git worktree seems to solve this in a supported way allowing you to have several working copies to one git repo. Use case would be as before but instead of checking out it locally it would work on a new worktree located in the .git/tmp/. That way everything would be consistent all the way and there are no magical switches between working copies.

[lilydjwg]

I like the idea. Today I had a hard time to merge conflicts of my .vim config because my lovely editor became unconfigured during the time.

git worktree is great, but what if I've already checked out the branch elsewhere? (Maybe I shouldn't do this?)

[grimmySwe]

@lilydjwg

git worktree is great, but what if I've already checked out the branch elsewhere? (Maybe I shouldn't do this?)

Do you mean as a worktree or in another way?

The work tree parts will only create internal work branches like subrepo/my-subrepo so it should be low risk that you already have that checked out. And perform some sort of check that it's not used.

[lilydjwg]

I checked out my subrepo as a worktree, so I can work there like a standalone repo, viewing logs and diffs, preparing pull requests etc (and I don't need to clone again. I'm in China, and GitHub is slow.)

I guess performing a check in advance is the best we can do. I guess conflicts would happen anyway if I didn't mess up things and kept the worktree.

[grimmySwe]

@ingydotnet I have done some initial tests here and found the git worktree approach very nice. It almost feels like an improvement from the original in that if you end up with a conflict in the merge/rebase steps you can see this in a separate worktree instead of ending up in a mysterious state in your regular working copy.

I have the following ideas:

• branch - Creates a new worktree with _subrepo/foo_ branch checked out
• commit - Takes the current state on _subrepo/foo_ and commits on HEAD. I consider setting a worktree requirement here so that if you want to do commit you need to have a worktree. And the worktree should be up to date with no loose ends.
• pull - Uses branch, then merge/rebase, if there is no conflict it will use commit and then remove the worktree, otherwise it will leave the worktree and give a good instruction on how to proceed.

[grimmySwe]

So I took a shot at this and created a prototype that actually works well so far. In my personal opinion it even improves the flow. Having a separate worktree with possible merge conflicts is much easier then ending up in your local copy with the subrepo checked out.

I will test it out some more but if there is anyone else that is interested you can find it in branch issue/150, note though that it requires git version 2.5 or higher and it's based in the issue/216 branch.

[gknops]

Rewriting files in the work directory is a deal breaker for me as I have processes (automatic header generation) monitoring file changes. Those go off while sub-repo is doing it's thing, causing all kinds of trouble. Pitty. Could this be revisited soon please?

[grimmySwe]

@ingydotnet I think that this would be best to include in 0.4.0 as it will set the git requirement and we probably don't want to do that in a minor release.

[grimmySwe]

Did some rebasing so if everything works out with the master I should be able to fit this as well.

[grimmySwe]

@ingydotnet This is now included in release/0.4.0 so please have a look and see what you think.

[grimmySwe]

Released in 0.4.0