Git-subrepo: How does it work?

[qwertie]

The Readme file explains well how to use git subrepo, but I'm left wondering how it works on a technical level.

Not that I _completely_ understood how the "other" git-subrepo works, but a nice start would be to say how git subrepo it is similar to and different from that earlier attempt.

[radix-platform]

The tests in source tree can explain details. Also you can set up your own test environment such as https://habr.com/post/428493/ and see all details in your test repos and .gitrepo files.

[grimmySwe]

There is short overview here: https://github.com/ingydotnet/git-subrepo/wiki/Basics

The main trick of git-subrepo is the .gitrepo file, it will include references to connect the main repo with the subrepo. git-subrepo will use low level git commands to move between the main repo and the subrepo "scope".

In the new 0.4.0 we added support for worktrees so when you move between the different scopes, you don't have to overwrite the entire tree all the time.

Please let me know if you have more specific question.

[qwertie]

One good way to understand git-repo is to read the source code... but there are some hurdles to understand first so I prepared this description. However, after that I found an easier way (see below).

Source code intro

Apart from its reliance on pseudo-global variables (and the fact it's a bash script), I found the single main source file to be succinct and readable, albeit underdocumented.

In order to understand this file, you first have to understand how git-subrepo is invoked. When you write git subrepo foo bar, git invokes git-subrepo foo bar, so $1=foo and $2=bar. The next interesting thing is that git-subrepo does is bash+:import :std can. This command calls into git-subrepo.d/bash+.bash, which is short but entirely unreadable for non-bash-gurus. What this command does is "import" functions from bash+.bash into the main script. Specifically,

• The bash+:can() function is made available in the main script, renamed to can
• The :std parameter causes the list of functions named within bash+:export:std() { @ use die warn; } to be made available as commands in the outer script, i.e. bash+:use is renamed use, bash+:die is renamed die and bash+:warn is renamed warn.

After setting some global vars, git-subrepo defines numerous functions and calls main "$@" (which passes "$@", the list of arguments given to the script itself, into main()). main sets a lot of "local" variables but in bash, local variables are inherited by called functions, so all these variables act like globals.

The next important part is that main calls get-command-options whose main job is to parse command-line options into various global variables. It also saves the main command name into $command and saves other non-option parameters into $command_arguments. (The tricky part of this function is the call involving git rev-parse --parseopt, which has the effect of transforming the local parameters of get-command-options into another format, best explained by example: if git-subrepo clone /tmp/shared --force was called then set -- -f -- 'clone' '/tmp/shared' is executed in the script, which causes $1=-f, $2=--, $3='clone', $4='/tmp/shared'. In this example, $command becomes clone and $command_arguments becomes /tmp/shared.)

Finally, main calls the command function, e.g. command:clone for git subtree clone.

The key to understanding the command functions is to know what command-setup does. For example, command-setup +subrepo_remote subdir:guess-subdir calls get-params +subrepo_remote subdir:guess-subdir, which means

• the first non-option parameter to clone is required and should be stored in $subrepo_remote
• the second parameter is optional and, if missing, call guess-subdir to choose it

With all these preliminaries out of the way it becomes relatively straightforward to understand what each individual command does. Helpfully the o command is often used instead of comments, which writes verbose messages on the screen when you use the -v option.

An easier way

Once I did all that I discovered that the -d option causes git-subrepo to report most of the external commands it executes (see the RUN() function), so you can see the commands it is sending to git. If my assumption is valid that the most important work is ultimately performed by git, all you have to do is look up (google) what those commands actually do.

Example:

~~~
$ git subrepo clone [email protected]:/git/tool/xyzutil -v -d

• Assert that working copy is clean: /home/username/git/xyz
• Check for worktree with branch subrepo/xyzutil

git rev-parse HEAD

• Determine the upstream head branch.
  git ls-remote [email protected]:/git/tool/xyzutil
  [email protected]'s password:
  
  
  
  • Fetch the upstream: [email protected]:/git/tool/xyzutil (master).
    
    git fetch --no-tags --quiet [email protected]:/git/tool/xyzutil master
    
    [email protected]'s password:
  
  
  • Get the upstream subrepo HEAD commit.
    
    git rev-parse FETCH_HEAD^0
  
  
  • Create ref 'refs/subrepo/xyzutil/fetch'.
    
    git update-ref refs/subrepo/xyzutil/fetch 04ccf1661feefa095f8ffbb929ed2aabc567dbe0
  
  
• Make the directory 'xyzutil/' for the clone.
  mkdir -p -- xyzutil
• Commit the new 'xyzutil/' content.
  
  
  
  • Check that '04ccf1661feefa095f8ffbb929ed2aabc567dbe0' exists.
  
  
  • Make sure '04ccf1661feefa095f8ffbb929ed2aabc567dbe0' contains the upstream HEAD.
  
  
  • Put remote subrepo content into 'xyzutil/'.
    
    git read-tree --prefix=xyzutil -u 04ccf1661feefa095f8ffbb929ed2aabc567dbe0
  
  
  • Put info into 'xyzutil/.gitrepo' file.
    
    git cat-file -e 6d4c8f0963e9d3d7976748b4fcf89b89c4dbadd1:xyzutil/.gitrepo
    
    git config --file=xyzutil/.gitrepo subrepo.remote [email protected]:/git/tool/xyzutil
    
    git config --file=xyzutil/.gitrepo subrepo.branch master
    
    git config --file=xyzutil/.gitrepo subrepo.commit 04ccf1661feefa095f8ffbb929ed2aabc567dbe0
    
    git rev-parse 04ccf1661feefa095f8ffbb929ed2aabc567dbe0
  
  
  • 04ccf1661feefa095f8ffbb929ed2aabc567dbe0 == 04ccf1661feefa095f8ffbb929ed2aabc567dbe0
    
    git config --file=xyzutil/.gitrepo subrepo.parent 6d4c8f0963e9d3d7976748b4fcf89b89c4dbadd1
    
    git config --file=xyzutil/.gitrepo subrepo.method merge
    
    git config --file=xyzutil/.gitrepo subrepo.cmdver 0.4.0
    
    git add -f -- xyzutil/.gitrepo
    
    git add -f -- xyzutil/.gitrepo
  
  
  • Commit to the 'development' branch.
    
    git commit -m git subrepo clone [email protected]:/git/tool/xyzutil subrepo: subdir: "xyzutil" merged: "04ccf16" upstream: origin: "[email protected]:/git/tool/xyzutil" branch: "master" commit: "04ccf16" git-subrepo: version: "0.4.0" origin: "https://github.com/ingydotnet/git-subrepo" commit: "5d6aba9"
  
  
  • Remove worktree:
  
  
  • Create ref 'refs/subrepo/xyzutil/commit'.
    
    git update-ref refs/subrepo/xyzutil/commit 04ccf1661feefa095f8ffbb929ed2aabc567dbe0
    
    Subrepo '[email protected]:/git/tool/xyzutil' (master) cloned into 'xyzutil'.
    
    ~~~
  
  

Interpretation:

• git ls-remote [email protected]:/git/tool/xyzutil Get list of branches and tags in remote repo, including HEAD, with hashes.
• git fetch --no-tags --quiet [email protected]:/git/tool/xyzutil master: fetch the master branch of xyzutil. This fetches the entire history of master. It's worth understanding that files ("blobs") and folders ("trees") in git are both stored as "objects" in the repo, and that these objects don't know or care where they are located in any given commit, worktree, or index. So files and folders inside a subrepo are stored the same way as "normal" files or folders. In fact, this command doesn't "materialize" the downloaded files anywhere - they are not in any branch, not in the index, not in any commits. They exist only in .git/objects and are logically mixed together with files from outside the subrepo, since .git/objects is essentially one giant cryptographic hashtable (though they might _happen_ to be stored in a separate packfile). --no-tags avoids creating copies of tags from the remote repo.
• git rev-parse FETCH_HEAD^0: gets 04ccf166..., the hashcode of the remote master (it is also the hashcode of the remote HEAD but I don't think that's important), on stdout.
• git update-ref refs/subrepo/xyzutil/fetch 04ccf166...: uses update-ref to create a named reference to the commit with has 04ccf166 (remote HEAD). Normally "refs" are branches (refs/heads/*) and tags (refs/tags/*), but the name of this new reference is refs/subrepo/xyzutil/fetch so it is neither a branch nor a tag, but a "subrepo".
• mkdir -p -- xyzutil: Create xyzutil (-p allows creating multiple folders at once)
• git read-tree --prefix=xyzutil -u 04ccf166... The hash refers to the remote HEAD. This command reads the tree (directory contents) with hashcode 04ccf166... into the index (staging area) at a subfolder called xyzutil. the -u option updates the files in the work tree "with the result of the merge" (that's what the documentation says, but it also says merging is optional and AFAICT no merge will happen here. git docs have a habit of being confusing.)
• git cat-file -e 6d4c8f0963e9d3d7976748b4fcf89b89c4dbadd1:xyzutil/.gitrepo This one stumped me for awhile. First of all, turns out the script had called git rev-parse HEAD earlier to get this hashcode. Following the documentation of git cat-file over to this link leads me to believe that <rev>:<path> means "print out the contents of <path> inside <rev>"... but since xyzutil/.gitrepo doesn't exist yet, that can't possibly be what it does... right? Actually, the command fails, and git-subrepo responded to this failure by silently creating a new .gitrepo file, empty except for a comment. I guess the idea here is to copy .gitrepo from the remote subrepo if it exists.
• Each git config command adds a line to xyzutil/.gitrepo
• git add -f -- xyzutil/.gitrepo stages .gitrepo even if .gitignore would ignore it.
• git commit -m <message> commits (the message contains newlines, which are apparently suppressed by echo)
• git update-ref refs/subrepo/xyzutil/commit 04ccf166 makes another ref to the remote subrepo's commit to master.

Summary: git subrepo clone simply fetches a remote repo, stages it in a subfolder, creates two refs to it, and commits the changes. It's a normal one-parent commit, and it creates no branches or tags.

The commit objects from the subrepo's master are fetched; however, these commits are nearly invisible as they are unreachable from the branches, tags, and git reflog. However they are not shown by git fsck --unreachable because they _are reachable_ from the refs/subrepo/* refs, and in fact you can view them with git log refs/subrepo/<folder>/commit.

I haven't tested this, but presumably refs/subrepo/* refs won't transfer to/from remotes during pushes and pulls because git doesn't know what they are. so when you do a normal push, the new commit is indistinguishable on the remote repo from a normal commit. I infer therefore that

• the remote repo won't receive the entire history of the subrepo even though it exists locally
• others who pull from the remote repo won't get the refs/subrepo refs.
• nevertheless, others who don't have the refs can still do git subrepo push ... or git subrepo fetch ... because the .gitrepo file contains the necessary information.

[ingydotnet]

@qwertie great code dive and explanation.

-v and -d are your friends to figuring out what a command does.

Your assumption is correct that git-subrepo is just a set of orchestrations of git plumbing commands. This is in keep with the git itself, which has porcelain commands built from plumbing.

I am planning to update the bash code to my latest standards, once the current outstanding PRs have been applied. Hopefully it will become even cleaner.

[qwertie]

I appended more info above ... let me know if anything is mistaken.

Interestingly after all this, I read the documentation of git subrepo commit and have no clue what it means. I'm wondering, let's say I edit files in the subrepo inside the bigger repo, twice, creating two commits in master. How do I push these commits to the remote subrepo's master (which, after all, is unaware that it is a subrepo)?

[admorgan]

git subrepo commit reintegrates changes created on a subrepo branch into the the base repo. git subrepo push is what would push those changes back to the subrepos master.