Bfg-repo-cleaner: git notes after cleanup disappear

[gavvvr]

Hello. I used BFG to clean repo after migration from SVN. I had a git note for each commit where was an info about revision number. But after BFG did its job, i do not see notes with git log anymore.

[rtyley]

Hmm... the BFG rewrite all refs, and _notes_ are tracked under refs/notes/commits... but it's a bit more complicated than that. Because that ref is actually pointing to a note-bookkeeping commit:

$ git cat-file -p refs/notes/commits
tree 7b562120c12cffd6dc00948a032221bfb2a18079
parent fa041425fa1ed57267c92dfbb7baf5d1b6edbed2
author Roberto Tyley <[email protected]> 1479761077 +0000
committer Roberto Tyley <[email protected]> 1479761077 +0000

Notes added by 'git notes add'

...and the 'tree' of that commit (the file-tree of that commit) is actually a flat folder filled with one file per-note, with each file named after the commit that it's the note for:

$ git cat-file -p refs/notes/commits^{tree}
100644 blob b17d3b6ea799a69ff03be9bac7de2974284a2e2b    ad5db9067a3a1cb2a5f6529230eb5bfae41be87d
100644 blob e3dc505782b931439bc0b064338ff201336bd0c3    c4667ae189c73faff39ac6f6eb69610ea67b679e

$ git cat-file -p refs/notes/commits:c4667ae189c73faff39ac6f6eb69610ea67b679e
This is my note that I wrote for commit c4667ae!

...in order for notes to be preserved by a BFG rewrite, it would have to rename each of those files from the old commit id to the new commit id - at the moment it doesn't rewrite any filenames at all.

So that explains why you can't see any of those notes after a BFG rewrite. As a fix for yourself, you could use the object-id-map.old-new.txt file found under yourRepoName.bfg-report to manually rewrite those filenames.

[gavvvr]

I used git commands to transfer my notes, here is my simple script, it is slow tho.

#!/bin/bash

while read string; do
  hashesArray=($string)
  git notes copy ${hashesArray[0]} ${hashesArray[1]}
  git notes remove --ignore-missing ${hashesArray[0]}
done <object-id-map.old-new.txt

[gavvvr]

So will it be fixed or considered not an issue?

[rtyley]

It is an issue, obviously, for people who use Git notes - and I appreciate you taking the time to report it so I know about it. It should be a relatively small fix, though the idea of cleaning filenames for git commit ids is something the BFG hasn't done before, and _might_ lead to stackoverflow issues. I have limited free time and unfortunately can't always fix them immediately on being reported- but as I said, I do appreciate you reporting them.

[gavvvr]

Will be very useful, cause i think many people clean repo after migration from svn and commit->revision mapping is usually stored in notes.
Everyone forgets about notes, even git filter-branch ;(

[abliss]

@gavvvr 's script works, but this way is much faster and leaves you with a shorter notes-history:

cat object-id-map.old-new.txt | git notes --copy
cat object-id-map.old-new.txt | cut -d' ' -f 1 | git notes remove --stdin

You might want to follow that up with:

git notes prune
git update-ref refs/notes/commits $(git commit-tree refs/notes/commits^{tree} -m "notes squashed")

[gavvvr]

@abliss Hello. That really worked fast. But it increased repostory size almost 2 times!
I did not get the latter 2 commands. What are they supposed to do?

[ghuser]

+1 for fixing this issue.

[Vampire]

Be aware that if you use the build for PR #147 or if it got merged, then the recommendation of @abliss will not work properly as without my PR onto the PR multiple source commits are assigned to the same target commit as the pruned commits will be reported to be their nearest non-empty parent after the transition which is plainly wrong. The notes copying then copies a random one (actually the first alpha-numerically by source commit) to the target commit which is most often not the correct result. With my PR on that PR this will not happen or even better you can just filter out the according lines easily with grep.