Git-lfs: Not all files are migrated with git lfs migrate import

[tcamin]

We’re trying to migrate a largish (22k commits) repo to Git LFS, but are facing some issues with some specific files that seems not to migrate:

❯ git lfs migrate import --include=Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely --everything --verbose
migrate: Sorting commits: ..., done
migrate: commit 92d98a4fad452573a6773effb781b863f59a4622: Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely
migrate: Rewriting commits: 100% (21195/21195), done
  develop                                                   bf01409b82a35a3bf1c00b13bd49d39b0aaf0e8e -> ccdc1dbfca07daf403fb3c6bcd9281dc1b4b3ad1
....
.... other branches and tags
....
migrate: Updating refs: ..., done
migrate: checkout: ..., done

❯ GIT_LFS_SKIP_SMUDGE=1 git checkout 92d98a4fad452573a6773effb781b863f59a4622
Checking out files: 100% (1795/1795), done.
Note: checking out '92d98a4fad452573a6773effb781b863f59a4622'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 92d98a4fa adding Optimizely SDK

subito git/92d98a4...
❯ cat Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely | head -n 5
����
        X�K��L���ޑ���
                     3���!<arch>
#1/20           1474927163  503   20    100644  62236     `
__.SYMDEF�

As shown here the files seems to be matched during migration and the history gets indeed rewritten at commit 92d98a4fad452573a6773effb781b863f59a4622 however when I checkout that commit I don't see the LFS pointer, but instead I still see the entire binary data.

If I checkout without the GIT_LFS_SKIP_SMUDGE=1 I'm getting:

Encountered 1 file(s) that should have been pointers, but weren't:
    Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely

which means that the .gitattributes was properly updated but the file was not migrated.

Any advise on how to debug this?

[ttaylorr]

As shown here the files seems to be matched during migration and the
history gets indeed rewritten at commit [...] however when I checkout
that commit I don't see the LFS pointer, but instead I still see the
entire binary data.

You are correct in saying that the issue is likely to do with the fact
that a .gitattributes somewhere in your repository is marking that file
as tracked by Git LFS, although the git lfs migrate export invocation
made an attempt to get rid of it.

During migration, 'git lfs migrate' attempts to write the necessary
changes to your .gitattributes file in order to "undo" any previously
tracked Git LFS entries.

For example, if your .gitattributes contains:

*.pdf filter=lfs diff=lfs merge=lfs -text

And you run 'git lfs migrate export --include="my.pdf"', then your
.gitattributes will be updated to:

*.pdf filter=lfs diff=lfs merge=lfs -text
my.pdf -filter -diff -merge=lfs text

But, 'git lfs migrate ' both only consider the
.gitattributes at the root of the repository. In practice, if you have
_multiple_ .gitattributes files in your repository, others of them can
affect a file and cause it to appear to Git as being tracked by Git LFS.

We should consider these in the future, but we don't currently. In the
meantime, you can find out which .gitattributes file is changing Git's
perception of your file with:

$ git check-attr -a <filename>

And then use 'git filter-branch' to rewrite the file yourself.

[tcamin]

The repo did not contain any .gitattribute before migration, so I'm unsure this is the problem.

To recap:

1. I run git lfs migrate including a single file
2. From the logs I see that the file get migrated and the history gets rewritten
3. I checkout the branch containing the migrated file with GIT_LFS_SKIP_SMUDGE=1
4. I'm expecting that the single file contains the LFS pointer, but instead I see the entire binary data

Additionally if I checkout the branch containing the migrated file without GIT_LFS_SKIP_SMUDGE=1 I get the warning

Encountered 1 file(s) that should have been pointers, but weren't:
    Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely

[tcamin]

I'm trying to debug this further:

.git/lfs/objects contains a single object matching the size of the migrated file mentioned above, and that's what I would expect.

however git lfs ls-files returns an empty result

[ttaylorr]

It's possible that the checkout didn't occur properly, either. Could you
try doing a hard-reset to the new state of your repository?

If not that, can you create a reproducible example of your work and
share it with us?

[tcamin]

I'm still investigating the issue. Are multiple root commits an issue for the migrator?

git rev-list --max-parents=0 HEAD is returning 3 commits on our repo

[ttaylorr]

I'm still investigating the issue. Are multiple root commits an issue
for the migrator?

Git can merge two unrelated histories, and if both sides of that merge
have a root commit (which they both will if the other side of the merge
takes in the initial commit), then you will have >1 "root" commit.

Do any of those listed look like your project's initial commit?

[tcamin]

Among the 3 there's one commit that indeed looks like the initial commit, I was wondering if multiple root commits could cause some problems when migrating. From your answer I assume this is not an issue for the LFS migrator.

Currently I'm trying to understand why the migration of a single ref is duplicating the number of commits in the repo.

My initial repo returns:

❯ git rev-list --all --count
21213

After running

git lfs migrate import --include=Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely --include-ref=feature/partecipate_sdk --verbose

The number of commits

❯ git rev-list --all --count
40546

I end up with 2 develop branches (from were feature/partecipate_sdk was created) with duplicated commits (the migrated and not migrated commits). I don't understand if this is somehow to be expected.

[ttaylorr]

It's very possibly related to having multiple "root" commits in your
repository.

But, without a reproducible example, I'm not sure how I can best help.
Is your repository open-source, or can you create and share a
reproducible example of your problem?

[tcamin]

The repo isn't open-source and unfortunately I cannot share the code.

I will try to reproduce the issue and post an update if I manage to do so.

[bk2204]

What may be helpful to try to demonstrate an example is git fast-export --anonymize. That will produce a dump of your repository that contains no private information that you can then run through git fast-import to produce a new repository. You'll need to recreate the Git LFS configuration, though.

[nhatkhai]

I think I had reach the same issue when using git lfs migrate import (not .gitattribute file before migrate). I found some file in some commit aren't converted into LFS. I currently have to do a workaround by run git lfs migrate, and git tree-filter to correcting couple files that not migrate correctly. It is not a ideal, and very slow. (It was Windows LFS 2.4.2).

[christofblenkle]

Among the 3 there's one commit that indeed looks like the initial commit, I was wondering if multiple root commits could cause some problems when migrating. From your answer I assume this is not an issue for the LFS migrator.

Currently I'm trying to understand why the migration of a single ref is duplicating the number of commits in the repo.

My initial repo returns:

❯ git rev-list --all --count
21213

After running

git lfs migrate import --include=Subito/Pods/Optimizely-iOS-SDK/Optimizely.framework/Versions/A/Optimizely --include-ref=feature/partecipate_sdk --verbose

The number of commits

❯ git rev-list --all --count
40546

I end up with 2 develop branches (from were feature/partecipate_sdk was created) with duplicated commits (the migrated and not migrated commits). I don't understand if this is somehow to be expected.

Still interessed in that commit problem?
best regards
Christof

[bk2204]

I'm reopening this because it seems to still be occurring for some people. However, we're still looking for some sort of testcase to reproduce it.

[nhatkhai]

I would love to give you the example - but is it not a shareable respos. However, the repos have couple .docx, .pdf files, and code files. The problem may occurred if you have many commits of the same LFS files that had not been converted yet. And then you do the migrate on that range of commits to hope they got convert into LFS. I often see issues in this type of cases.

[nhatkhai]

I suguesst try to make this happen on version 2.4.2 first. Because that was the version I used, and do some work around by:
1) tree-filter to add .gitattributes files in
2) run the migrate with equivalent include list of files for LFS to the .gitattributes
This seem to make it work 100% of time (sample pool is about couple 1000+ commits).

[bk2204]

Right now, in order for us to make any progress on this, we'll need an minimal, complete, verifiable example. We're not seeing this issue on our test repositories, so without a testcase, we won't be able to reproduce this, which means we won't be able to fix it.

I understand this is frustrating, and we'd really like to fix this, but without more information, we don't know what specifically is broken.

[christofblenkle]

Hi all,
under some specific conditions, which I don’t know, lfs migrate import is not able to move refs (tags, branches) from the old commit to the new created one. As a consequence of this, “git gc” can’t remove the old commits.
Our database are unfortunately not sharable, but I can try to describe our way to get rid of these old commits.

1) Create a map file (--object-map=) while running lfs migrate import to get a correlation between old commits and new commits created by lfs.
2) Collect all commits from your original database and the working lfs database (git rev-list –all)
3) Identify all so-called double commits. These are commits existing in the original and the working lfs database.
4) Check these double commits for still existing refs (git show –s ) and move every found ref manually or by script to the corresponding new commit (from your created map file) with these git commands: git tag –f, git update-ref.
5) Run git gc –prune=now and check your commit count in your working lfs database
…

Best regards
Christof

[oschwald]

UPDATE: Please ignore. I was using an older version. Updating to the newest version fixes the issue I described.

I am also running into this when trying to migrate a large repo with --everything. In my case, it appears the lightweight tags are being migrated but annotated tags are not. This is also reproducible on a new repo by creating an annotated tag and then doing git lfs migrate import --everything ....

Minimal test case:

#!/bin/bash

mkdir /tmp/lfs-test
cd /tmp/lfs-test
git init
touch file
git add file
git commit -m "test"
git tag -a -m "test" test-tag

echo
echo "Head before"
git show HEAD

echo
echo "Tag before"
git show test-tag

git lfs migrate import --everything --include file

echo
echo "Head after"
git show HEAD

echo "Tag after"
git show test-tag

I saw https://github.com/git-lfs/git-lfs/issues/3568 but that seems to be different.

[ttaylorr]

I am also running into this when trying to migrate a large repo with
--everything.

I'm sorry to hear that you're having trouble. I have a suggestion that
I'll share in my response below that I think should clear things up.

Certainly if that doesn't help, it would be very helpful if you could
share the contents of your repository (or a reproducible test case) so
we could figure out where the defect is.

In my case, it appears the lightweight tags are being migrated but
annotated tags are not. This is also reproducible on a new repo by
creating an annotated tag and then doing git lfs migrate import --everything ....

I know that in much-older version of Git LFS (I'm thinking that it was
earlier than v2.4.0) there used to be a bug that we didn't migrate
annotated tags correctly. Some quick searching indicates that I fixed
this in https://github.com/git-lfs/git-lfs/pull/2780.

I'm wondering if you're using a version of Git LFS from before then? If
so, can you please upgrade to a version at or after v2.4.0? (Of course,
the most recently released version is the best one, but anything after
v2.4.0 will do ;-).)

I think that the _actual_ fix is in 33841d44 (git/githistory: teach
*refUpdater to rewrite local tags, 2017-12-11). Looking at the diff, I
think that the issue is that we _did_ migrate all of the objects
required to rebuild the annotated tag, just never the tag itself.

[nhatkhai]

My work around still work 100%. (--tree-filter or --index-filter to add correct .gitattributes to every commits, before run lfs migrate commands). Work every days since I'm first suggested.

[gjabouley-invn]

Hello, i was facing similar issue #3853 today while trying to migrate a Bitbucket repository too fat for Github (files bigger than 100MB).

I eventually succeed, but i'm not very happy with the solution...
And sorry, it is yet another closed-source repository, so i cannot share it. But i was able to reproduce the following several time.

Executed git commands for the migration

git clone --mirror ${BITBUCKET_REPO_URL} lfs_simple && pushd lfs_simple
(1) git lfs migrate import --everything --verbose --include="*.wav,*.psd" --object-map=../mapping_file.map.txt
(2) git reflog expire --expire-unreachable=now --all && git gc --prune=now
git push --mirror ${GITHUB_REPO_URL}

Before step 1 (lfs migrate import) --> Big repo with 204 commits

$ du -sh *
1.0K config
1.0K description
1.0K HEAD
40K hooks
1.0K info
1.3G objects
8.0K packed-refs
0 refs

$ git rev-list --all --count
204

$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 7398
packs: 1
size-pack: 1.20 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

After step 1 (lfs migrate import) --> Looks good, 90 new commits created, LFS files created, object files should be deleted by incoming git gc

$ du -sh *
1.0K config
1.0K description
1.0K HEAD
44K hooks
1.0K info
324M lfs
1.3G objects
8.0K packed-refs
130K refs

$ git rev-list --all --count
294

$ git count-objects -vH
count: 732
size: 352.82 KiB
in-pack: 7398
packs: 1
size-pack: 1.20 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

After step 2 (git gc) --> Objects size did not decrease, we have a problem!!!

$ du -sh *
1.0K config
1.0K description
1.0K HEAD
44K hooks
9.0K info
324M lfs
1.3G objects
8.0K packed-refs
28K refs

$ git rev-list --all --count
294

$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 7823
packs: 1
size-pack: 1.20 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

Issue description:
A tag was still referencing an old commit!
Turns out even if git lfs migrate says cc-0.13.0 4874eff37b5ee256c3e005c0e2d41e6128312494 -> 67527caaa783e897981a3ea0d79918ff91de5fbf, command git show -s cc-0.13.0 still says 4874eff37b5ee256c3e005c0e2d41e6128312494
So git gc could not perform properly...

Found solution so far:
Search if there are some tags that are still pointing to old OIDs that are suppposed to get migrated by git lfs migrate. You get such information into the --object-map=mapping_file.map.txt file.
Following script shall be executed within git repository. It will not make any modification in the repository, untill you un-echo the git tag command...

MAP_FILE=../mapping_file.map.txt
git for-each-ref | grep tags | while read -r oid type tag; do
        while IFS=, read -r old_oid new_oid; do
                if [[ "$oid" == "$old_oid" ]]; then
                        echo TAG $tag still pointing to old_oid $old_oid instead of $new_oid
                        echo git tag -f $(basename $tag) $new_oid
                fi
        done < $MAP_FILE
done

After fixing this :feelsgood: tag, git gc is able to perform proper cleanup, ending with a correct repository status:

$ du -sh *
1.0K config
1.0K description
1.0K HEAD
44K hooks
9.0K info
324M lfs
1022M objects
8.0K packed-refs
28K refs

$ git rev-list --all --count
204

$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 7399
packs: 1
size-pack: 1020.99 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

[gjabouley-invn]

My work around still work 100%. (--tree-filter or --index-filter to add correct .gitattributes to every commits, before run lfs migrate commands). Work every days since I'm first suggested.

Thanks, i did try your recommendation but ended up in same situation with a :feelsgood: tag pointing to an old commit.
Only @christofblenkle solution worked for me, with a bit of automation (see end of above post for a script)

By the way, here is the command to add correct .gitattributes files (that need to be well crafted) to every commits of a repository:

git filter-branch --tag-name-filter cat --index-filter "cp /<path_to_file>/.gitattributes . && git add .gitattributes" -- --all

[nhatkhai]

@gjabouley-invn, I think this migrate issue already resolved. But I never remove my work around in my script to verify.
Your problem is a common git problems which is changing history like (rebase, tree filters). Git so not move tags, or other branches during or after change its history. So in do in so, it need addition work. But it is not going to be 100% correct by just scripting.

[windwalker78]

Hi,

We are willing to help, by providing you with a test case. Can you PM me, so I can return a 120MB download link?

Regards,
Todor

[bk2204]

I don't believe GitHub provides us with a PM capability, but you can email such a link to my username at github.com. Thanks for offering to provide a testcase.

[bk2204]

Just as a note, #4133 fixes a particular instance of this (#4120) for which we have a test case. That may be the only outstanding issue here, but it's possible that there are additional things we've missed, so I'm going to leave this open for now. If you're seeing problems and #4120 sounds like it describes your case, please try using that PR (there are builds in the CI artifacts) or, if it's been merged by that point, try the latest master (of which there are also builds in the CI).

[ken-swyfft]

@bk2204 - FWIW, I just tried your fix on my repo where I'm having the same "git lfs migrate doesn't work problem" (see https://stackoverflow.com/questions/61823844/git-lfs-migrate-doesnt-seem-to-be-migrating-files).

Unfortunately, it didn't seem to fix our issue, at least.

In other words, after building git-lfs from your branch and then running this:

git lfs migrate import --everything --include="*.csv"

We're still seeing CSV files in the repo:

PS E:\source\swyfft_web_ken> git lfs migrate info --everything --include="*.csv"
migrate: Sorting commits: ..., done.
migrate: Examining commits: 100% (131623/131623), done.
*.csv   1.3 GB  2102/2102 files(s)      100%

And before you ask, unfortunately, no, this isn't a public repo, so I can't share it :-(.

I'll try one of the suggested workarounds.

[ken-swyfft]

FWIW, I tried using BFG, and it seems to have the same problem.

PS E:\source> java -jar c:\tools\bfg-1.13.0.jar --convert-to-git-lfs "*.{csv}" --no-blob-protection .\swyfft_web_github\


Using repo : E:\source\.\swyfft_web_github

Found 0 objects to protect
Found 428 tag-pointing refs : refs/tags/1.0, refs/tags/1.1, refs/tags/1.2, ...
Found 916 commit-pointing refs : HEAD, refs/heads/7821, refs/heads/9514, ...

Protected commits
-----------------

You're not protecting any commits, which means the BFG will modify the contents of even *current* commits.

This isn't recommended - ideally, if your current commits are dirty, you should fix up your working copy and commit that
, check that your build still works, and only then run the BFG to clean up your history.

Cleaning
--------

Found 131594 commits
Cleaning commits:       100% (131594/131594)
Cleaning commits completed in 424,721 ms.

Updating 1343 Refs
------------------

        Ref
 Before     After
        ----------------------------------------------------------------------------------------------------------------
--------------------
        refs/heads/7821                                                                                                |
 46974228 | d484c242
        refs/heads/9514                                                                                                |
 3ca1f30d | dd82273d
        refs/heads/BetaBranches/ap-Beta-3677_AnalysisRepriceTXQuotes                                                   |
 286ecd88 | 1d2cd8dd
        refs/heads/CH-4895-CommercialReprice                                                                           |
 e2347245 | e4f2bd3a
        refs/heads/CH-5037-RenewalManagementExpirationDate                                                             |
 c2dff9f4 | 11df4263
        refs/heads/CH-5538-FixForMissingAddressOnEmailQuote                                                            |
 fa02bce4 | a8418da0
        refs/heads/CH-5874-FixForCommercialTypesTestsFailures                                                          |
 e7695759 | 4bbcf938
        refs/heads/CH-5875-FixForMissingQuoteHistory                                                                   |
 3f59904d | 56a21b04
        refs/heads/CH-6296-AnonymousCommercialBindQuoteFix                                                             |
 ed90fba3 | 1d7fffbb
        refs/heads/CH-6604-RemoveOldJasmineTestRunnerPage                                                              |
 b28c9194 | d6df685d
        refs/heads/CH-BetterLoginErrorMessages                                                                         |
 15b7cc51 | 45ff40a2
        refs/heads/CH-CherryPickRenewalStyle                                                                           |
 df114c25 | 669794d6
        refs/heads/CH-CommercialBindQuoteTweaks                                                                        |
 16e14cbe | 7064ed3c
        refs/heads/CH-FixForFailedGenerateAuditDocsTask                                                                |
 1375649b | 4a6f498a
        refs/heads/CH-FixForGetAddressInfoShouldReturnCorrectState                                                     |
 1c2f0a9f | 23edf5d4
        ...

Updating references:    100% (1343/1343)
...Ref update completed in 10,017 ms.

Commit Tree-Dirt History
------------------------

        Earliest                                              Latest
        |                                                          |
        DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDmmmmmmmmmmmmmmmmmmmmmm

        D = dirty commits (file tree fixed)
        m = modified commits (commit message or parents changed)
        . = clean commits (no changes to file tree)

                                Before     After
        -------------------------------------------
        First modified commit | 0a53c184 | e49ce882
        Last dirty commit     | cfa1c9dd | dd582ba3

Changed files
-------------

        Filename                                      Before & After
        -----------------------------------------------------------------------------------------------------------
        ALAddresses_Quotes.csv                      | 9ee86177 ? 14ef4aef, 36fa8258 ? 7f13ee74
        ALAddresses_QuotesKen.csv                   | 583eb760 ? ac03faf0, af0470bb ? 6e4b4309
        AOPTerritorySinkhole.csv                    | 4882c71c ? 7f48caed
        AOPTerritorySinkholeFactors.csv             | a7931edc ? bac871cf
        AddressTerritories.csv                      | 16bc032f ? 8a99305a
        AutoGeneralFactors.csv                      | 2c721095 ? bb55197d
        AutoLossCost.csv                            | 11d4f480 ? daeadea3
        AutoMakeModel.csv                           | 237a441b ? d4ffeeed
        AutoMedical.csv                             | fa100a1e ? 7d481890
        AutoTerritory.csv                           | cb29d164 ? f588d5a0
        AutoTierRating.csv                          | 1ac1277a ? 02c87989
        AutoVehicleNumberDriverAge.csv              | b2cfdb91 ? ef571cfd
        BadRenewalFaxes.csv                         | 9f099996 ? 63ff2543, 71ee9459 ? 6163441b
        BadRenewalFaxesReal.csv                     | f13c57e2 ? f917e86c, 71ee9459 ? 6163441b
        BadRenewalFaxesTest.csv                     | 3a8416c2 ? a0af6cc9
        ...


In total, 170079 object ids were changed. Full details are logged here:

        E:\source\.\swyfft_web_github.bfg-report\2020-05-15\23-22-57

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

PS E:\source\swyfft_web_github> git lfs migrate info --everything --include="*.csv"
migrate: Sorting commits: ..., done.
migrate: Examining commits: 100% (165957/165957), done.
*.csv   1.3 GB  2016/2016 files(s)      100%

Am I correct in concluding from the output from the git lfs migrate info --everything --include="*.csv" that it didn't succeed?

[ken-swyfft]

FWIW, I tried the solution from @nhatkhai (using git filter-branch to add the .gitattributes file to every commit), and that didn't seem to work on our repo.

I suppose the next step is to try the solution proposed by @christofblenkle. I'm not much of a git wizard, and its internals are a bit of a mystery to me, so some of what he's describing doesn't make much sense, but I'll try to see if I can make enough sense of them to get it working.

[ymartin59]

I faces same issue with a large repository generated from a Subversion migration with svn-fast-all-export. Repository looks clean, only one root commit, only lightweight tags, no .gitattributes files, no .gitignore either.
I am providing a long list of patterns to migrate (file paths list have been retrieved from trees with various scripts):
git lfs migrate info --everything --include="*.pdf,*.PDF,*.docx,*.zip,*.ppt,*.pptx,*.xlsx,*.vsd,*.gdbtablx,*.gdbtable,*.lcmbiar,*.biar,*.mxd,*.spx,*.atx,*.dll,*.jar,*.exe,*.mp4" migrate: Sorting commits: ..., done. migrate: Examining commits: 100% (6753/6753), done. *.mxd 838 MB 558/558 files(s) 100% *.zip 741 MB 22/22 files(s) 100% *.gdbtable 671 MB 156/156 files(s) 100% *.dll 374 MB 940/940 files(s) 100% *.atx 216 MB 225/225 files(s) 100%
BFG fails to migrate too: it converts only 4 zip files where git lfs migrate converts 22 files.
git lfs migrate has missed many deleted files that matches patterns - and to confirm, I have processed "*.exe" files removal with BFG which then discarded 11 exe files (so missed by git lfs migrate)

svn-fast-all-export generates additional references in refs/backups/rX/PATH which are "dead-end" in Git history... and Bitbucket creates also its own references for Pull Request (refs/pull-requests/X/(from|merge) and stash-refs/pull-requests/X/(v0|from|to|merge|merge-clean) which may still hold blobs.
Another "stupid" question: is it possible some characters like spaces, underscores or unicode in file path/name may prevent blob to LFS pointer replacement?
I go on investigations.

[ymartin59]

Here is progress about my investigations.

In my specific case, the refs/backups/rX/tags/TAG references generated by svn-fast-all-export are created because of this common pattern: user has created a tag in revision R1 with incorrect "from" copy location (either path or revision) and deleted that same tag path in revision R2, to create it again with same tag path (here /tags/TAG/) in revision R3.

So in such a case, the very same file blob is added in a "master" commit at regular path location, but added and deleted too with a different file path in such a tag backup reference.

After adding migration rules for svn-fast-all-export to discard tag path /tags/TAG from revisions R1 to R2, resulting Git repository no longer contains any reference in refs/backups/.

And then git lfs migrate processed the repository without any hiccup.

If I am right, I should be able to create a test repository to reproduce issue. And probably the Bitbucket specific references for Pull Request may be a similar reason why git lfs migrate is in trouble to process all files.

[ymartin59]

If I am right, I should be able to create a test repository to reproduce issue. And probably the Bitbucket specific references for Pull Request may be a similar reason why git lfs migrate is in trouble to process all files.

As these references often contain commits history where same file blob are added multiple times - after amended or rebased commits are pushed over the same PR branch.

[nhatkhai]

@ymartin59, I had script sync SVN and git together, it also take care migration part. Due to the issue you experienced, I inject the .gitatributes with all the LFS setting for files I like to track for all commits. Then run git lfs migrate import --fixup. This work for me.

I would love to share this sync script publicly, but I cannot get in the company board just because I don't use MFA with my account.

Here is the snipped copy from my bash script (with modify to simplify the idea):

    # This WORKAROUND help git lfs migrate work more effective
    # MUST have all lfs filter hooks uninstalled global, system 
    # etc... including the C:\Program Files\Git\etc\gitconfig. 

    git lfs uninstall --local 2>/dev/null
    git branch -f lfsBranch master

    git filter-branch --prune-empty -f --index-filter '
      cp MY_IDEAL_GIT_ATTRIBUTES .gitattributes
      git add .gitattributes
      ' -- lfsBranch || exit 1

    git lfs install --local
    git lfs migrate import --fixup lfsBranch || exit 1
    echo DONE

[ymartin59]

@nhatkhai Thanks for your script. My repository is so large that git filter-branch is not an option for me. As far as I check, git lfs migrate does the job now I have discarded these "unexpected" references.

[amaidment]

I have been having similar issues to this (as per #4165), and had found that the solution suggested above worked.

However, subsequent to this, I found that if I removed the tags from our repository (which in my case was fine, as there were only 4, and they were all old & unnecessary), that I didn't experience this issue.