Git-lfs: Server-side Git LFS Missing Object, No Backups - Not able to duplicate repo

[WesWedding]

What do you do when .git/lfs/objects is missing an object and there is no local backup of the object anywhere?

How can I work around this problem? It renders me unable to do things like try to push my repo's complete history to a new repo because it will fail during git push -u new-origin --all

The exact steps that made this happen are hard to explain because I didn't do it myself, but basically someone managed to push to a repo without also pushing a LFS file and now there is a series of commits no one can check out because the related Git LFS object is missing.

I'd like to at least put a file in there that will allow me to pretend everything is okay but I tried to put the file contents of an older file version into the missing object's location, and git lfs fsck recognizes it as corrupt (probably due to SHA checking) and moves it to the "bad" folder.

Here are some commands I ran to find the offending commit, hopefully it helps with coming up with a workaround.

git lfs fetch --all
Scanning for all objects ever referenced...
✔ 574 objects found                                                                                                                                                     
Fetching objects...
Git LFS: (437 of 437 files, 1 skipped) 332.27 MB / 332.27 MB, 90.65 KB skipped                                                                                         
[4b5dcb2484fcce86c0b101e94ef4641786ff7106f04d9c648c8021e3dc0c3c80] Object does not exist on the server: [404] Object does not exist on the server
Warning: errors occurred

```
git lfs fsck
Object Assets/scenes/_SCENE_.unity (4b5dcb2484fcce86c0b101e94ef4641786ff7106f04d9c648c8021e3dc0c3c80) could not be checked: no such file or directory
Moving corrupt objects to /PATH_TO_PROJECT/.git/lfs/bad
rename /PATH_TO_PROJECT/.git/lfs/objects/4b/5d/4b5dcb2484fcce86c0b101e94ef4641786ff7106f04d9c648c8021e3dc0c3c80 /PATH_TO_PROJECT/.git/lfs/bad/4b5dcb2484fcce86c0b101e94ef4641786ff7106f04d9c648c8021e3dc0c3c80: no such file or directory

```git log -p --all -S 4b5dcb2484fcce86c0b101e94ef4641786ff7106f04d9c648c8021e3dc0c3c80
commit 2b2eacf213602cb4814109119ad76a7e6d32ee78
Author: NAME <EMAIL>
Date:   Fri Jan 13 16:36:57 2017 -0800

    Modified logic in ComputerAI so that the Game Manager will handle the cards for the ComputerAI.

diff --git a/Assets/scenes/_SCENE_.unity b/Assets/scenes/_SCENE_.unity
index 638c355..2dc93c3 100644
--- a/Assets/scenes/_SCENE_.unity
+++ b/Assets/scenes/_SCENE_.unity
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:674b11ad8e818fddd849eb3c050d9d4424fc131d2448b3c819438d569aa1d018
-size 92488
+oid sha256:4b5dcb2484fcce86c0b101e94ef4641786ff7106f04d9c648c8021e3dc0c3c80
+size 92824

My fear is that I'll need to do some sort of filter-branch or BFG magic and that all seems unpleasant or risky.

[ttaylorr]

@stickywes thanks for opening this and sorry that you're having trouble.

Unfortunately, unless an object is present on another developer's copy of the repository or on the server, it is unretrievable.

To work around this in your case, I would recommend removing references to the old versions of those OIDs by either removing the objects (or commit) entirely, or updating those missing OIDs to the latest known version of them.

In particular, I was unable to find any LFS object with that OID on GitHub.

[WesWedding]

@ttaylorr I appreciate the response!

I kind of figured that might be the case but I wanted some confirmation before I went that route, thank you.

[WesWedding]

Just in case someone's Google-fu brings them here, I am going to list some of the steps I took to fix this. My steps eventually required rewriting history; go grab BFG

Since I knew which introduced the problem was (see previous posts), I used git cat-file to navigate through the tree to get the blob ID for the problem file.

git cat-file -p 2b2eacf213602cb4814109119ad76a7e6d32ee78^{tree} (the hash is the commit id) in order to see the IDs for everything in the root tree of the problem commit.

Output looked like

100644 blob d47178f142af98c59fcc2b31f946a2977c5fe288    .gitattributes
100644 blob 02342ddf9916d4b43de055106138ba1e3c5a65b7    .gitconfig
100644 blob 4019f7cf4e81839d6b67a7e9ab6af783bba8fa71    .gitignore
040000 tree 31d7e6e9f125b7c744657270b7087a83de251891    Assets
040000 tree 1904165ec2b436c5ad3fa19be0cb183976e9d8c5    ProjectSettings
100644 blob f55bee1a2c07c729b7b1819590ca0fd1bcbebeed    README.md

I was looking for a file named Assets/scenes/_SCENE_.unity so I had to repeatedly navigate the tree via git-catfile.

Eventually cat-file -p 69701f7fa3a0725b9442d2dd47f55d7ebc848da5 (the tree ID for the Assets/scene/ folder) gave me the blobs I was looking for.

100644 blob 2dc93c3b7cd3ddb622ee6ee2340defb6873ea501    _SCENE_.unity
100755 blob 22575df22de4f4a6a435d4261e82afe905a1943f    _SCENE_.unity.meta

I created a text file with that ID in it, and then ran BFG to remove it from history.
java -jar bfg-1.12.15.jar -bi blob_ids

BFG recommends running additional commands; make sure you read the output of the command!

I still had some problems trying to push my new cleaned history to a new remote. Git lfs was crashing. The problem went away after git lfs prune and trying to push via SourceTree (which seems to do things a little differently), followed by git lfs fetch --all origin again. I probably had just needed to follow the prune command with the fetch and then a push (to the new remote) command.

If you don't have a new repo to push to, you're going to have the headache of trying to do a force push. There are many articles out there to guide you through the process (which you probably will mess up in some way).