Bfg-repo-cleaner: Size of the Repo Increased

[sidd-shah]

I followed the procedure as mentioned in the documentation but the size of my repo increased from 199MB to 355MB, i have no clue of how to revert the push as it cannot be seen in the list of commits also. Kindly help.

[rtyley]

Hi @sidd-shah - I try to help out when users have problems, but you've not provided enough context for me to be able to help you. If you like, send me some more information and I'll see if I can help:

• Can you supply all the commands run while doing this work, including git gc, push, etc?
• Do you still have the zipped archive of your original repo?
• How many people share this repo?
• Where is it hosted?

[sidd-shah]

Thanks for the quick reply!

• I followed the exact same commands as mentioned at https://rtyley.github.io/bfg-repo-cleaner/ in the usage section.
• I do not have the zipped archive of the original repo
• 2 people share the repo
• It is hosted on Bitbucket

The weird part is after completing the steps in usage the size of the git archive is being reduced locally(164mb) but when i check in bitbucket settings after pushing it shows 355mb.

Thanks again for the help.

[rishigoelbr]

Hi rtyley,

I am having the same problem. I used the procedure mentioned in the link https://rtyley.github.io/bfg-repo-cleaner/ but it actually increased the size of my repo in bitbucket. Here are the details:

Commands Executed:
git clone --mirror [email protected]:team/big-repo.git
java -jar bfg-1.12.3.jar --strip-blobs-bigger-than 2M big-repo.git
cd big-repo.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
git push

I took a copy of my original big-repo and created a new temporary repo in bitbucket so yes I have a back up.
This is my testing repo so nobody else uses this repo.
It is hosted on bitbucket.

So prior to this procedure, bitbucket showed that the size of repo is 32.6 MB and now it is showing me 63.7 MB.
Could you please try and see what's going on?

Thanks
Rishi

[lgauthier]

I'm running into the same issue with bitbucket as well. In the comments on this page it looks like you have to email bitbucket support and request an additional git gc on your repo.

[abbiekressner]

This is a problem for me as well. I emailed Bitbucket Support as @lgauthier suggested and am waiting for their response.

[abbiekressner]

[Update] Bitbucket Support responded within an hour or so, ran 'git gc', and my repository went from >2GB down to 196.9 MB. Success!

[nishatafshan]

@rtyley I am facing the same issue. My repository size has blown up. I followed the exact same steps mentioned in https://rtyley.github.io/bfg-repo-cleaner/ .
Before starting the cleanup I had around 6.4G data in the disk (du -sh .)
I ran the steps mentioned and locally before push I could see that the big files were listed in deleted files list log and also the size now was only 1.4G. Being happy with the result I did a git push.
It took quiet long and after about 2 hours I got below error:-
git push
Counting objects: 218648, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (81662/81662), done.
Writing objects: 100% (215818/215818), 666.89 MiB | 1.07 MiB/s, done.
Total 215818 (delta 119117), reused 210562 (delta 115381)
remote: Communication breakdown with Stash.
To ssh://git.../bigRepo.git
! [remote rejected] Branch -> Branch (pre-receive hook declined)
! [remote rejected] CUC-112 -> CUC-112 (pre-receive hook declined)
...
error: failed to push some refs to 'ssh://git.../bigRepo.git'

With the message I understood that the cleanup did not work as I was not able to completely push to repository. But to my surprise when I fresh cloned the repo and checked the size it is 7.4G (increased by 1G). I am really not sure what is going on here and in urgent need of support. I used bfg-1.12.8.jar for the cleanup. I am also attaching the inubit.git.bfg-report for a detailed look.
bfgReport.zip

[nishatafshan]

@rtyley Please give your advise on the above issue.

[rtyley]

Reasons why a repo can appear to get bigger after a BFG clean-up:

• NOT_ALL_REFS_UPDATED : Not _all_ refs can be pushed to the shared repository (because of a pre-receive hook rejecting them or some other reason) - this can mean you end up in a situation where two copies of history are now present in your shared repo, and so the size of the repo will increase slightly, rather than go down to the cleaned size you got in your local clean-up repo. The reasons for being unable to push refs can vary - if they are synthetic refs like refs/pull/nn/head in GitHub it may not even matter - but depending on your Git hosting setup, there could be several different reasons why the push is rejected, and these all fall outside the responsibility of the BFG
• OLD_HISTORY_REINTRODUCED : The repo is successfully pushed up, but then a copy of the old, dirty repo history is pushed up (perhaps by users who didn't delete their copies of the old repo - they should have done a fresh clone of the new repo), creating a size problem similar to NOT_ALL_REFS_UPDATED. This can get even worse if humans or software try to merge the two histories together, at this point you get the GUITAR_HERO_MERGE problem.
• HOST_REPO_SIZE_IS_BOGUS : This is not too bad, it's just that the figure that many hosting services (GitHub, BitBucket, etc) don't immediately clean away all the detritus from your clean-up on _their_ copy of the repo - so they report the size of [old unwanted rubbish] + [cleaned history]. If this is the only issue, you'll notice that the repo size is actually much smaller when you clone it down to disk (because you're only pulling down [cleaned history]). You can get the hosting service to run a git gc on your repo if you email them and ask nicely, like @abbiekressner did - and then the [old unwanted rubbish] will be trimmed away.

@nishatafshan thanks for including the bfg report, I appreciate that. Unfortunately, without access to the repo itself, problem diagnosis from my end can still be difficult. My best guess is that you have NOT_ALL_REFS_UPDATED. You will have to work out why you couldn't push those refs.

I have very limited time, don't get paid to work on the BFG, and have two children to look after. I can't always respond to user queries quickly, it's worth posting to StackOverflow: https://github.com/rtyley/bfg-repo-cleaner/blob/master/CONTRIBUTING.md#issues-and-questions - there are many more people who can answer that sort of question on Stackoverflow, you stand a good chance of getting your question answered quicker!

[nishatafshan]

@rtyley thank you for sparing some time for me. I am highly obliged. Thanks again.