Bfg-repo-cleaner: Git push has some rejections

[kazoompa]

Hi,

I recently used BFG to remove some old files and change a few texts in my test files. The clean operations went all OK without a problem. When I tried to push the changes I had the following error message at the end:

{code}
error: failed to push some refs to ''
{code}

These are some of the rejected messages:

{code}
! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
! [remote rejected] refs/pull/1/merge -> refs/pull/1/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)
! [remote rejected] refs/pull/10/merge -> refs/pull/10/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/11/head -> refs/pull/11/head (deny updating a hidden ref)
{code}

Am I in trouble? Did I just mess up my repository? Is there anyway to correct this error?

I would really appreciate any help, thanks.

[rtyley]

You're not in trouble! However this _is_ something I could cover with better documentation - issue #16 relates to this. The refs beginning 'refs/pull' are _synthetic_ read-only refs created by GitHub - you can't update (and therefore 'clean') them, because they reflect branches that may well actually come from other repositories - ones that submitted pull-requests to you.

So, while you've pushed all your real refs, the pull requests don't get updated. There's no real way to fix them with your existing repo with the GitHub admin tools. The severity of an issue depends on whether you were trying to remove any 'private' data:

• Private data : the pull requests still have that private data in their history - they will remain as a way for other people to retrieve that data. The only ways to recover from this are to a) delete the repo using the GitHub admin tools, and then recreate it, or b) contact GitHub support and ask them to delete the PRs for you (they're pretty good about it)
• Nothing Private : the pull requests still reference your old history - if any of them are still open, you don't want to merge them into your new fresh history! Get the submitters to rebase them on your new fresh history before merging.

[isomorphisms]

So if you get this error does that mean BFG has fixed the problem, or not?

[algarecu]

Hey there, could this be possibly related to the fact that BFG has duplicated most of my commit history after deleting a file from it? I have the bfg-report for reference.

[rtyley]

Hey there, could this be possibly related to the fact that BFG has duplicated most of my commit history after deleting a file from it? I have the bfg-report for reference.

In order to change Git history (ie to change a repo to make it look like that file was _never_ added) brand new commits _have_ to be created. This is how Git works - every commit-id reflects a hash of it's history - change the history, change the commit id. To delete a file, you have to create new 'clean' commit history, and throw away the old 'dirty' history.

So yes, your commit history has been duplicated, but that's essential. And this can be awkward if you have old pull requests lying around, pointing to the old history.

[takecare]

does this mean that bfg can't "clean" your history on any github repository that has had pull requests "touching" any file you're trying to remove?

what i mean is, you look at the history (on github) and you can see that the history has been re-written and the file(s) you're cleaning out cannot be seen on the commits but if you go through the pull request history or if you still have the hash for the commit the file was introduced in (as an example), you can still see it (and its contents). unless i'm wrong.

so in order for someone to remove a file for sure they'd have to:

1. run bfg, re-writing history
2. push (get this rejected error but history has been re-written)
3. ask github staff to delete all the PRs involving this file (?)
4. what about the commits?

i have a public repo in which (i believe) this issue is easily seen, in case i didn't explain myself appropriately.

[jochenwezel]

I just try to run bfg without deleting any files, without any open push requests, and it fails, too:

This is what I did:

D:\temp-cleanup-cwm>git clone --mirror https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
Cloning into bare repository 'cammIntegrationPortal.git'...
remote: Counting objects: 5979, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5979 (delta 1), reused 0 (delta 0), pack-reused 5976 eceiving objects: 100% (5979/5979), 14.99 MiB | 2.27
Receiving objects: 100% (5979/5979), 15.78 MiB | 2.27 MiB/s, done.

Resolving deltas: 100% (3801/3801), done.
Checking connectivity... done.

D:\temp-cleanup-cwm>cd temp-cleanup-cwm
Das System kann den angegebenen Pfad nicht finden.

D:\temp-cleanup-cwm>cd cammIntegrationPortal.git

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 5979, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5871/5871), done.
Writing objects: 100% (5979/5979), done.
Total 5979 (delta 3984), reused 1806 (delta 0)

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git push
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/CompuMasterGmbH/cammIntegrationPortal.git'

D:\temp-cleanup-cwm\cammIntegrationPortal.git>

After that I tried to follow the suggestions at http://stackoverflow.com/questions/34265266/remote-rejected-errors-after-mirroring-a-git-repository and did following:

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git config -e

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git remote update
Fetching origin

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git push
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/CompuMasterGmbH/cammIntegrationPortal.git'

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 5979, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5871/5871), done.
Writing objects: 100% (5979/5979), done.
Total 5979 (delta 3984), reused 1992 (delta 0)

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git push
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/CompuMasterGmbH/cammIntegrationPortal.git'

D:\temp-cleanup-cwm\cammIntegrationPortal.git>

with the following config with updated fetches:

[core]
        repositoryformatversion = 0
        filemode = false
        bare = true
        symlinks = false
        ignorecase = true
        hideDotFiles = dotGitOnly
[remote "origin"]
        url = https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
        fetch = +refs/heads/*:refs/heads/*
        fetch = +refs/tags/*:refs/tags/*
        fetch = +refs/change/*:refs/change/*
        mirror = true

But still without effect (see output above) :-(

The repository in this case is:

No (open) push requests, only 1 additional branch - so it should be very simply, isn't it?

[javabrett]

No (open) push requests, only 1 additional branch - so it should be very simply, isn't it?

I think you mean Pull Requests. You are distinguishing open PRs from closed ones, however GitHub is retaining a (read-only) ref for each PR, regardless of whether it is open or closed. This allows you to later view the PR and the diffs it represents, even if that history would have been otherwise lost/garbage-collected if it were not merged into an ongoing branch.

Since you originally followed the BFG documentation and used --mirror, your config would have had both mirror = true and a complete-wildcard fetch refspec of fetch = +refs/*:refs/*. You have since amended to fetch config, but the mirror config remains. This means your git push will behave like it has the --mirror switch added. Assuming you had already fetched and rewritten all your refs/pull refs, this will account for the read-only error.

How you deal with this depends on what you are using BFG for and what you want to achieve. If you want sensitive or large files _completely_ removed from the GitHub repo, then you need to contact GitHub support according to https://help.github.com/articles/remove-sensitive-data/ (emphasis mine):

This article tells you how to make commits with sensitive data unreachable from any branches or tags in your GitHub repository. However, it's important to note that those commits may still be accessible in any clones or forks of your repository, directly via their SHA-1 hashes in cached views on GitHub, and through any pull requests that reference them. You can't do anything about existing clones or forks of your repository, but you can permanently remove all of your repository's cached views and pull requests on GitHub by contacting GitHub Support.

In other words, GitHub Support might be able to provide further info on how you can discard the old history referenced by PRs.

If you just want bulk removed from what default (non-mirror) clones see, then you don't have to worry about these, as they aren't fetched by default, but be aware that the size of the repo on GitHub will actually _increase_ slightly as a result.

[markkolich]

Since this is a top search result for "deny updating a hidden ref", I figured I'd leave this here to help out the next person...

If you've used BFG to clean up a repo, and you're trying to force push back to GitHub, make sure you're not trying to force push back to a protected branch. I wasted several hours staring at [remote rejected] ... (deny updating a hidden ref) thinking the discussion in this issue was the cause of my grief, only to realize it's a total red herring.

Buried deep in the deny updating a hidden ref noise, was a "protected branch hook declined" failure right in my terminal.

If you're in the same boat, disable/delete branch protection, and try another git push -f --mirror:

[mark (master)]~/workspace/foobar.mirror$ git push -f --mirror
...
remote: Resolving deltas: 100% (34037/34037), done.
To https://github.com/foo/bar.git
 + a5fb13391...9b66d3db1 master -> master (forced update)    <<<---!!!!!!
 + 0978357d5...f987b9bd2 release -> release (forced update)    <<<---!!!!!!
...
 ! [remote rejected]     refs/pull/100/head -> refs/pull/100/head (deny updating a hidden ref)

Note the familiar (forced update) from Git indicating the git push -f --mirror worked fine. Subsequent clones of this repo after the force push also confirmed success.

TL;DR — I could be totally mistaken, but the [remote rejected] ... (deny updating a hidden ref) complaints appear to be just warnings (for the reasons covered earlier in this thread) and do not seem to impact the ultimate success of a force push.

@rtyley perhaps something to clarify in future documentation?

[antoniocascais]

We faced the same problem described in this issue. We had an existent PR in our Repo, so we assume that was the root cause of the push rejection.

Here's what we did (let's say that our repo was named awesome-tool):

1) Created a new empty repo with name awesome-tool-tmp;
2) In our cloned repo, we changed the URL with git set-url to the url of the new awesome-tool-tmp repo;
3) git push (at this point, both awesome-tool and awesome-tool-tmp have the same content and commits in master;
4) used the tool as described here https://rtyley.github.io/bfg-repo-cleaner/. git push now worked (because there was no PR in the tmp repo, we assume)
5) renamed awesome-tool to awesome-tool-tmp2
6) renamed awesome-tool-tmp to awesome-tool
7) deleted awesome-tool-tmp2 (the repo with the passwords)
8) asked the ppl to delete their folders and clone the repo again

And we were able to successfully remove passwords from git history.

Might not be the best solution for everyone, but in our case was enough.

[rodrigolece]

I think that I accidentally found a solution (I'm not sure I can explain why it works). Same thing, I had PRs (closed) on GitHub and git push was being rejected.

1. I cloned the local copy of my repo (using git clone --mirror but providing the path to my local directory).
2. I cleaned this copy using BFG.
3. git remote was showing origin as pointing to my local repo, so I removed origin and added it but this time pointing to my repo hosted on GitHub.
4. From the clean mirror copy, I forced a push using git push -f --set-upstream origin master.

As far as I can tell, my repo in GitHub is now clean. Git pulling from my original repo doesn't work, but now I can easily delete that folder and clone a new copy.

Any ideas for/against doing it this way?