Bfg-repo-cleaner: push denied after deleting files with bfg

Created on 29 Apr 2014 · 28Comments · Source: rtyley/bfg-repo-cleaner

After deleting select files from my repo with bfg, I'm denied when using git push.

The commands that I used (repo name changed to protect the innocent):

git clone --mirror /path/to/my/repo.git
bfg -D file1 repo.git
bfg -D file2 repo.git
bfg -D file3 repo.git
cd repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push

The output from 'git push'

Counting objects: 857, done.
Delta compression using up to 40 threads.
Compressing objects: 100% (280/280), done.
Writing objects: 100% (817/817), 6.25 MiB, done.
Total 817 (delta 513), reused 803 (delta 503)
remote: error: denying non-fast-forward refs/heads/develop (you should pull first)
remote: error: denying non-fast-forward refs/heads/feature/align_breakpoints (you should pull first)
remote: error: denying non-fast-forward refs/heads/master (you should pull first)
To /var/git/sharchaea/genome_align/
! [remote rejected] develop -> develop (non-fast-forward)
! [remote rejected] feature/align_breakpoints -> feature/align_breakpoints (non-fast-forward)
! [remote rejected] master -> master (non-fast-forward)
error: failed to push some refs to '/var/git/sharchaea/genome_align/'

Any idea what's going on?
Thanks

Source

nick-youngblut

👍1

Most helpful comment

in work with github i see the same errors (clean with -rt flag):

 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to '[email protected]:***/***.git'

akaguny on 12 Jan 2018

👍8

All 28 comments

Can you tell me who's hosting the remote Git server? GitHub, BitBucket, perhaps an internal corporate service?

I think it's quite likely the remote server has been configured to reject non-fast-forward updates - meaning that it's disallowing history-rewriting! Does the remote have receive.denyNonFastForwards true set in it's config?

rtyley on 29 Apr 2014

Sorry for not specifying that in my initial comment. I was trying to push to GitHub, so I'm guessing that history-rewriting is not the issue.

nick-youngblut on 29 Apr 2014

Are you sure you cloned with --mirror? Could you repeat the steps just to be sure?

http://rtyley.github.io/bfg-repo-cleaner/#usage

rtyley on 29 Apr 2014

🎉1

I repeated twice before I posted the original comment. The commands in my original comment pasted from my history (with file names changed).

nick-youngblut on 30 Apr 2014

I was unable to do a git push after deleting a file. I got the following error:

fatal: remote error: 
  You can't push to git://github.com/orthodoc/my.git
  Use https://github.com/orthodoc/my.git

I did use the --mirror flag when cloning

orthodoc on 11 May 2014

You cloned using a read-only url. This is why you couldn't push your
updated history back.

https://help.github.com/articles/which-remote-url-should-i-use
On 10 May 2014 23:31, "Biswajit Dutta Baruah" [email protected]
wrote:

I was unable to do a git push after deleting a file. I got the following
error:

fatal: remote error:
You can't push to git://github.com/orthodoc/my.git
Use https://github.com/orthodoc/my.git

I did use the --mirror flag when cloning

—
Reply to this email directly or view it on GitHubhttps://github.com/rtyley/bfg-repo-cleaner/issues/41#issuecomment-42756358
.

rtyley on 11 May 2014

Does that mean that I have to clone like this:

git clone --mirror https://github.com/orthodoc/my.git

orthodoc on 11 May 2014

Yup - use either that or the SSH url. The generic git protocol you're using
is not passing any authentication credentials, so GitHub is obviously not
going to accept pushes to your repo!
On 10 May 2014 23:44, "Biswajit Dutta Baruah" [email protected]
wrote:

Does that mean that I have to clone like this:

git clone --mirror https://github.com/orthodoc/my.git

—
Reply to this email directly or view it on GitHubhttps://github.com/rtyley/bfg-repo-cleaner/issues/41#issuecomment-42756652
.

rtyley on 11 May 2014

👍1

I did that. Successfully pushed the changes. Some refs failed to go through. Probably Issue #36 that you have already explained. I checked history of my repo and the its squeaky clean. bfg made my day. I can't thank you enough.

orthodoc on 11 May 2014

Hi,
I am facing similar issue. I have cloned using ssh url.
git clone --mirror ssh://git@server/repo.git

java -jar bfg-1.12.8.jar --strip-blobs-bigger-than 100M repo.git
cd repo.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive

Above steps cleans my repo.git from 6.4G to 1.2G. Later I try to push.
git push

But I get below error:-
Counting objects: 217307, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (80946/80946), done.
Writing objects: 100% (214361/214361), 666.42 MiB | 1.15 MiB/s, done.
Total 214361 (delta 118358), reused 209184 (delta 114750)
remote: You are attempting to update refs that are reserved for Stash's pull request
remote: functionality. Stash manages these refs automatically, and they may not be
remote: updated by users.
remote:
remote: Rejected refs:
remote: refs/pull-requests/190/from
remote: refs/pull-requests/247/from
remote: refs/pull-requests/247/merge
To ssh://git@server/repo.git
! [remote rejected] Branch -> Branch (pre-receive hook declined)
...
error: failed to push some refs to 'ssh://git@server/repo.git'

How should i fix it. Should I try with https url?

nishatafshan on 18 Jan 2016

The error message is fairly clear - Stash won't let you push refs under "pull-requests". There's a few things you could do, including renaming that branch in your local, making it non-tracking, deleting it etc. I don't know what the impact on clean-up will be in Stash if it is retaining those refs - maybe there is a clean-up process for those.

javabrett on 19 Jan 2016

@javabrett You are correct. I had to get rid of the pull requests. I got them merged / declined and then started with the bfg clean up steps. There is another way to get rid of pull request from mirror clone - http://christoph.ruegg.name/blog/git-howto-mirror-a-github-repository-without-pull-refs.html

nishatafshan on 20 Jan 2016

@javabrett @rtyley Would you have any ideas on how to resolve below issue.

Post cleanup when I try push I am getting below error:-
Counting objects: 218648, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (81662/81662), done.
Writing objects: 100% (215818/215818), 666.89 MiB | 1.07 MiB/s, done.
Total 215818 (delta 119117), reused 210562 (delta 115381)
remote: Communication breakdown with Stash.
To ssh://repo.git
! [remote rejected] Branch ->Branch (pre-receive hook declined)
! [remote rejected] CUC-112 -> CUC-112 (pre-receive hook declined)
...
error: failed to push some refs to 'ssh://repo.git'

nishatafshan on 26 Jan 2016

It looks like a hook, pre-receive hook, is declining the push, but may be doing-so silently. You need to establish why that hook is rejecting the pushes to those refs, which may involve making the hook output more-verbose. Perhaps those refs are locked/protected by Stash.

javabrett on 26 Jan 2016

@javabrett there are no explicit hooks set in stash for the repo. How do I find pull more details to judge the cause?

nishatafshan on 27 Jan 2016

I recommend:

web search of that error message and go through the list of possible causes
checking the Stash server logs
contacting Atlassian support

javabrett on 27 Jan 2016

in work with github i see the same errors (clean with -rt flag):

 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to '[email protected]:***/***.git'

akaguny on 12 Jan 2018

👍8

in work with github i see the same errors (clean with -rt flag):

 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to '[email protected]:***/***.git'

Hi @akaguny , what did you mean by "clean with -rt flag"?

I am also trying to clean up my repository following the manual and then pushing it back into github using these commands:

>git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
>git reflog expire --expire=now --all
>git gc --prune=now --aggressive
>git push

Then I got the same errors as @akaguny:

Counting objects: 7641, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3731/3731), done.
Writing objects: 100% (7641/7641), 56.76 MiB | 16.46 MiB/s, done.
Total 7641 (delta 3812), reused 7641 (delta 3812)
remote: Resolving deltas: 100% (3812/3812), done.
To https://github.com/abc/???.git
 + bef1e57...12792c1 bioconductor -> bioconductor (forced update)
 + 2504b81...ecb033d gh-pages -> gh-pages (forced update)
 + 4365d3b...43a7c82 master -> master (forced update)
 + 2eac6e7...346947d v0.1.0 -> v0.1.0 (forced update)
 + 4b2eca8...e522f0d v0.1.1-beta -> v0.1.1-beta (forced update)
 + d61480b...51034ad v0.2.0 -> v0.2.0 (forced update)
 + e5e5a97...e8b889d v0.2.1 -> v0.2.1 (forced update)
 + 29bca00...4709246 v0.3.0 -> v0.3.0 (forced update)
 + a53becf...ff9fa7b v0.3.1 -> v0.3.1 (forced update)
 + 135edb8...f734108 v0.3.2 -> v0.3.2 (forced update)
 ! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/head -> refs/pull/17/head (deny updating a hidden ref)
...
 ! [remote rejected] refs/pull/95/head -> refs/pull/95/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/abc/???.git'

Does anyone know the solution?
Many thanks!!

trvinh on 18 Feb 2019

If you need to delete Pull Requests (and their hidden branches) permanently (because they contain history you want to delete), you have to contact GitHub Support with that request.

javabrett on 20 Feb 2019

👍4

thanks so much @javabrett. I will try to contact them.

trvinh on 20 Feb 2019

Isn't this just simply because you rewrite history? After running these commands you have a completely different history (in the eyes of github for example). I had to force push because of it, but it makes sense to me.

RWOverdijk on 20 Mar 2019

@RWOverdijk did you rewrite a repo cloned from GitHub, including pull-requests, and try and push that rewritten history back, to GitHub? That's what a lot of the issues in this issue relate to, and the topic of my comment.

The refs/pull/ namespace in GitHub is read-only, and will retain old, pre-rewrite history, and block you from pushing new history.

javabrett on 20 Mar 2019

I did do that, but using push --force made github yield to my bidding.

RWOverdijk on 20 Mar 2019

Interesting. Perhaps GitHub opened this up? To confirm beyond doubt you'd need to:

have a GitHub repo with Pull Requests with history that will be rewritten
clone the repo in a way that ensures those PR branches are cloned
note head commit on some PR
run BFG and check that the PR commit hash changed
force push back to GItHub
check that the PR on GitHub has the new history, and that old history can't be referenced through the PR(s).

In my recollection, PRs were always read-only, so this would fail without contacting GitHub support and asking for PRs to be dropped.

javabrett on 20 Mar 2019

Ah, no I don't do this through a PR. I force push to master. I may have missed that part 😅

RWOverdijk on 20 Mar 2019

Because I had never pushed to the origin before, you need to set it with the push, so...