bfg-repo-cleaner 🚀 - Crash with LargeObjectException

Wow- that sounds like you must have a commit message which is over 5 MB in size? Do you know how come the commit message is so long?

rtyley on 10 Jul 2013

I cant tell you, why the message is so long. I just migrated a CVS repository to GIT.
Does it help, if I reduce the size of the message afterwards?

ironm4nchen on 11 Jul 2013

Could you give me the full stack trace thrown by the BFG?

Without fully testing it, I _think_ The BFG assumes that commit messages are no bigger than 1MB in size (a consequence of this line: https://github.com/rtyley/bfg-repo-cleaner/blob/v1.7.0/bfg-library/src/main/scala/com/madgag/git/bfg/GitUtil.scala#L55 ). To help me understand what a better default for that value would be, it would be good to know why that commit message is so long, so I can better judge what size large commit messages are likely to be. Do you know precisely which commit it is that's causing the problem?

Your repository (at 4GB) is big, but I would expect The BFG to be able to handle it.

rtyley on 11 Jul 2013

Stacktrace :

Exception in thread "main" org.eclipse.jgit.errors.LargeObjectException$ExceedsLimit: Object exceeds 5.242.880 limit, actual size is 5.712.116
at org.eclipse.jgit.lib.ObjectLoader.getCachedBytes(ObjectLoader.java:191)
at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:859)
at org.eclipse.jgit.revwalk.RevWalk.parseNew(RevWalk.java:823)
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:722)
at com.madgag.git.package$RichObjectId.asRevCommit(package.scala:133)
at com.madgag.git.bfg.GitUtil$$anonfun$hasBeenProcessedByBFGBefore$3.apply(GitUtil.scala:65)
at com.madgag.git.bfg.GitUtil$$anonfun$hasBeenProcessedByBFGBefore$3.apply(GitUtil.scala:65)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at com.madgag.git.bfg.GitUtil$.hasBeenProcessedByBFGBefore(GitUtil.scala:65)
at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:45)
at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:35)
at scala.Option.map(Option.scala:145)
at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:34)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
at com.madgag.git.bfg.cli.Main.main(Main.scala)

ironm4nchen on 11 Jul 2013

We also encountered the same issue. It sounds like a hard coded limit. Is there a way to make this an adjustable parameter that can be input from the command line?

Thanks

Richard

chenri on 11 Jul 2013

It is great to see that this has been addressed so quickly in source code. However, the download link at http://rtyley.github.io/bfg-repo-cleaner/ still only shows the old version bfg-1.7.0.jar. Can you update that or provide more detailed info on how to build the jar file out of the source file?

Thanks!

Richard

chenri on 13 Jul 2013

@chenri & @ironm4nchen - busy weekend, but I've uploaded a provisional release of BFG 1.8.0 - could you test the jar download from this url, and report back? If it fixes the issue for you I'll do a formal release:

https://oss.sonatype.org/content/repositories/commadgag-1004/com/madgag/bfg/1.8.0/bfg-1.8.0.jar

$ bfg --strip-blobs-bigger-than 1K --repo-contains-massive-non-file-objects 20M

Oh, and if you can explain how you got a commit message that's over 5MB in size, I'd be genuinely interested!

rtyley on 13 Jul 2013

Hi Roberto,

Thank you very much for your quick response.

We have a team of more than 100 developers and some of them

are not very familiar with git or careful enough and so junk
files/msgs got into the git repo. This particular one was injected
into the git repo a year ago due to someone repeating
a short line a huge number of times (editing error?) so that a single
msg line was 6Mb long. We never detected this until we tried bfg last week.
This will be a perfect example for getting rid of such junk data.

Your change does work. It just got completed but I have

not looked at the results yet. Our git repo is huge. It has 3.6 mil objects
and 240,000 commits. Checked out size is 5.8Gb. Your bfg took

real 38m10.622s
user 212m57.161s
sys 6m15.407s

That is about 4 hours real cpu time. But we estimate that if
filter-branch is used, it will take about 20 hours real time.

To be in line with your option of

--strip-blobs-bigger-than

May I suggest that you change

--repo-contains-massive-non-file-objects

to

--massive-non-file-objects-bigger-than

That seems to give more meaning to the value after that option.

Also, is there a way to specify multiple file types to clean out?

For example, if we want to clean out all files of type

jpg
png
gif
3gp
ipa
ogg
ogv
webm

It would be nice to go through bfg only once.

Again, thanks for such good work and quick response.

Richard

On Sat, Jul 13, 2013 at 02:59:37PM -0700, Roberto Tyley wrote:

@chenri & @ironm4nchen - busy weekend, but I've uploaded a provisional release
of BFG 1.8.0 - could you test the jar download from this url, and report back?
If it fixes the issue for you I'll do a formal release:

https://oss.sonatype.org/content/repositories/commadgag-1004/com/madgag/bfg/
1.8.0/bfg-1.8.0.jar

$ bfg --strip-blobs-bigger-than 1K --repo-contains-massive-non-file-objects 20M

Oh, and if you can explain how you got a commit message that's over 5MB in
size, I'd be genuinely interested!

—
Reply to this email directly or view it on GitHub.*

chenri on 14 Jul 2013

due to someone repeating
a short line a huge number of times (editing error?) so that a single msg line was 6Mb long. We never detected this until we tried bfg last week.

Thanks! It's instructive to know the kind of things that can happen in the wild.

Your change does work. It just got completed but I have not looked at the results yet. Our git repo is huge. It has 3.6 mil objects and 240,000 commits. Checked out size is 5.8Gb. Your bfg took

real 38m10.622s
user 212m57.161s
sys 6m15.407s

That is about 4 hours real cpu time. But we estimate that if filter-branch is used, it will take about 20 hours real time.

... so in terms of 'human-time', the BFG took 38 minutes, which is much better than the estimated 20 hours for git-filter-branch! Being multi-threaded and taking over all CPU cores is Good Thing :smile: glad to hear that this fix lets you successfully use the BFG.

To be in line with your option of --strip-blobs-bigger-than May I suggest that you change

--repo-contains-massive-non-file-objects

to

--massive-non-file-objects-bigger-than

I hear you, but to be correct it would have to be --massive-non-file-objects-smaller-than (because the setting supports objects _smaller than_ the threshold argument, not bigger.

Also, is there a way to specify multiple file types to clean out?

The argument to --delete-files is a glob expression, so you can specify as many file types as you like for a single run of the BFG:

$ bfg --delete-files *.{txt,log,jpg,png,gif,3gp,ipa,ogg,ogv,webm}

Personally, unless there are literally _many thousands_ of chubby little binary files, I use the --strip-blobs-bigger-than switch in preference to --delete-files when trying to slim down a repository.

If you specify something like --strip-blobs-bigger-than 128K you are likely to strip out almost everything that's making your repo big without removing any actual code (source files should be smaller than that threshold) , and without removing any files that you're _currently_ using, because of the way The BFG handles protected commits. The BFG will find and remove all problematic stuff, and you won't have to make sure you list _all_ the relevant file extensions.

You can, if you wish, combine both switches in a single run of The BFG:

$ bfg --strip-blobs-bigger-than 128K --delete-files *.{txt,log,jpg,png,gif,3gp,ipa,ogg,ogv,webm}

Anyway, that's good, let me know how it goes - and thanks for replying.

rtyley on 14 Jul 2013

Hi Roberto,

Thanks for getting back to me so quickly.

The option name

--massive-non-file-objects-smaller-than

appears to be a very good choice.

Using

--delete-files '*.{txt,log,jpg,png,gif,3gp,ipa,ogg,ogv,webm}'

works. (Note that I have to quote the glob in bash session.)

One problem with specifying

--strip-blobs-bigger-than

is that you are assuming all valid files are small. This is
not case in general. All kinds of scenarios can happen. In our git
repo, we have xml, json, html, js files that are more than 20Mb in
size! You can argue that this is crazy. I agree, but in real
world git repos we can't blindly delete files like those
via --strip-blobs-bigger-than option. It is going to do damage
to the repo which can only be detected much later.

Another thing that concerns me is that after the bfg clean up,
I notice that bfg did not change anything about those commits
with huge msg lines. So is there a way to rewrite those
long msg lines via bfg? Again, using filter-branch we can do
this, but then we are back to square one with the speed issue.
I hope there is an option in bfg to handle this, yet the --help
does not seem to indicate an option in bfg for that.
Is such an option available in bfg?

Thanks

Richard

On Sun, Jul 14, 2013 at 12:36:09PM -0700, Roberto Tyley wrote:

due to someone repeating
a short line a huge number of times (editing error?) so that a single msg
line was 6Mb long. We never detected this until we tried bfg last week.
Thanks! It's instructive to know the kind of things that can happen in the
wild.
Your change does work. It just got completed but I have not looked at the
results yet. Our git repo is huge. It has 3.6 mil objects and 240,000
commits. Checked out size is 5.8Gb. Your bfg took

real 38m10.622s
user 212m57.161s
sys 6m15.407s

That is about 4 hours real cpu time. But we estimate that if filter-branch
is used, it will take about 20 hours real time.
... so in terms of 'human-time', the BFG took 38 minutes, which is much better
than the estimated 20 hours for git-filter-branch! Being multi-threaded and
taking over all CPU cores is Good Thing :smile: glad to hear that this fix lets
you successfully use the BFG.
To be in line with your option of --strip-blobs-bigger-than May I suggest
that you change

--repo-contains-massive-non-file-objects

to

--massive-non-file-objects-bigger-than
I hear you, but to be correct it would have to be
--massive-non-file-objects-smaller-than (because the setting supports objects
smaller than the threshold argument, not bigger.
Also, is there a way to specify multiple file types to clean out?
The argument to --delete-files is a glob expression, so you can specify as many
file types as you like for a single run of the BFG:

$ bfg --delete-files *.{txt,log,jpg,png,gif,3gp,ipa,ogg,ogv,webm}

Personally, unless there are literally many thousands of chubby little binary
files, I use the --strip-blobs-bigger-than switch in preference to
--delete-files when trying to slim down a repository.

If you specify something like --strip-blobs-bigger-than 128K you are likely to
strip out almost everything that's making your repo big without removing any
actual code (source files should be smaller than that threshold) , and without
removing any files that you're currently using, because of the way The BFG
handles protected commits. The BFG will find and remove all problematic stuff,
and you won't have to make sure you list all the relevant file extensions.

You can, if you wish, combine both switches in a single run of The BFG:

$ bfg --strip-blobs-bigger-than 128K --delete-files *.{txt,log,jpg,png,gif,3gp,ipa,ogg,ogv,webm}

Anyway, that's good, let me know how it goes - and thanks for replying.

—
Reply to this email directly or view it on GitHub.*

chenri on 15 Jul 2013

I notice that bfg did not change anything about those commits with huge msg lines.

@chenri I've opened a new RFE issue for this in #21 - it would be good to support this if it's possible to do it in a nice way. Can you comment on that issue to describe what exact modification you think should take place (truncation, substitution, etc)?

rtyley on 15 Jul 2013

I think being able to do substitution will be very useful and powerful:

--replace-msg-lines 's{^.{1000000,1000000000}$}/replaced msg line/s'

Note that this assumes that the powerful perl compatible extended
regex is supported in bfg.

Thanks

Richard

On Mon, Jul 15, 2013 at 12:32:43AM -0700, Roberto Tyley wrote:

I notice that bfg did not change anything about those commits with huge msg
lines.
@chenri I've opened a new RFE issue for this in #21 - it would be good to
support this if it's possible to do it in a nice way. Can you comment on that
issue to describe what exact modification you think should take place
(truncation, substitution, etc)?

—
Reply to this email directly or view it on GitHub.*

chenri on 15 Jul 2013

I'm trying to leverage bfg to clean up a repo converted from cvs and still running into this problem. Using the "massive-non-file-objects-sized-up-tp" flag doesn't seem to change the limit that's reported in the error. Am I missing something?

java -jar bfg-1.11.1.jar --strip-blobs-bigger-than 1M --protect-blobs-from master,test --massive-non-file-objects-sized-up-to 20M myrepo.git

Using repo : /Users/blah/myrepo.git

Exception in thread "main" org.eclipse.jgit.errors.LargeObjectException$ExceedsLimit: Object f839c0f7f552be6064c9745c1c7ca10e910f136f exceeds 5,242,880 limit, actual size is 5,472,604
at org.eclipse.jgit.lib.ObjectLoader.getCachedBytes(ObjectLoader.java:191)
at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:859)
at org.eclipse.jgit.revwalk.RevWalk.parseNew(RevWalk.java:823)
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:722)
at com.madgag.git.package$RichObjectId.asRevCommit(package.scala:180)
at com.madgag.git.bfg.GitUtil$$anonfun$hasBeenProcessedByBFGBefore$3.apply(GitUtil.scala:64)
at com.madgag.git.bfg.GitUtil$$anonfun$hasBeenProcessedByBFGBefore$3.apply(GitUtil.scala:64)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at com.madgag.git.bfg.GitUtil$.hasBeenProcessedByBFGBefore(GitUtil.scala:64)
at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:47)
at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:34)
at scala.Option.map(Option.scala:145)
at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:33)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
at com.madgag.git.bfg.cli.Main.main(Main.scala)

rufusisnodufus on 27 Feb 2014

Using the "massive-non-file-objects-sized-up-tp" flag doesn't seem to change the limit that's reported in the error.

That's curious. There _is_ an integration test against that particular piece of code:

https://github.com/rtyley/bfg-repo-cleaner/blob/ff4a758b/bfg/src/test/scala/com/madgag/git/bfg/cli/MainSpec.scala#L80-L88

Are you able to share the repo with me? Sadly, debugging & fixing this problem will take a lot longer if I don't have access to a reproducible test-case.

rtyley on 27 Feb 2014

Unfortunately I can't :-/ Could there be a limit in the jvm somewhere that this is bumping up against?

Example of what I was talking about though:
java -jar bfg-1.11.1.jar --strip-blobs-bigger-than 1M --protect-blobs-from master,broken-3-0 --massive-non-file-objects-sized-up-to 20M test.git
Exception in thread "main" org.eclipse.jgit.errors.LargeObjectException$ExceedsLimit: Object f839c0f7f552be6064c9745c1c7ca10e910f136f exceeds 5,242,880 limit, actual size is 5,472,604

java -jar bfg-1.11.1.jar --strip-blobs-bigger-than 1M --protect-blobs-from master,broken-3-0 --massive-non-file-objects-sized-up-to 1M test.git
Exception in thread "main" org.eclipse.jgit.errors.LargeObjectException$ExceedsLimit: Object 1e25e255696f87585d18e00e51c4ddcede3a02ce exceeds 5,242,880 limit, actual size is 5,956,072

java -jar bfg-1.11.1.jar --strip-blobs-bigger-than 1M --protect-blobs-from master,broken-3-0 --massive-non-file-objects-sized-up-to 100M test.git
Exception in thread "main" org.eclipse.jgit.errors.LargeObjectException$ExceedsLimit: Object f839c0f7f552be6064c9745c1c7ca10e910f136f exceeds 5,242,880 limit, actual size is 5,472,604

I would expect that limit to change given the different inputs, but it doesn't seem to. Unless I'm misunderstanding what the flag is meant to do.

rufusisnodufus on 27 Feb 2014

Got it!

I extended the heap size in java and that coupled with the massive... flag seemed to do the trick. Here's my working command for posterity:

java -Xms512m -Xmx4g -jar bfg-1.11.1.jar --strip-blobs-bigger-than 1M --protect-blobs-from master,broken-3-0 --massive-non-file-objects-sized-up-to 25M test.git

rufusisnodufus on 27 Feb 2014

👍2

Hmm... that's very weird. I wouldn't have expected that to work, but well done for getting it going!

rtyley on 27 Feb 2014

Got the same issue but the above flag didnt do the trick. The previous maintainer of this particuliar repo once thought it was a good idea to commit all photos... Any idea what can I do ?
Ideally I would like to keep the history but this big commit keep me from uploading the repo to github so I will probably get rid of the history and start with a clean repo if there is no solution.

Scanning packfile for large blobs: 31419
Scanning packfile for large blobs completed in 392 ms.
Found 1 blob ids for large blobs - biggest=142363579 smallest=142363579
Total size (unpacked)=142363579
Found 21716 objects to protect
Found 5 commit-pointing refs : HEAD, refs/heads/master, refs/remotes/bc/master, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit d93ad1ed (protected by 'HEAD')

Cleaning
--------

Found 859 commits
Cleaning commits:        77% (662/859)Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner.apply(ObjectIdCleaner.scala:75)
    at com.madgag.git.bfg.cleaner.RepoRewriter$$anonfun$clean$1$1$$anonfun$apply$mcV$sp$3.apply(RepoRewriter.scala:112)
    at com.madgag.git.bfg.cleaner.RepoRewriter$$anonfun$clean$1$1$$anonfun$apply$mcV$sp$3.apply(RepoRewriter.scala:111)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at com.madgag.git.bfg.cleaner.RepoRewriter$$anonfun$clean$1$1.apply$mcV$sp(RepoRewriter.scala:110)
    at com.madgag.git.bfg.cleaner.RepoRewriter$$anonfun$clean$1$1.apply(RepoRewriter.scala:103)
    at com.madgag.git.bfg.cleaner.RepoRewriter$$anonfun$clean$1$1.apply(RepoRewriter.scala:103)
    at com.madgag.git.bfg.Timing$.measureTask(timing.scala:39)
    at com.madgag.git.bfg.cleaner.RepoRewriter$.clean$1(RepoRewriter.scala:103)
    at com.madgag.git.bfg.cleaner.RepoRewriter$.rewrite(RepoRewriter.scala:144)
    at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:59)
    at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:34)
    at scala.Option.map(Option.scala:146)
    at com.madgag.git.bfg.cli.Main$.delayedEndpoint$com$madgag$git$bfg$cli$Main$1(Main.scala:33)
    at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:27)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
    at com.madgag.git.bfg.cli.Main.main(Main.scala)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$uncachedClean$1.apply(ObjectIdCleaner.scala:87)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$uncachedClean$1.apply(ObjectIdCleaner.scala:85)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 28 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.model.CommitArcs.cleanWith(Commit.scala:57)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$1.apply(ObjectIdCleaner.scala:102)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$1.apply(ObjectIdCleaner.scala:98)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 39 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:126)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:125)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.Map$Map3.foreach(Map.scala:161)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:125)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 51 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:126)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:125)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
    at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:125)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 69 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:126)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:125)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:125)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 88 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:126)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:125)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
    at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:125)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 106 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:126)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:125)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
    at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:125)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 125 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.madgag.git.bfg.MemoUtil$$anonfun$concurrentCleanerMemo$1$$anon$1.apply(memo.scala:60)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:126)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4$$anonfun$5.apply(ObjectIdCleaner.scala:125)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:125)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 144 more
Caused by: org.eclipse.jgit.errors.LargeObjectException: e989c403271269db9dabc5a6c85588fa128d4ca0 exceeds size limit
    at org.eclipse.jgit.internal.storage.file.LargePackedWholeObject.getCachedBytes(LargePackedWholeObject.java:98)
    at org.eclipse.jgit.treewalk.CanonicalTreeParser.reset(CanonicalTreeParser.java:202)
    at com.madgag.git.bfg.model.Tree$.entriesFor(Tree.scala:39)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:119)
    at com.madgag.git.bfg.cleaner.ObjectIdCleaner$$anonfun$4.apply(ObjectIdCleaner.scala:118)
    at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    ... 162 more

a5e on 10 Feb 2015

Got the same issue but the above flag didnt do the trick.

@a5e can you include the full command-line invocation you're using? The BFG flag is --massive-non-file-objects-sized-up-to <size> by the way.

rtyley on 10 Feb 2015

👍1

Oups, I was referring to the trick for increasing java heap size. My fault. I ran with the BFG flag and it took care of it.

a5e on 10 Feb 2015

Bfg-repo-cleaner: Crash with LargeObjectException

Most helpful comment

All 20 comments

Related issues