Esmvaltool: Large binaries in the repository

[mattiarighi]

Due to an erroneous commit in #364, a large amount (~600 MB) of binaries landed in the repository.

Although they have been deleted by the next commit, they are still downloaded when cloning the repository.

A solution proposed by @bjoernbroetz allows to clean the data from the repository but requires an action by all users. Further information to be posted here.

@ESMValGroup/esmvaltool-developmentteam

[bjoernbroetz]

So the way to go here will be (as @bouweandela already wrote here ) to rewrite the history of that branch and then all of us should execute

git gc 

on all our local clones.

[jvegasbsc]

I have seen that we don have the usual binary formats in our .gitignore files.
I will add them and look for a way to prevent this from happening again in a more general way

[bjoernbroetz]

A further measure to prevent us from committing large binary files to the repository could be to setup a pre-commit hook.

[jvegasbsc]

Somehing like this https://stackoverflow.com/questions/7147699/limiting-file-size-in-git-repository?

[bjoernbroetz]

The branch is fixed, see here.
If you still have to run the git fetch && git gc on your local clone

[bjoernbroetz]

Correction: Please, issue

git reflog expire --expire=now --all &&  git gc --prune=now --aggressive

on your local clone.

[zklaus]

This can cause you to loose your stash, so git stash pop might not work anymore.
If you are not using git stash or make sure that there is no stash currently active, that is not an issue.

[zklaus]

If you already lost your stash you can find its commit (look for the blue stash label in gitk or try something like git log stash). Then with the hash you can do

git stash apply 5ce5647dc1ae4916b44725612ab26e55399c3591

to get the stash back and git stash to stow it away again.

[zklaus]

For the developers @bjoernbroetz @jvegasbsc, here is a stackoverflow answer that helps to find large objects in the repo.

[ledm]

I have run the commands:
git gc git reflog expire --expire=now --all && git gc --prune=now --aggressive
And also:
git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 | cut -c 1-12,41- | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest
from the stack overflow page linked by @zklaus.

However, I'm not entirely sure that these files have the desired effect. My ESMValTool directory is still ~1.3 GB, and the stack overflow command above still says that my repository contains 8x 54MB surge height files.

What is the expected result of these commands? (How can I tell if they worked?) Thanks!

[bjoernbroetz]

It is expected that the size of the local clone goes down to about the size of a newly cloned ESMValTool repository, i.e.: currently ca 144M (there are still some binaries there that will be removed later...)

If everything fails you could consider to create a new local clone.

[valeriupredoi]

guys, after a fresh git repo clone git fetch don't work no more and I suspect neither will git stash. Bit of a bummer because I use them both...

[zklaus]

Hej @valeriupredoi, could you be a little more specific? I just did a fresh clone followed by git fetch without a problem, so it does not seem to be a problem with the server repo per se.
What kind of error are you getting?

[valeriupredoi]

hey @zklaus man, no error, git clone and followed it by git fetch and nothing shows up after git fetch, it returns to prompt

[zklaus]

@valeriupredoi Could it be that there simply is nothing to fetch? In that case there is no output and that is usually what you expect for a fetch right after the clone.
You can try git fetch -v to get more verbose output.

[valeriupredoi]

the verbose option works well and I can see the branches again cheers :grinning:

[fserva]

Hi, I am in a situation similar to that described by @ledm. After following the instructions, the folder is still > 700 MB. It seems that most space is taken by the pack files, in ./git/objects/pack/
-r--r--r-- 1 serva serva 574K Oct 2 11:30 pack-676cd0e0b9cb013bd885d2aa85c8e2398ced78bd.idx
-r--r--r-- 1 serva serva 253K Oct 2 11:43 pack-633725fee65918461caecf0146d92cd7f781e117.idx
-r--r--r-- 1 serva serva 583M Oct 2 11:53 pack-676cd0e0b9cb013bd885d2aa85c8e2398ced78bd.pack
-r--r--r-- 1 serva serva 11M Oct 2 11:53 pack-633725fee65918461caecf0146d92cd7f781e117.pack
Do you know if it is safe to delete the larger file?
And secondly, I still have the folder "doc" and "interface_scripts" in the local branch, while they are absent from version2_development. Shouldn't they be automatically purged when pulling? Can these be removed, as well? Many thanks

[bouweandela]

See git clean --help for help on how to remove untracked stuff. If you do not have (m)any local changes, by far the easiest solution might be to just start from a clean clone of the github copy of the repository.