First-contributions: Repository size is too large.

[sagar-gavhane]

🐞 Problem


Due to repository size is too large, cloning and pushing changes towards origin repository makes terrible developer experience. Currently repository size is above than 15MB+. I don't know ideal size of standard repository but somehow we need to fix it.

🎯 Goal


If repository size is under 5MB, it will gain good developer experience for cloning and pushing to repository.

💡 Possible solutions

We can remove unnecessary files or I think contribution.md file is one of the large file, instead of putting developer name in markdown file we can show developer names on first contribution website which will serve from RESTful API.

[irhamdz]

:+1:

[digitaldina]

instead of putting developer name in markdown file we can show developer names on first contribution website

Where can I find the website? Not seeing it anywhere on the readme.

Suggestion:
-The translations for the tutorial should be on the github page so we can delete all the translation pages, significantly decreasing the size of the repository.

Possible fix:
-If no website exists, we should make one using github pages (if that's an ok idea?..)
Hope to be able to help out with this!

[Roshanjossey]

@dinaelhanan, here's a link to the webapp https://firstcontributions.github.io/

[Roshanjossey]

@sagar-gavhane, Contributors.md is just under 500KB at the moment. We won't be saving significantly by removing that. Also, I'm not sure if you understand the significance of that file. The point of that file is to help beginners help practice the process. The content of that file is not really important.

Bigger files/ directories in this repo are the images and .git directory.

We can't remove images as it helps a lot. We could look at reducing the size of .git directory but this shouldn't be at the cost of loosing commit histories.

[digitaldina]

Oh I see, it links to the github readme.

[kingokeke]

I believe I have found the file responsible for the huge size of the repo. My clone of the repo resulted in a download of over 100MB. I'd not have gone through with it if I had known about the size of the repo before hand. The location of the errant file is in the ".git/objects/pack" folder as can be seen in the attached snapshot. I'm still a n00b as concerns the workings of Git and Github, but if that file has gained over 100MB in size in the two months since this issue was posted, then this may prove a problem in future for others who may want to follow through with the tutorial but have data caps on their internet (for example, myself). If this can be remedied in some way, kindly do so. Thanks.

[digitaldina]

I'll look into it if I can, thanks @kingokeke for pointing that out!

[CraftyDH]

Since the actual files are small you could move at the end of each year the master branch to a new branch called 2018 and then do a git rebase from that branch to the master branch therefore keeping the history and making it smaller.

[digitaldina]

@CraftyDH good idea. @Roshanjossey I think this is a good idea!

[Roshanjossey]

Reopening this issue as we've had people reach out to us on slack that some text editors (Like Atom) is slow when opening Contributors.md.

The repository size is about 220MB now. In this, about 210MB is in .git.

Re initializing git will bring us down to about 20MB.

Reducing the entries in Contributors.md to about 1000 entries will help text editors too.

@CraftyDH , I don't understand how the solution you suggested works. Even if we branch out, all those git objects will still be there. We won't be really saving much. Can you try this out?

[CraftyDH]

@Roshanjossey a simpler option would be to simply have the clone code say that you can use the depth flag to download only the last 100 commits. For example:
git clone https://github.com/this-is-you/first-contributions.git --depth 100

This works because each commit is a copy of the entire repo, not a diff.

[Roshanjossey]

@CraftyDH, Using depth will work for this project. But when the contributors works on other projects, they shouldn't do it.

This project should act like a tutorial/ playground for the best practices we follow in open source communities. Teaching something that's tailored for this repo is not a good way to proceed.

[CraftyDH]

I meant we should have a side note for people who the size is too large to download after all it is a usefull to know when working on large projects which are large themselves.

[CraftyDH]

@Roshanjossey for what I said earlier we create a branch called 2019 based on the first and tell everyone to commit to that branch as we are working on that at the end of the year, create a new branch called 2020 and squash merge 2019 into that.

git checkout 9dceaffe4654c2da4d54d9f9b4efd531e94cfefb // First commit
git checkout -b 2019 // Make branch 2019
git merge master --squash // Squash all commits into 1
git commit

git checkout master
git merge 2019 // Merge all new commits back to the master branch.

[Roshanjossey]

Again, this would be something that open source community doesn't suggest you do.

Also, the repo size is still at 50 MB with .git taking up 40MB. I think we can do better.

$ git clone --depth 100 https://github.com/firstcontributions/first-contributions.git Cloning into 'first-contributions'... remote: Enumerating objects: 68370, done. remote: Counting objects: 100% (68370/68370), done. remote: Compressing objects: 100% (31032/31032), done. remote: Total 68370 (delta 40733), reused 64866 (delta 37321), pack-reused 0 Receiving objects: 100% (68370/68370), 37.54 MiB | 9.36 MiB/s, done. Resolving deltas: 100% (40733/40733), done. Checking connectivity... done. ~  Public  cd first-contributions  ✔ ~  Public  first-contributions  du -h .  ✔   master 64K ./additional-material/git_workflow_scenarios 172K ./additional-material/translations 240K ./additional-material 8.7M ./assets 4.0K ./.git/branches 44K ./.git/hooks 8.0K ./.git/refs/heads 8.0K ./.git/refs/remotes/origin 12K ./.git/refs/remotes 4.0K ./.git/refs/tags 28K ./.git/refs 8.0K ./.git/logs/refs/heads 8.0K ./.git/logs/refs/remotes/origin 12K ./.git/logs/refs/remotes 24K ./.git/logs/refs 32K ./.git/logs 40M ./.git/objects/pack 4.0K ./.git/objects/info 40M ./.git/objects 8.0K ./.git/info 40M ./.git 612K ./translations 24K ./.github 50M .

[CraftyDH]

@Roshanjossey because each commit is a full snapshot of the whole repo not just a diff the only way to make it smaller is to download less commits.

[Roshanjossey]

each commit is a full snapshot of the whole repo

This is not true (at least for this project)

Lets compare this repo to a fork of this repo

$ git clone https://github.com/this-is-you/first-contributions.git  
Cloning into 'first-contributions'...
remote: Enumerating objects: 39882, done.
remote: Total 39882 (delta 0), reused 0 (delta 0), pack-reused 39882
Receiving objects: 100% (39882/39882), 33.74 MiB | 7.66 MiB/s, done.
Resolving deltas: 100% (22613/22613), done.
Checking connectivity... done.
$ du -sh first-contributions 
41M first-contributions
$ du -sh first-contributions/.git  
35M first-contributions/.git
$ du -sh first-contributions --exclude=.git  
5.1M    first-contribution

git clone https://github.com/firstcontributions/first-contributions.git  
Cloning into 'first-contributions'...
remote: Enumerating objects: 111837, done.
remote: Total 111837 (delta 0), reused 0 (delta 0), pack-reused 111837
Receiving objects: 100% (111837/111837), 207.88 MiB | 10.89 MiB/s, done.
Resolving deltas: 100% (65933/65933), done.
Checking connectivity... done.
$ du -sh first-contributions 
222M    first-contributions
$ du -sh first-contributions/.git  
212M    first-contributions/.git
$ du -sh first-contributions --exclude=.git  
11M first-contributions

this-is-you/first-contributions has 16771 commits
firstcontributions/first-contributions has 45120 commit. So, 28349 commit ahead of this-is-you/first-contributions

If git had saved a snapshot object for all of these 28349 commits, this repo would be about

28349 * 5MB // I've taken 5MB as the size of this-is-you/first-contributions excluding .git

This is more than 138 GB

[sersorrel]

Each commit is _conceptually_ a snapshot of the whole repository, but git doesn't store duplicate copies of files – so if a file isn't changed by a commit, a new copy of that file won't be added.

Even copies of files which differ aren't stored in their entirety, though. If that was the case, then this repository would have grown far beyond a manageable size a long time ago; there are currently almost 30,000 different revisions of the Contributing.md file (via git rev-list --objects --all), increasing in size pretty linearly from 0 bytes to 800 KiB, which if you do the maths (30000 × 800KiB × 0.5) adds up to over 10 GiB!

This roughly matches up with the actual sizes of the stored blobs, too:

$ ./sizes.sh | rg Contributors.md | awk '{ print $2 }' | rg KiB | rg -o '\d+(\.\d+)?' | jq -s add
12531807
$ units 12531807KiB GiB
    * 11.951262
    / 0.083673169

(where sizes.sh is as described here)

It's honestly pretty impressive that the repository is only about 200 MiB on disk.

Working out what makes up most of the size of the repository when compressed is not as easy, unfortunately. A fair chunk of the files stored (almost 70MiB) are old JavaScript source maps from the gh-pages branch, which seem unlikely to compress well; maybe those shouldn't be committed? The minified JS also won't compress well – storing unminified JS might be better.

[Roshanjossey]

@anowlcalledjosh,

A fair chunk of the files stored (almost 70MiB) are old JavaScript source maps from the gh-pages branch

Yes, this is the problem we're trying to solve here. We used to have code for our web app here which was later moved to firstcontributions/firstcontributions.github.io with git filter-branch

Technically, we should be able to fix this by using git gc. Stack Overflow reference. I tried this and it works in my local. There was a significant reduction in repo size. But the problem is GitHub does gc at their end and it doesn't prune things that we don't need.

This is why I think re-initializing this repo is a good way to go.

[Helderjfl]

Because of the Contributors.md size, when there is a conflict, the online editor doesn't work to solve it.

[krishnadevz]

create branches in repository then size can be spilted .or remove unwanted things from the repo.

[Celestine4cpp]

The cloning is making it a terrible experience for a newbie like me. This is an open issue since 2018 and yet it has not been fixed. Please veteran contributors, kindly look into this.

[Roshanjossey]

@Celestine4cpp , I understand the pain point. This is my first priority now. I've to do some clean up and I'll be ready to significantly reduce the size.

[Celestine4cpp]

Thank you @Roshanjossey

[Roshanjossey]

$ du -sh first-contributions
18M first-contributions
$ du -sh first-contributions/.git 
8.2M    first-contributions/.git

We've brought this down to 18MB now. Closing this.