Gatsby: Reduce size of files checked into the repo

[jlengstorf]

Description

We have several files checked into the repo that are pretty large (this one is 8.6MB). They all appear to be blog images, so I don't think there's any reason to keep them at that size.

The repo is currently _huge_ right now (I'm watching @jjprevite download it and we're at 200MB+ / over 5 minutes to clone), so finding and shrinking all these images should make it much faster to clone the repo for development.

Steps to reproduce

Do a fresh clone of the repo after clearing caches.

Expected result

The repo should clone quickly for development.

Actual result

Repo takes a _long_ time to clone.

[KyleAMathews]

Makes sense! No need to keep images larger than e.g. 3x their intended size (we build 3x/2x/1x resolutions).

You might need to rewrite history though for older versions.

We could also start encouraging people to do shallower checkouts to speed up the initial checkout given Gatsby is 3+ years old now.

[jlengstorf]

You can find all files over 1M in the repo by running this in the root of the Gatsby repo:

find . -type f -size +1M -print0 | xargs -0 ls -Shal | head

It looks like the main offender is this 44MB monster. Do we need to have that checked in? If so, should we move that package to its own repo to avoid huge downloads for everyone else?

[KyleAMathews]

Not sure what that's there for. @ThatOtherPerson — do you remember why you added it?

[roachnt]

I can help with this. Where do you recommend I start?

[jlengstorf]

@roachnt I'd run the find command posted above, then resize any images there to be their @2x web size (1,240px wide). That ought to eliminate the image bloat.

There are also two pretty large MP4s (~10MB each) in this post: https://www.gatsbyjs.org/blog/2018-04-11-trying-out-gatsby-at-work-and-co/ — should those be moved to the Gatsby YouTube channel? @KyleAMathews any thoughts on that?

Otherwise, we need to hear back from @ThatOtherPerson about the screenshot plugin to see if we can drop that tarball.

@roachnt if you want to take a crack at the images, that would be an amazing help!

[jlengstorf]

@roachnt a better command that ignores built output and node_modules:

find . -type f -size +500k  -not -path "./.git/*" -not -path "*/node_modules/*" -not -path "*/.cache/*" -not -path "*/public/*" -print0 | xargs -0 ls -Shal

[KyleAMathews]

Youtube or handbrake — I've checked in other MP4s and handbrake normally shrinks them down to a few hundred kbs.

[roachnt]

I can optimize these images as well, just running them through ImageOptim before resizing takes ~20-30% off the images on average.

[KyleAMathews]

Nice! Just don't do lossy compressions please.

[roachnt]

Can do! 👍

[roachnt]

6507 here's a PR for the images!

[Ankcorn]

How about using Git LFS https://git-lfs.github.com/ for all the discussed items.

[m-allanson]

How about using Git LFS https://git-lfs.github.com/ for all the discussed items.

Hmmmmm interesting! That looks like it could be pretty useful, what happens if you try and interact with the repo _without_ installing git-lfs first?

I'd be concerned that requiring git-lfs could be too much of a barrier for new contributors - particularly if they're just wanting to submit blog posts or documentation changes. It'd make the process of checking out Gatsby different to 99% ( ⚠️ _made-up stats warning_ ⚠️ ) of other projects on GitHub.

[Ankcorn]

GIt for windows comes with git-lfs included. But on mac and linux you need to install it. I will test the behavior with other git clients and check back here with the result.

[roachnt]

7841

[cassiebeckley]

@KyleAMathews sorry, I somehow missed this! That is Chrome. It's a special build that works on Lambda. It'd probably be possible to have it on S3 or something and download before deploy

[m-allanson]

@ThatOtherPerson would something like this make a good replacement for the checked-in copy of Chrome? https://github.com/alixaxel/chrome-aws-lambda