Gatsby: huge accessibility problem / support partial cloning with git filter spec

[mrhut10]

Summary

being a huge mono repo, cloning is a huge accessibility problem for those with poor/costly internet.
can we support git partial cloning using git filter spec??

Basic example

wanting to make a PR request into the "gatsby-source-shopify" npm package one must first clone the gatsby mono repo.

it downloads 740mb and size on disk is 1.8GB.

whereas @lukebennett88 and I only needed folders for our planed PR to be made is the

• "packages/gatsby-source-shopify" (a mire 228KB)
• or alternatively the whole "packages/" folder (only 12MB)

Can we support partial cloning with git filter spec to only clone the "packages" folder (12 MB)
the cloning packages folder workflow could be like this
"git clone --filter=sparse:oid=packages/.gitfilterspec"

Motivation

I am very fortunate to have good satellite internet but still feel like this huge mono repo is a barrier for myself. yet there are much less fortunate than me. many people in 3rd world countries are learning to code on pore internet and this alone could prevent then forking and providing PR's that suit them.

[pieh]

Does this work on GitHub? You linked to GitLab documentation. When I was researching this some time ago, GitHub didn't support it. We only could do partial checkout

[pieh]

We also recommend shallow clones git clone --depth=1 [...] ( https://www.gatsbyjs.org/contributing/setting-up-your-local-dev-environment/ ). While it doesn't help as much as partial cloning - this at least is known to work

[tesseralis]

The largest contributor is docs/, and the biggest contributor of that is images and video. I wonder if we can use something like git-lfs to make this faster.

[tesseralis]

@mrhut10 have you tried using sparse-checkout? I tried the method in the "Cloning in Sparse Mode" section and it seems to make it easier to clone things!

Please let us know if that works—if it's helpful, we can update our contributing docs to state that sparse checkouts are an option for decreasing download times.

[mrhut10]

Does this work on GitHub? You linked to GitLab documentation. When I was researching this some time ago, GitHub didn't support it. We only could do partial checkout

apologies @pieh I can't pretend to know if it does or doesn't from what i could gather it was a core feature of git itself documented here

but that being said sounds like a difficult thing to set up.

@tesseralis solution of sparse-checkout sounds like heaps better and easier to set up solution for users!!!! I'll have a play with that now to get familiar with it.

whatever the dream solution is loved to help document it to help more users like me whose internet isn't a dream.

[mrhut10]

I followed the tutorial @tesseralis provided here

and resulted in the repo only being 41 MB and was so fast!!!
and now contains all the packages folder I need. 👍

(packages should be the only folder @lukebennett88 and I need to make a PR to the gatsby-source-shopify plugin )

thank you @tesseralis for your solution is so awesome. 🥇

any recommendation on what's the best way to document this workflow officially to allow users to with pore internet to do the same?

[tesseralis]

@mrhut10 I think the best place to put repo setup instructions is here: https://www.gatsbyjs.org/contributing/setting-up-your-local-dev-environment/#fork-clone-and-branch-the-repository.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[danabrit]

Tagging in @AishaBlake for the documentation opportunity that @mrhut10 has so graciously offered!

[mrhut10]

Tagging in @AishaBlake for the documentation opportunity that @mrhut10 has so graciously offered!

There are some interesting things I had to learn that I haven't mentioned here too, for example when I had a pr sent to my fork you have to be careful how to check it out. As can easily lead to downloading the whole archive. Whereas I did find a way to only check out changes.

Happy to chat further about if someone wants to document the workflow.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜