Carthage: Question: Allow shallow copy

[HHuckebein]

We are using a mix of 3rd party dependencies (AFNetworking an alike) and frameworks developed internally.
I wonder if carthage could have a keyword denoting "--depth 1" when a repository is fetched.

[karstenlitsche]

Hello old friend ;-)
As I understand carthage clones the repositories into ~/Library/Caches/org.carthage.CarthageKit only once. So git 'shallow copy' has no benefits.

[HHuckebein]

Nice to hear from you too.
We are using carthage update --platform iOS --no-use-binaries --use-submodules which creates for AFNetworking a inside Carthage/Checkout a clone with the complete history (which is in our case not necessary)

[mdiep]

I don't think saving some disk space is worth the complexity here.

[karstenlitsche]

Currently there are no parameters defined.
see checkoutSubmodule: https://github.com/Carthage/Carthage/blob/master/Source/CarthageKit/Git.swift#L321

[HHuckebein]

I'm not talking about space but time to download in generell or, if the repo has a large history, that the time packaging it and/or buffer space prevents the downloading at all. (Especially if security measures in place are very strict).

[HHuckebein]

@karstenlitsche Thx for pointing out the right place

[kenkistudio]

sorry for bringing up this old discussion. @HHuckebein i'm actually in supportive of shallow clone for the same reason.
Wondering if any plan to include it in future release?

[mdiep]

No, in fact, this is something that we don't want to do for fetching. Doing it for local checkouts from the Carthage cache is also not a great idea because it adds complexity for users.

[seivan]

@mdiep I am sorry to open this again, but I need to distribute binaries for Cocoapods, and Submodules. This causes the repository to grow large for initial installs.

Is there a good reason to clone the entire tree if the user has specified in their Cartfile that they are only interested in the prebuilt binary tagged 2.2.2?
From the looks of it, it clones the entire tree and that is currently 2 gigs worth.

[mikhailmelnik]

I have a different issue which requires the same resolution. On the build machine git plays weird with HTTPS breaking on large repositories so setting depth to 1 helps in many cases.

[werediver]

Can Carthage checkout repositories into its cache with --depth 1 option?

I don't think the cons of shallow copying in CocoaPods apply to Carthage.

[mdiep]

I think those cons do apply to Carthage's cache—although perhaps not as extremely.

[werediver]

@mdiep Those cons are about the extra load on the central spec-repository hosting. Carthage doesn't have any central repos, thus can't overload anything. The cache is a local entity, can't see how it can be overloaded.

That's what I see after a shallow overview of this post.

[mdiep]

Ironically, this practice can be much more expensive than full fetches/clones, especially over the long term. It is usually preferable to pay the price of a full clone once, then incrementally fetch into the repository, because then Git is better able to negotiate the minimum set of changes that have to be transferred to bring the clone up to date.

[werediver]

That's about computational cost of a fetch vs. time of download (for fat repositories). CocoaPods' master spec-repository is really _fat_, but common dependencies aren't (so) — that's my point.

I dropped in thinking about storing some binaries in a Git-repository, though, and LFS is anyway a better option in such a case. Thank you for your replies.

[mdiep]

It's not just about computational cost of a fetch—it's also about the number of bits that need to be downloaded.

[werediver]

Yeah, of course. But you're talking about 1 MB (optimal packaging for a fetch from HEAD^ to HEAD) vs. 2 MB (suboptimal packaging for a fetch from HEAD^ to HEAD) where I'm talking about 2 MB (suboptimal packaging for a fetch from HEAD^ to HEAD) vs. 200 MB (optimal packaging for a fetch from zero to HEAD with the whole looong history).

And in a long term perspective a shallow repository will converge to almost full clone (as mentioned in that great explanatory post), so the problem of suboptimal packaging will also cease. This way it seems to me that a shallow copy is good for repositories with a long history (except for CocoaPods' master spec repository and alike, because of _extremely_ heavy load) and doesn't make significant difference for small ones.

But I'm not an expert in Git internals, so this is just a conclusion from what I see have been discussed in these couple of threads.

[sudeepdino008]

I'm having the same issue.

Ironically, this practice can be much more expensive than full fetches/clones, especially over the long term. It is usually preferable to pay the price of a full clone once, then incrementally fetch into the repository, because then Git is better able to negotiate the minimum set of changes that have to be transferred to bring the clone up to date.

The purpose of carthage repo is not to do regular fetches or push. Updating the version does not happen regularly. Cloning the minimum required amount (just a specific version tag) is a better user experience IMO. That is the case even when they are not stuck with slow internet services.

We've ended up just sharing a zipped carthage folder within our team so many times. We should really look to solve this in some way if not shallow clones. Some of the repos are really large (firebase, branch etc.) and this is a major pain point.

Cocoapods does a shallow clone. I don't understand how the complexity is increased in case of Carthage.