Straight.el: Add option for shallow clones

[raxod502]

Cloning Git repos can take a while. It would be nice to offer a config option to clone repositories shallowly by default.

[vyp]

In regards to freezing versions into a version lockfile, do you know of a way for shallow clones to clone at a depth needed for a particular git revision (i.e. the revision specified in the lockfile)? Because if not, shallow clones would not work with freezing, right?

[raxod502]

Good point. I guess we would have to do something like clone with a default depth (say, 100) and then if the ref doesn't exist, fetch the rest of the history.

It is possible to fetch a specific commit, but unfortunately that doesn't work with all server configurations (and, in particular, it doesn't work with GitHub):

$ git fetch origin f63eb28cca61edfa338041da4b4a7fff378c621c
error: Server does not allow request for unadvertised object f63eb28cca61edfa338041da4b4a7fff378c621c

[vyp]

I found that since git version 2.11, git clone has a --shallow-since=<date> option. I haven't tested anything at all, but if I understand correctly, it means initially if one does not have a version lockfile, and if they want to use shallow clones, then straight can shallow clone with a depth of 1. And then later when they invoke straight-freeze-versions, the date (of the recorded git revision) is also recorded in the version lockfile, which can then later be used with --shallow-since when the user wants to reproduce their setup. (And then I assume you'd just git reset --hard <the recorded revision> after the shallow clone and it would hopefully work.)

This sounds too good to be true though so I'm probably wrong, but I can't test this at the moment... but I still wanted to mention it now. :wink:

[vyp]

Actually I just tried this and it seems to work! :tada:
Not sure how robust this method would be though. :woman_shrugging:

Edit: For future reference in case I forget, the <date> format I used was YYYY-MM-DD. So for example, git clone https://github.com/raxod502/straight.el --shallow-since=2017-08-01.

[raxod502]

That is pretty sweet. :+1:

It'll make the lockfiles a little ugly and harder to hand-edit so I'll make this an opt-in option.

[vyp]

Sometimes shallow cloning using --shallow-since doesn't work properly and does a full clone instead, even if the date given matches with the commit/tag. I think that it might be due to local time or timezone differences or something, but I haven't bothered to figure out what exactly it could be.

However, it seems to reliably shallow clone if I just subtract 1 day from the date of the commit/tag, which I feel like is an acceptable trade off. If we do this, a time/date library would be needed and thankfully emacs seems to have one, linking here for future reference: https://www.gnu.org/software/emacs/manual/html_node/elisp/Time-Calculations.html#Time-Calculations.

I'm not sure how --shallow-since would work (or not) with submodules though, haven't messed with it.

[raxod502]

It might honestly be better to just do #149 instead, since it'll have mostly the same effect for UX: you're acquiring the package in some nonstandard way for speed, but then when it's time to do actual development you probably want to convert it to a normal repository. (The reason being that shallow clones are full of caveats, as you've observed, whereas I pretty much already know how to implement #149.)

[xuchunyang]

The size of the EmacsMirror git repo epkgs is large

~ $ du -sh ~/.emacs.d/straight/repos/* | sort -hr | head
311M    /Users/xcy/.emacs.d/straight/repos/epkgs
150M    /Users/xcy/.emacs.d/straight/repos/language-detection.el
130M    /Users/xcy/.emacs.d/straight/repos/elpa
37M /Users/xcy/.emacs.d/straight/repos/helm
35M /Users/xcy/.emacs.d/straight/repos/melpa
26M /Users/xcy/.emacs.d/straight/repos/magit
24M /Users/xcy/.emacs.d/straight/repos/flycheck
15M /Users/xcy/.emacs.d/straight/repos/notmuch
9.2M    /Users/xcy/.emacs.d/straight/repos/web-mode
8.5M    /Users/xcy/.emacs.d/straight/repos/dired-sidebar

Shallow clone reduces the size quite a lot (311M -> 26M)

~ $ git clone --depth 1 file:////Users/xcy/.emacs.d/straight/repos/epkgs
Cloning into 'epkgs'...
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 9 (delta 0), reused 3 (delta 0)
Receiving objects: 100% (9/9), 6.58 MiB | 5.59 MiB/s, done.
~ $ du -sh epkgs
26M epkgs

And I think there is no point to keep the whole history of this repo, maybe it is a good idea to shallow clone this repo.

[severen]

As someone who has their OS on a small partition, I would quite like this feature. There's not much point in storing all of the history on disk (by default), especially for things that I know I won't want to do anything with, such as epkgs or Melpa.

[raxod502]

Rip. GitHub didn't pick up that that PR "partially" closes this issue :P