Yay: Feature request: shallow clone for git repos

[heitorPB]

It would save some time to do a shallow clone of a git repo, instead of getting the full history:

git clone --depth 1 --recursive --shallow-submodules <repo>

[Morganamilo]

It would save some time to do a shallow clone of a git repo, instead of getting the full history:

Prove it.

[heitorPB]

Prove it.

$ time git clone https://github.com/KiCad/kicad-source-mirror.git
Cloning into 'kicad-source-mirror'...
remote: Enumerating objects: 76, done.
remote: Counting objects: 100% (76/76), done.
remote: Compressing objects: 100% (59/59), done.
remote: Total 189715 (delta 34), reused 35 (delta 17), pack-reused 189639
Receiving objects: 100% (189715/189715), 356.21 MiB | 9.87 MiB/s, done.
Resolving deltas: 100% (150876/150876), done.

real    0m44.684s
user    0m43.036s
sys 0m9.057s

$ time git clone --depth 1 --recursive --shallow-submodules https://github.com/KiCad/kicad-source-mirror.git
Cloning into 'kicad-source-mirror'...
remote: Enumerating objects: 4407, done.
remote: Counting objects: 100% (4407/4407), done.
remote: Compressing objects: 100% (4054/4054), done.
remote: Total 4407 (delta 1250), reused 1015 (delta 321), pack-reused 0
Receiving objects: 100% (4407/4407), 20.78 MiB | 6.25 MiB/s, done.
Resolving deltas: 100% (1250/1250), done.

real    0m8.779s
user    0m2.518s
sys 0m0.800s

It is faster to get only the last commit of a branch than to get the full history of a repo. Enough for you?

[Morganamilo]

Those are upstream git repos and is makepkg's job to handle. Yay only downloads the AUR repos which are small enough that a shallow clone would have no impact.

And as far as makepkg goes, they refuse to support this: https://bugs.archlinux.org/task/52957?

[Jguer]

I think it's just a confusion of what yay actually git clones when you pull a PKGBUILD.
When yay says it's cloning a repo (and let's use kicad-git as the example you chose) it's cloning https://aur.archlinux.org/kicad-git.git that only has a the very small PKGBUILD file and not the place where the code actually lives https://git.launchpad.net/kicad.

% time git clone https://aur.archlinux.org/kicad-git.git                    /tmp
Cloning into 'kicad-git'...
remote: Enumerating objects: 122, done.
remote: Counting objects: 100% (122/122), done.
remote: Compressing objects: 100% (110/110), done.
remote: Total 122 (delta 12), reused 122 (delta 12)
Receiving objects: 100% (122/122), 36.69 KiB | 1.08 MiB/s, done.
Resolving deltas: 100% (12/12), done.
git clone https://aur.archlinux.org/kicad-git.git  0.05s user 0.03s system 11% cpu 0.628 total

% time git clone --depth 1 --recursive --shallow-submodules https://aur.archlinux.org/kicad-git.git
Cloning into 'kicad-git'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 0), reused 5 (delta 0)
Unpacking objects: 100% (5/5), done.
git clone --depth 1 --recursive --shallow-submodules   0.06s user 0.04s system 13% cpu 0.764 total

Although this data has no statistical relevance since it's not done in a controlled and measured environment, it relies only on one sample and it won't be possible to replicate specifically it should at least hint that there is no advantage to shallow cloning PKGBUILD repos (specially when they are so useful for diffing).

In relation to makepkg clones: not our job, @Morganamilo s link also shows they are aware. Maybe in alternative you can shallow git clone the repo and then run makepkg, I do think it should pick up the source.

Closing.

[heitorPB]

It makes sense. For some unknown reason, I thought it was yay's job to
parse the PKGBUILD... Thanks for clearing things up.

On Sun, 23 Jun 2019, 07:41 J Guerreiro, [e-mail] wrote:

I think it's just a confusion of what yay actually git clones when you
pull a PKGBUILD.
When yay says it's cloning a repo (and let's use kicad-git as the example
you chose) it's cloning https://aur.archlinux.org/kicad-git.git that only
has a the very small PKGBUILD file and not the place where the code
actually lives https://git.launchpad.net/kicad.

% time git clone https://aur.archlinux.org/kicad-git.git /tmp
Cloning into 'kicad-git'...
remote: Enumerating objects: 122, done.
remote: Counting objects: 100% (122/122), done.
remote: Compressing objects: 100% (110/110), done.
remote: Total 122 (delta 12), reused 122 (delta 12)
Receiving objects: 100% (122/122), 36.69 KiB | 1.08 MiB/s, done.
Resolving deltas: 100% (12/12), done.
git clone https://aur.archlinux.org/kicad-git.git 0.05s user 0.03s system 11% cpu 0.628 total

% time git clone --depth 1 --recursive --shallow-submodules https://aur.archlinux.org/kicad-git.git
Cloning into 'kicad-git'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 0), reused 5 (delta 0)
Unpacking objects: 100% (5/5), done.
git clone --depth 1 --recursive --shallow-submodules 0.06s user 0.04s system 13% cpu 0.764 total

Although this data has no statistical relevance since it's not done in a
controlled and measured environment, it relies only on one sample and it
won't be possible to replicate specifically it should at least hint that
there is no advantage to shallow cloning PKGBUILD repos (specially when
they are so useful for diffing).

In relation to makepkg clones: not our job, @Morganamilo
https://github.com/Morganamilo s link also shows they are aware. Maybe
in alternative you can shallow git clone the repo and then run makepkg, I
do think it should pick up the source.

Closing.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Jguer/yay/issues/972?email_source=notifications&email_token=ADGWRRVDYHDH2RVJQTP7HVDP35HMJA5CNFSM4H2XNPY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYK3PRQ#issuecomment-504739782,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADGWRRXQNUX3DG77TQAJARTP35HMJANCNFSM4H2XNPYQ
.

[soloturn]

@Jguer , @Morganamilo you are right when you say it makes sense. but - otoh - makepkg is very modular and crealting a file "makepkg-shallow" with the following contents and call it instead of makepkg would do the trick.

#!/bin/bash 

git() {
  if [[ $# -gt 1 && $1 == 'clone' ]]; then
    /bin/git "$@" --depth=1
  else
    /bin/git "$@"
  fi
}

source /bin/makepkg "$@" 

i know it woudl be not beautiful to add a script like this to the yay codebase. but it is so small that one might try it, switchable, in yay, and if it works suggst to add it upstream.

sources:

• andrew gregory, make-shallow.sh https://gist.github.com/andrewgregory/770cbedfa2da3817d762483155a330b8
• allan mcrae, saying if somebody comes up with a switch he'd consider adding it: https://bugs.archlinux.org/task/34677

[Jguer]

just those /bin/ entries will create issues for users

this is a makepkg/PKGBUILD specific issue, not a package fetching and dependency resolving issue.

I would prefer the switch to be added on makepkg as we support makepkg options

[qiujiangkun]

Sorry for my misunderstanding